Prediction system, control method, and control program

Abstract

To enable generation and outputting of a prediction image showing the condition of a target site of a subject.SOLUTION: A prediction system is provided, comprising a prediction information acquisition unit for acquiring (a) a subject image capturing a target site of a subject at a first point in time and (b) first prediction information regarding the target site at a second point in time after a given period from the first point in time; and a prediction image generation unit configured to generate and output a prediction image showing the condition of the target site at the second point in time from the subject image on the basis of the first prediction information.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to a prediction system, a control method, and a control program for predicting the condition of a target part of a human body. [Background technology]

[0002] As described in Patent Document 1, it has been proposed to use a neural network to assist in the diagnosis of osteoporosis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2008-36068 Summary of the Invention

[0004] A prediction system according to one embodiment of the present disclosure includes: a prediction information acquisition unit that acquires (a) a subject image depicting a target area of ​​a subject having a specified disease at a first time point; and (b) first prediction information regarding the onset or progression of the disease related to the target area at a second time point that is a specified period of time after the first time point; a prediction image generation unit having a prediction image generation model that generates a prediction image that predicts the state of the target area at the second time point from the first prediction information and a converted image in which the file format of the subject image is converted to a specified file format; and a prediction information generation unit that generates the first prediction information, wherein the predictive image generation model is generated by machine learning using first learning data including first data of the same type as the subject image and second data of the same type as the first prediction information, and first training data including third data related to the second data, and the first prediction information is generated based on at least the subject image.

[0005] Furthermore, a control method according to one aspect of the present disclosure is a control method for a prediction system, and includes a prediction information acquisition step of acquiring (a) a subject image showing a target area of ​​a subject having a specified disease at a first time point, and (b) first prediction information regarding the onset or progression of the disease related to the target area at a second time point a specified period of time has elapsed since the first time point; a prediction image generation step of generating and outputting a prediction image that predicts the state of the target area at the second time point from the first prediction information and a converted image in which the file format of the subject image has been converted to a specified file format; and a prediction information generation step of generating the first prediction information, wherein the prediction system has a prediction image generation model capable of generating the prediction image using the subject image and the first prediction information, and the first prediction information is generated based on at least the subject image.

[0006] Furthermore, the prediction system according to each aspect of the present disclosure may be realized by a computer. In this case, the control program for the prediction system that causes the computer to operate as each part (software element) of the prediction system to realize the prediction system, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present disclosure.

[0007] When the prediction system is realized by multiple computers, the prediction system may be realized by the computers by operating each computer as a part (software element) provided in each of the multiple computers that make up the prediction system. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating a configuration example of a prediction system according to an embodiment of the present disclosure. [Figure 2] FIG. 10 is a block diagram illustrating a configuration example of a prediction system according to another aspect of the present disclosure. [Figure 3] FIG. 1 is a block diagram illustrating an example of a configuration of a prediction system according to an embodiment of the present disclosure. [Figure 4] FIG. 10 is a diagram illustrating an example of the configuration of a neural network included in a predicted image generating unit. [Figure 5] 1 is a flowchart showing an example of the flow of processing performed by the prediction system according to the first embodiment. [Figure 6] FIG. 10 is a block diagram illustrating an example of a configuration of a prediction system according to another aspect of the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating an example of a configuration of a neural network included in a prediction information generating unit. [Figure 8] 10 is a flowchart showing an example of the flow of processing performed by the prediction system according to the second embodiment. [Figure 9] FIG. 10 is a block diagram illustrating an example of a configuration of a prediction system according to another aspect of the present disclosure. [Figure 10] 10 is a flowchart showing an example of the flow of a learning process of a neural network included in a prediction information generating unit. [Figure 11] 10 is a flowchart showing another example of the flow of processing performed by the prediction system according to the second embodiment. [Figure 12] FIG. 10 is a block diagram illustrating an example of a configuration of a prediction system according to another aspect of the present disclosure. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of a neural network included in a predicted image generating unit. [Figure 14] 10 is a flowchart showing an example of the flow of a learning process of a neural network included in an intervention effect prediction unit. [Figure 15] 11 is a flowchart showing an example of the flow of processing performed by the prediction system according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] In order to improve the condition of the target area of ​​the subject and slow the onset and progression of the disease, it is desirable to start interventions such as treatment and lifestyle guidance as early as possible.

[0010] In order to start intervention early, it is important to make the subject aware of the predicted results regarding the condition of the target area of ​​the subject and to make the subject understand the need for intervention.

[0011] [Embodiment 1] (Outline of the forecasting system) A prediction system according to one aspect of the present disclosure is a system that generates and outputs a predicted image that predicts a change in the state of a target part of a subject's body. Here, the target part may be any part of the subject's body, such as the whole body, head, eyes, oral cavity, neck, arms, hands, torso, waist, buttocks, legs, and feet. The predicted image may also be an image that predicts a change in the state of any of the target part's skin, hair, eyeballs, teeth, gums, muscles, fat, bones, cartilage, joints, and intervertebral discs.

[0012] The predicted image may be an image that predicts changes that will occur in a target area of ​​a subject affected by a disease. For example, the predicted image may be an image that shows at least one of the shape (e.g., abdominal circumference, chest circumference, height, swelling, atrophy, joint angle and curvature, etc.) and appearance (e.g., posture, wrinkles, spots, redness, cloudiness, darkening, yellowing, etc.) of the target area of ​​the subject.

[0013] In one example, the subject may have a disease in the target region. In this case, the predicted image may be an image that predicts a change in symptoms in the target region of the subject due to the influence of the disease. This specification will be described using an example of a prediction system that generates and outputs a predicted image that predicts a change in symptoms in the target region of the subject. In this case, the predicted image is an image that shows the effect of the subject's disease on the target region. Here, the "image that shows the effect on the target region" may be any image that shows a change in the shape of the target region affected by the disease, a qualitative or quantitative change caused in the tissue of the target region by the disease, or the like.

[0014] The disease may include at least one of obesity, alopecia, cataracts, periodontal disease, rheumatoid arthritis, Heberden's nodes, hallux valgus, osteoarthritis, spondylosis deformans, compression fractures, and sarcopenia. The disease may also include (i) syndromes such as metabolic syndrome and locomotive syndrome, which represent a group of pathological conditions consisting of a variety of symptoms, and (ii) physical changes such as aging and tooth alignment.

[0015] The prediction system generates a predicted image that predicts symptoms in the target area at a second time point after a predetermined period has elapsed from the first time point, based on a subject image that shows the target area of ​​the subject at a first time point. In this specification, the term "subject image" may also refer to image data that shows the subject image.

[0016] Here, the first time point may be, for example, the time point at which a subject image of the target area of ​​the subject is acquired. The first time point may typically be the time point at which a subject image of the current state of the target area of ​​the subject is acquired. That is, the first time point may substantially refer to the present time. Furthermore, the predetermined period may be any period that has elapsed since the first time point, such as six months, one year, five years, ten years, or fifty years. That is, the second time point may substantially refer to any time point in the future. The predetermined period is not limited to one period, but may include multiple periods. That is, the predicted image may include images generated by predicting symptoms of the target area of ​​the subject at multiple times, such as six months, one year, five years, ten years, and fifty years after the first time point.

[0017] The subject image is an image showing a target part of a subject at a first time point. In one example, the subject image may be an external image of the subject's whole body, head, upper body, lower body, upper limbs, or lower limbs. In another example, the subject image may be a medical image of the target part of the subject taken for examining the subject. The medical image may include at least one of an X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a PET (Positron Emission Tomography) image, and an ultrasound image of the subject. In another example, the subject image may be an image showing at least one of the shape (e.g., abdominal circumference, chest circumference, height, swelling, atrophy, joint angle and curvature, etc.) and appearance (e.g., posture, wrinkles, spots, redness, cloudiness, darkening, yellowing, etc.) of the target part of the subject.

[0018] Furthermore, the prediction system uses first prediction information about the target region at a second time point in addition to the target image to generate a predicted image.

[0019] The first prediction information may be information about symptoms in the target area of ​​the subject at a second time point. In this case, the first prediction information may include information indicating symptoms in the target area of ​​the subject at multiple time points, such as six months, one year, five years, ten years, and fifty years after the first time point. The first prediction information is information including, for example, predictions about symptoms that are likely to occur in the target area of ​​the subject, the timing of the onset of the symptoms, and the degree of progression of the symptoms in the target area of ​​the subject.

[0020] The first prediction information may be information regarding at least one of the shape and appearance of the target part of the subject at the second time point. In this case, the first prediction information may be information indicating at least one of the shape (e.g., waist circumference, chest circumference, height, etc.) and appearance (e.g., posture, wrinkles, blemishes, etc.) of the target part of the subject at multiple time points, such as six months, one year, five years, ten years, and fifty years after the first time point.

[0021] The first prediction information may be information related to at least one of the shape and appearance of the target site, which is related to a disease of the target site. The first prediction information may include the following information as information that is likely to cause changes in the target site of the subject. For example, the first prediction information may include: (i) information related to obesity, such as weight, body mass index (BMI), abdominal circumference, visceral fat mass, blood pressure, blood glucose level, lipids, uric acid level, liver function index, etc.; (ii) information related to alopecia, such as hair count, sex hormone levels, Norwood classification, Ludwig classification, etc.; (iii) information related to cataracts, such as visual acuity, field of view, degree of opacity, Emery-Little classification, etc.; (iv) information related to periodontal disease, such as pain and swelling level, number of remaining teeth, gingivitis index, periodontal pocket depth, etc.; and (v) information related to rheumatoid arthritis, such as pain level, swelling level, joint angle, joint range of motion, Larsen classification, Stein classification, etc. (vi) information related to Heberden's nodes may include information on the degree of pain, degree of swelling, range of joint motion, etc.; (vii) information related to hallux valgus may include information on the degree of pain, degree of swelling, range of joint motion, HV angle, M1-M2 angle, etc.; (viii) information related to osteoarthritis may include information on the degree of pain, degree of swelling, joint angle, range of joint motion, degree of stiffness, joint cartilage thickness, Kellgren-Laurence (KL) classification, presence or absence of lameness, etc.; (ix) in the case of spondylosis osteoarthritis, information on the degree of pain, degree of spinal curvature, range of spinal motion, KL classification, etc.; (x) information related to compression fractures may include information on the degree of pain, range of spinal motion, etc.; and (xi) information related to sarcopenia may include information on muscle mass, walking speed, grip strength, etc.

[0022] In this way, the prediction system generates and outputs a predicted image that predicts the state of the target part at a second time point based on a subject image showing the target part of the subject at a first time point and first prediction information regarding the target part at a second time point after a predetermined period of time has passed since the first time point.

[0023] The prediction system can output the state of the target region at the second time point as a visually easy-to-understand predicted image. Furthermore, since the predicted image is generated from a subject image that is an image of the subject himself, it is a realistic image that is persuasive to the subject. Therefore, for example, if the predicted image is presented to the subject by a doctor in charge of the subject, the subject can recognize the state of the target region at the second time point and easily understand the need for intervention.

[0024] The predicted image may be an image simulating an external image of any of the subject's whole body, head, upper body, lower body, upper limbs, and lower limbs. In another example, the predicted image may be an image simulating a medical image of a target part of the subject obtained during an examination of the subject. In another example, the predicted image may be an image showing at least one of the shape (e.g., abdominal circumference, chest circumference, height, swelling, atrophy, joint angle and curvature, etc.) and appearance (e.g., posture, wrinkles, spots, redness, cloudiness, darkening, yellowing, etc.) of the target part of the subject. For example, if the subject image is an appearance image showing the subject's current skin appearance (e.g., wrinkles, spots, redness, cloudiness, darkening, yellowing, etc.) and the first prediction information is information about the degree of wrinkles, spots, redness, cloudiness, darkening, or yellowing of the subject's future skin, the prediction system can output an image showing the subject's future skin appearance as a predicted image based on the subject image and the first prediction information. For example, if the subject image is a medical image showing the subject's current joint angle and the first prediction information is information about the subject's future joint angle, the prediction system can output the following two images as predicted images: (1) a medical image showing the subject's current joint angle and, based on the first prediction information, a medical image showing the subject's future joint angle; (2) an appearance image showing the subject's current joint appearance and, based on the first prediction information, an image showing the subject's future joint appearance as a predicted image.

[0025] (Configuration of prediction system 100) Next, the configuration of a prediction system 100 including a prediction device 1 that acquires a subject image and first prediction information, and generates and outputs a predicted image from the subject image based on the first prediction information will be described with reference to Fig. 1. In one embodiment of the present disclosure, the prediction device 1 of the prediction system 100 can function independently as the above-mentioned prediction system. Fig. 1 is a block diagram showing an example configuration of the prediction system 100 in a medical facility 5 that has the prediction device 1 installed.

[0026] 1, the prediction system 100 includes a prediction device 1 and one or more terminal devices 2 communicatively connected to the prediction device 1. In this manner, the prediction system 100 may include the prediction device 1 and a device (e.g., the terminal device 2) capable of presenting a predicted image output from the prediction device 1.

[0027] The prediction device 1 is a computer that acquires a subject image and first prediction information, generates and outputs a predicted image from the subject image based on the first prediction information, and transmits the predicted image to the terminal device 2. The prediction device 1 may be connected to a LAN in a medical facility 5, as shown in Fig. 1. The configuration of the prediction device 1 will be described later.

[0028] The terminal device 2 receives the predicted image from the prediction device 1 and presents the predicted image. The terminal device 2 may be a computer or the like used by a medical professional such as a doctor belonging to the medical facility 5. The terminal device 2 may be connected to a LAN in the medical facility 5 as shown in FIG. 1. The terminal device 2 may be, for example, a personal computer, a tablet terminal, a smartphone, or the like. The terminal device 2 has a communication unit for transmitting and receiving data to and from other devices, an input unit such as a keyboard and a microphone, a display unit capable of displaying the predicted image, and the like.

[0029] In the prediction system 100, the prediction device 1 and the terminal device 2 are provided as separate entities, but the prediction device 1 and the terminal device 2 may be integrated. For example, the prediction device 1 may have a display unit capable of displaying a predicted image, thereby providing the functions of the terminal device 2.

[0030] The prediction system 100 may further include a first prediction information management device 3, a subject image management device 4, and an electronic medical record management device 9.

[0031] The first prediction information management device 3 is a computer that functions as a server for managing the first prediction information. The first prediction information management device 3 may be connected to a LAN of a medical facility 5, as shown in FIG. 1. In this case, the prediction device 1 may obtain the first prediction information of the subject from the first prediction information management device 3.

[0032] The subject image management device 4 is a computer that functions as a server for managing subject images. The subject image management device 4 may be an image of a subject who has undergone a medical examination regarding the condition of a target body part at a medical facility 5. In one example, the subject image may be a medical image taken within the medical facility 5. The subject image management device 4 may be communicably connected to an imaging device, such as an X-ray imaging device, within the medical facility 5. In this case, images taken by the imaging device may be recorded in the subject image management device 4, for example, via a LAN. The subject image management device 4 may be connected to a LAN in the medical facility 5, as shown in FIG. 1. In this case, the prediction device 1 may acquire the subject image from the subject image management device 4.

[0033] The electronic medical record management device 9 is a computer that functions as a server for managing electronic medical record information of subjects who have received medical treatment at the medical facility 5. The electronic medical record management device 9 may be connected to a LAN of the medical facility 5, as shown in FIG. 1. In this case, the prediction device 1 may acquire basic information related to the subject from the electronic medical record management device 9. The basic information is information included in the electronic medical record information, and may include at least one of the subject's gender, age, height, weight, and information indicating the condition of the target part of the subject at a first time point.

[0034] FIG. 1 shows an example in which a LAN (local area network) is installed within a medical facility 5, and a prediction device 1, a terminal device 2, a first prediction information management device 3, a subject image management device 4, and an electronic medical record management device 9 are connected to the LAN, but this is not limiting. For example, the network within the medical facility 5 may be the Internet, a telephone communication line network, an optical fiber communication network, a cable communication network, a satellite communication network, or the like. The LAN within the medical facility 5 may be communicably connected to an external communication network. In this case, for example, the terminal device 2 may be a computer used by a patient, or the like.

[0035] In the prediction system 100, the prediction device 1 may be directly connected to at least one of the terminal device 2, the first prediction information management device 3, the subject image management device 4, and the electronic medical record management device 9 without using a LAN. In addition, the number of the terminal device 2, the first prediction information management device 3, the subject image management device 4, and the electronic medical record management device 9 that can communicate with the prediction device 1 may be multiple. Furthermore, multiple prediction devices 1 may be introduced in the prediction system 100.

[0036] (Configuration of prediction system 100a) The prediction device 1 may not be a computer installed in a predetermined medical facility 5, but may be communicably connected to a LAN installed in each of a plurality of medical facilities 5 via a communication network 6. Fig. 2 is a block diagram showing an example configuration of a prediction system 100a according to another embodiment of the present disclosure.

[0037] The prediction system 100a shown in Figure 2 includes a medical facility 5a, a medical facility 5b, a prediction device 1 communicatively connected to each device of the medical facilities 5a and 5b via a communication network 6, and a first prediction information management device 3 communicatively connected to the prediction device 1.

[0038] The medical facility 5a includes a terminal device 2a, a subject image management device 4a, and an electronic medical record management device 9a, all of which are communicatively connected. Meanwhile, the medical facility 5b includes a terminal device 2b, a subject image management device 4b, and an electronic medical record management device 9b, all of which are communicatively connected. Hereinafter, when there is no need to distinguish between the terminal devices 2a and 2b, and the medical facilities 5a and 5b, they will be referred to as the "terminal device 2" and the "medical facility 5," respectively.

[0039] 2 shows an example in which LANs of medical facilities 5a and 5b are connected to a communication network 6. The prediction device 1 is not limited to the configuration shown in FIG. 2 as long as it is communicably connected to devices in each medical facility via the communication network 6. For example, the prediction device 1 and the first prediction information management device 3 may be installed in the medical facility 5a or the medical facility 5b.

[0040] A prediction system 100a having such a configuration may include a first prediction information management device 3a installed in a medical facility 5a and a first prediction information management device 3b installed in a medical facility 5b. In this case, the prediction device 1 can acquire the first prediction information and a subject image of the subject Pa from the first prediction information management device 3a and a subject image management device 4a in the medical facility 5a, respectively. The prediction device 1 can then transmit a predicted image of the condition of the target region of the subject Pa to a terminal device 2a installed in the medical facility 5a. The prediction device 1 can acquire the first prediction information and a subject image of the subject Pb from the first prediction information management device 3b and a subject image management device 4b in the medical facility 5b, respectively. The prediction device 1 can then transmit a predicted image of the condition of the target region of the subject Pb to a terminal device 2b installed in the medical facility 5b.

[0041] In this case, the first prediction information and subject image of each subject may include identification information unique to each medical facility 5 that examines each subject, and identification information unique to each subject that is assigned to each medical facility 5. The identification information unique to each medical facility 5 may be, for example, a facility ID. Furthermore, the identification information unique to each subject may be, for example, a patient ID. Based on this identification information, the prediction device 1 can correctly transmit a prediction image that predicts the condition of the target area of ​​the subject to the terminal device 2 of each medical facility 5 where the subject was examined.

[0042] (Configuration of prediction systems 100 and 100a) Next, the configuration of the prediction systems 100 and 100a will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of the prediction systems 100 and 100a according to one embodiment of the present disclosure. For convenience of explanation, components having the same functions as components already described will be denoted by the same reference numerals, and their description will not be repeated.

[0043] The prediction system 100, 100a shown in Figure 3 includes a prediction device 1, one or more terminal devices 2 communicatively connected to the prediction device 1, a first prediction information management device 3, and a subject image management device 4.

[0044] (Configuration of prediction device 1) The prediction device 1 includes a control unit 7 that performs overall control of each unit of the prediction device 1, and a storage unit 8 that stores various data used by the control unit 7. The control unit 7 includes a prediction information acquisition unit 71, a predicted image generation unit 72, and an output control unit 73. The storage unit 8 stores a control program 81 that is a program for performing various controls of the prediction device 1.

[0045] <Prediction information acquisition unit 71> The prediction information acquisition unit 71 acquires the subject image from the subject image management device 4 and acquires the first prediction information from the first prediction information management device 3. The subject image and the first prediction information are input data to the prediction image generation unit 72.

[0046] The subject image and the first prediction information will be described using several disease examples. For example, if the disease is obesity, the subject image may be an image of the current subject's whole body or abdomen, and the first prediction information may be information regarding the subject's weight, BMI, abdominal circumference, visceral fat mass, blood pressure, blood glucose level, lipids, uric acid level, or liver function value. For example, if the disease is alopecia, the subject image may be an image of the current subject's whole body or head, and the first prediction information may be information regarding the subject's hair count, sex hormone levels, Norwood classification, or Ludwig classification. For example, if the disease is cataract, the subject image may be an image of the current subject's head (face) or eyes, and the first prediction information may be information regarding the subject's visual acuity, visual field, degree of cloudiness of the eye lens, or Emery-Little classification. For example, if the disease is periodontal disease, the subject image may be an image of the subject's current head (face) or oral cavity, and the first prediction information may be information regarding the subject's tooth or gum pain level, tooth or gum swelling level, number of remaining teeth, gingival inflammation index, or periodontal pocket depth. If the disease is periodontal disease, the subject image may be an image of the subject's open mouth or a closed mouth. For example, if the disease is rheumatoid arthritis, the subject image may be an image of the subject's current whole body, upper limbs, or lower limbs, and the first prediction information may be information regarding the pain level, swelling level, joint angle, joint range of motion, Larsen classification, or Stein-Blocker classification of the subject's whole body, upper limbs, or lower limbs. For example, if the disease is Heberden's nodes, the subject image may be an image of the subject's current hand, and the first prediction information may be information regarding the pain level, swelling level, or joint range of motion of the subject's hand. For example, if the disease is hallux valgus, the subject image may be an image showing the subject's current foot, and the first prediction information may be information regarding the degree of pain, degree of swelling, joint range of motion, HV angle, or M1-M2 angle of the subject's foot.For example, if the disease is osteoarthritis, the subject image may be an image of the subject's current whole body, upper limbs, or lower limbs, and the first prediction information may be information regarding the degree of pain, degree of swelling, joint angle, joint range of motion, or KL classification of the subject's whole body, upper limbs, or lower limbs. For example, if the disease is spondylosis deformans, the subject image may be an image of the subject's current whole body, neck, thoracic, or lumbar region, and the first prediction information may be information regarding the degree of spinal curvature, range of motion, or KL classification of the subject. For example, if the disease is compression fracture, the subject image may be an image of the subject's current whole body or lumbar region, and the first prediction information may be information regarding the degree of spinal curvature, range of motion, or KL classification of the subject. For example, if the disease is sarcopenia, the subject image may be an image of the subject's current whole body, upper limbs, or lower limbs, and the first prediction information may be information regarding the subject's muscle mass. The subject image may be a medical image taken during the diagnosis and treatment of each disease. For example, if the disease is osteoarthritis of the knee, the subject image may be an X-ray image showing the subject's current knee joint, and the first prediction information may be information about the angle between the subject's tibia and femur two years from now.

[0047] <Predicted Image Generator 72> The predicted image generation unit 72 generates and outputs a predicted image that predicts the state of the target region at a second time point from the subject image based on the first prediction information. The predicted image generation unit 72 may generate an image that imitates at least a part of the subject image used to generate the predicted image. The predicted image generated by the predicted image generation unit 72 may also be an image that shows the effect of a disease that has occurred in the target region on the target region. The generated predicted image may include an image related to a region of the subject that has not changed at the second time point since the first time point. In other words, the predicted image may include an image related to a region that has changed from the first time point to the second time point and an image related to a region that has not changed from the first time point to the second time point.

[0048] The predicted image generation unit 72 may have any known image editing function and video editing function. In this case, the predicted image generation unit 72 generates a predicted image by converting the subject image into an editable file format and then modifying the subject image based on the first prediction information. For example, if the subject image is an image of the subject's current lower limbs and the first prediction information is information about the angle between the subject's tibia and femur two years from now, the predicted image generation unit 72 first converts the subject image into a predetermined file format. The predicted image generation unit 72 modifies the angle between the tibia and femur in the subject image after the file format conversion based on the first prediction information, and generates a predicted image.

[0049] The predicted image generation unit 72 may have a predicted image generation model capable of generating a predicted image using the subject image and the first prediction information. Here, the predicted image generation model may be a neural network trained using multiple image data showing the target area as training data. For example, a convolutional neural network (CNN), a generative adversarial network (GAN), an autoencoder, or the like may be applied as the predicted image generation model.

[0050] The predicted image generation unit 72 inputs the target person image and the first prediction information to the predicted image generation model, causing it to output a predicted image. The predicted image generation unit 72 outputs the predicted image output from the predicted image generation model (i.e., generated by the predicted image generation unit 72).

[0051] The predicted image generation model is a calculation model used when the predicted image generation unit 72 performs calculations based on input data. The predicted image generation model is generated by performing machine learning, which will be described later, on the neural network of the predicted image generation unit 72.

[0052] <Output control unit 73> The output control unit 73 transmits the predicted image output from the predicted image generation unit 72 to the terminal device 2. The output control unit 73 may transmit at least one of the subject image and the first prediction information used to generate the predicted image to the terminal device 2 together with the predicted image.

[0053] The prediction device 1 may be configured to include a display unit (not shown). In this case, the output control unit 73 may cause the display unit to display the predicted image. In this case, the output control unit 73 may cause the display unit to display, together with the predicted image, at least one of the subject image and the first prediction information used to generate the predicted image.

[0054] By including the predicted image generation unit 72 having a predicted image generation model, the prediction systems 100 and 100a can generate and output a realistic predicted image in which the state of the target part at the second time point is reflected in the subject image. This allows the prediction systems 100 and 100a to allow the subject to clearly recognize the state of the target part at the second time point.

[0055] In one example, a trained predictive image generation model may be pre-installed in the prediction device 1. Alternatively, the prediction device 1 may further include a first training unit 74 that performs training processing for the predictive image generation unit 72.

[0056] <First Study Section 74> The first learning unit 74 controls the learning process for the neural network that the predicted image generating unit 72 has.

[0057] (Learning process for predictive image generation model) The learning process for generating a predictive image generation model to which a generative adversarial network (GAN) is applied will be described below with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of a neural network included in the predictive image generation unit 72.

[0058] As shown in Fig. 4, the predicted image generation model to which a generative adversarial network is applied has two networks: a generator network (hereinafter referred to as generator 721) and a classifier network (hereinafter referred to as classifier 722). The generator 721 can generate an image that looks like a real image as a predicted image from the first prediction information and the subject image. On the other hand, the classifier 722 can distinguish between image data (fake images) from the generator 721 and real images from a first training dataset 82 described below.

[0059] First, the first learning unit 74 acquires the subject image and the first prediction information from the storage unit 8 and inputs them to the generator 721.

[0060] The generator 721 generates predicted image candidates (fake images) from the subject image and the first prediction information. The generator 721 may generate predicted image candidates by referring to real images included in the first training dataset 82.

[0061] Here, the first training dataset 82 is data used in machine learning to generate a predictive image generation model. The first training dataset 82 may include any real image that the generator 721 aims to reproduce as faithfully as possible. For example, the first training dataset 82 may include real medical images captured in the past. Here, the medical images may include, for example, at least one of X-ray image data, CT image data, MRI image data, PET image data, and ultrasound image data of target regions of each of a plurality of patients.

[0062] The first training dataset 82 may include first training data and first teacher data. The first training data is, for example, data of the same type as the subject image and data of the same type as the first prediction information. "Data of the same type as the subject image" refers to image data of the same target region as the target region in the subject image, taken from the same angle, and of the same type of image, such as a medical image or an external appearance image. "Data of the same type as the first prediction information" refers to information related to the shape and appearance of the same target region related to the same disease, if the first prediction information is information related to the shape and appearance of the target region related to the same disease. The first teacher data is data of the same type as the predicted image, and is data of the same person that was acquired more time than the first training data. The first teacher data is data related to the "data of the same type as the first prediction information," which is the first training data. "Data of the same type as the predicted image" refers to image data of the same target region as the target region in the predicted image, taken from the same angle, and of the same type of image, such as a medical image or an external appearance image.

[0063] Next, the first learning unit 74 inputs the predicted image candidates generated by the generator 721 and the real images included in the first learning dataset 82 to the classifier 722.

[0064] The classifier 722 receives as input the genuine images from the first training data set 82 and the predicted image candidates generated by the generator 721, and outputs the probability that each image is a genuine image.

[0065] The first learning unit 74 calculates a classification error that indicates how accurate the probability output by the classifier 722 is. The first learning unit 74 iteratively improves the classifier 722 and the generator 721 using backpropagation. At this time, the weights and biases of the classifier 722 are updated to minimize the classification error (i.e., to maximize the classification performance). On the other hand, the weights and biases of the generator 721 are updated to maximize the classification error (i.e., to maximize the probability that the classifier 722 will mistake a predicted image candidate for a real image).

[0066] The first learning unit 74 updates the weights and biases of the classifier 722 and the weights and biases of the generator 721 until the probability output by the classifier 722 meets a predetermined standard. This enables the predicted image generation unit 72 to generate a predicted image that is indistinguishable from the real thing.

[0067] (Processing performed by prediction systems 100 and 100a) The flow of processing performed by the prediction systems 100 and 100a will be described below with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the flow of processing performed by the prediction systems 100 and 100a according to this embodiment.

[0068] First, in step S1, the prediction information acquisition unit 71 acquires a subject image and first prediction information (input data) (prediction information acquisition step).

[0069] Subsequently, in response to the input of the target person image and the first prediction information, the predicted image generating unit 72 generates a predicted image in step S2 and outputs the predicted image (predicted image generating step).

[0070] [Embodiment 2] Other embodiments of the present disclosure will be described below. For convenience of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0071] The prediction systems 100 and 100a according to the above-described embodiments include a prediction device 1 that acquires the first prediction information from the first prediction information management device 3, but are not limited to this. For example, the first prediction information may be generated by the prediction device 1A. The configuration of the prediction systems 100 and 100a including such a prediction device 1A will be described with reference to FIG. 6. FIG. 6 is a block diagram showing an example of the configuration of the prediction systems 100 and 100a according to another aspect of the present disclosure.

[0072] (Configuration of prediction device 1A) The prediction device 1A includes a control unit 7A that performs overall control of each unit of the prediction device 1A, and a storage unit 8 that stores various data used by the control unit 7A. In addition to a prediction information acquisition unit 71, a predicted image generation unit 72, and an output control unit 73, the control unit 7A further includes a prediction information generation unit 75.

[0073] 6 illustrates an example in which the prediction device 1A includes the first learning unit 74, but is not limited to this. In one example, a trained predictive image generation model may be pre-installed in the prediction device 1A.

[0074] <Prediction information generating unit 75> The prediction information generation unit 75 generates first prediction information regarding the target part at a second time point, which is a predetermined period of time after the first time point, from a subject image that shows the target part of the subject at the first time point, and outputs the first prediction information to the prediction information acquisition unit 71.

[0075] The prediction information generation unit 75 may have a prediction information generation model capable of estimating the first prediction information from a subject image. The prediction information generation model is a model capable of estimating the first prediction information from a subject image of the subject and basic information about the subject. Here, the prediction information generation model may be a neural network trained using patient information about patients with diseases in the target area as training data. For example, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM), or the like may be applied as the prediction information generation model.

[0076] Patient information includes, for example, status information indicating the status of the target area of ​​each patient, obtained at multiple points in the past, and associates the status information for each patient with information indicating the points in time at which the status information was obtained.

[0077] The prediction information generation unit 75 inputs data related to the target person image into the prediction information generation model, and outputs first prediction information. The prediction information generation unit 75 outputs the first prediction information output from the prediction information generation model (i.e., generated by the prediction information generation unit 75).

[0078] The prediction information generation model is a calculation model used when the prediction information generation unit 75 performs calculations based on input data. The prediction information generation model is generated by performing machine learning, which will be described later, on the neural network of the prediction information generation unit 75.

[0079] Hereinafter, the configuration of the prediction information generating unit 75 will be further described with reference to Fig. 7, taking as an example a case where a neural network is applied as a prediction information generation model. Fig. 7 is a diagram showing an example of the configuration of the neural network included in the prediction information generating unit.

[0080] 7, the prediction information generation unit 75 includes an input layer 751 and an output layer 752. The prediction information generation unit 75 performs a calculation on input data input to the input layer 751 based on a prediction information generation model, and outputs prediction information from the output layer 752.

[0081] The prediction information generation unit 75 in FIG. 7 includes a neural network including an input layer 751 and an output layer 752. The neural network may be any neural network suitable for handling time-series information, such as an LSTM. The neural network may also be any neural network suitable for handling both time-series information and location information, such as a ConvLSTM network that combines a CNN and an LSTM. The input layer 751 can extract features of temporal changes in input data. The output layer 752 can calculate new features based on the features extracted by the input layer 751, the temporal changes in the input data, and initial values. The input layer 751 and the output layer 752 each include multiple LSTM layers. Each of the input layer 751 and the output layer 752 may include three or more LSTM layers.

[0082] The input data input to the input layer 751 may be, for example, parameters indicating feature amounts extracted from a subject image showing a target part of the subject at a first time point. In this case, the prediction information generation unit 75 can output first prediction information regarding the target part at a second time point after a predetermined period has elapsed since the first time point.

[0083] The prediction information generating unit 75 outputs, as the first prediction information, for example, a prediction result of the onset or progression of a disease in a target area of ​​the subject at a second time point after a predetermined period has elapsed since the first time point. Specifically, the prediction information generating unit 75 outputs, as the first prediction information, for example, the degree of symptoms of each disease in the subject at the second time point, the classification of each disease, and information indicating the time when invasive treatment will be required for the target area. The first prediction information shown here is an example and is not limited to this.

[0084] Furthermore, the prediction information generating unit 75 may output information indicating the QOL of the subject as third prediction information based on the above-mentioned first prediction information. Specifically, the prediction information generating unit 75 outputs, as the third prediction information, at least one of information regarding pain occurring in the target part of the subject, information regarding the subject's catastrophic thinking, information regarding the subject's motor ability, information indicating the subject's life satisfaction, and information regarding the degree of stiffness in the target part of the subject.

[0085] Information indicating the subject's QOL is information that includes at least one of the following: Information about the pain occurring in the target area of ​​the subject Information about the subject's catastrophic thinking Information about the subject's athletic ability Information indicating the subject's life satisfaction.

[0086] Information indicating the subject's QOL may include information regarding the subject's (1) physical function, (2) role functioning (physical), (3) bodily pain, (4) overall well-being, (5) vitality, (6) social functioning, (7) role functioning (mental), and (8) mental health.

[0087] Information related to QOL may include, for example, information such as the SF-36 (36-Item Short-Form Health Survey), VAS (Visual Analog Scale), NEI VFQ-25 (The 25-item National Eye Institute Visual Function Questionnaire), GOHAI (General Oral Health Assessment Index), WOMAC (Western Ontario and McMaster Universities Osteoarthritis Index), and RDQ (Roland-Morris Disability Questionnaire).

[0088] The prediction information generation unit 75 may generate at least a part of the first prediction information used by the prediction image generation unit 72 to generate a prediction image. In this case, the prediction information acquisition unit 71 may acquire the remaining first prediction information from the first prediction information management device 3.

[0089] (Processing performed by prediction systems 100 and 100a) The flow of processing performed by the prediction systems 100 and 100a of the second embodiment will be described below with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the flow of processing performed by the prediction systems 100 and 100a of the second embodiment.

[0090] First, in step S11, the prediction information generating unit 75 acquires a subject image (input data) (image acquiring step).

[0091] Subsequently, in response to the input of the subject image, the prediction information generating unit 75 generates first prediction information in step S12 and outputs the first prediction information to the prediction information acquiring unit 71 (first prediction step).

[0092] Then, the prediction information acquisition unit 71 inputs (a) the first prediction information acquired from the prediction information generation unit 75 and (b) a subject image (input data) that was acquired from the subject image management device 4 before the acquisition of the first prediction information, or that will be acquired simultaneously with or after the acquisition of the first prediction information, to the prediction image generation unit 72 (not shown).

[0093] Subsequently, in response to the input of the target person image and the first prediction information, the predicted image generating unit 72 generates a predicted image in step S13 and outputs the predicted image (predicted image generating step).

[0094] [Modification] In the prediction device 1A, the prediction information generation unit 75 generates the first prediction information based on the subject image. In contrast, in the prediction device 1B according to this modification of the second embodiment, the prediction information generation unit 75B generates the first prediction information based on basic information in addition to the subject image. The configuration of a prediction system 100, 100a including such a prediction device 1B will be described with reference to FIG. 9. FIG. 9 is a block diagram showing a modification of the configuration of the prediction system 100, 100a according to another aspect of the present disclosure. The prediction device 1B further includes a prediction information generation unit 75B in the control unit 7B.

[0095] The prediction device 1B includes a control unit 7B that performs overall control of each unit of the prediction device 1B, and a storage unit 8B that stores various data used by the control unit 7B. In addition to a prediction information acquisition unit 71, a predicted image generation unit 72, and an output control unit 73, the control unit 7B further includes a prediction information generation unit 75B and a basic information acquisition unit 76.

[0096] With the above-described configuration, prediction device 1B can generate information on symptoms that are closer to symptoms that may occur or have occurred in the target region at the second time point, i.e., more accurate first prediction information, based on the subject image captured at the first time point. As a result, prediction device 1B can generate an image showing symptoms that are closer to symptoms that may occur or have occurred in the target region at the second time point, i.e., a prediction image showing more accurate prediction information.

[0097] <Basic information acquisition section 76> The basic information acquisition unit 76 acquires basic information, which is information related to the subject, from the electronic medical record management device 9. The electronic medical record management device 9 is a computer that functions as a server for managing electronic medical record information of subjects who have received medical treatment at the medical facility 5 or a medical facility other than the medical facility 5. The electronic medical record information may include basic information and medical interview information about the subject. The basic information is input data that is input to the prediction information generation unit 75B in addition to the subject image.

[0098] The basic information includes at least one of the subject's gender, age, height, and weight, and information indicating the condition of the target region of the subject at the first time point. The basic information may further include at least one of the subject's BMI, race, occupational history, exercise history, medical history of diseases related to the target region, information regarding the shape and appearance of the target region, biomarker information, and genetic information. The basic information may also include, for example, information such as the severity of symptoms of diseases related to the target region of the subject. The basic information may also include, for example, information included in the subject's electronic medical record information. The basic information may be interview information obtained from the subject through an interview conducted at a medical facility 5 or the like, and may include, for example, information related to the subject's QOL at the first time point.

[0099] According to the above configuration, prediction device 1B can acquire basic information about the subject from electronic medical record management device 9 in addition to the subject image, and transmit a predicted image predicting the state of the target area of ​​subject Pa to terminal device 2a installed in medical facility 5a. Prediction device 1B can acquire basic information about the subject from electronic medical record management device 9 in addition to the subject image, and transmit a predicted image predicting the state of the target area of ​​subject Pb to terminal device 2b installed in medical facility 5b.

[0100] In one example, the trained prediction information generation model may be pre-installed in the prediction device 1B. Alternatively, the prediction device 1B may further include a second learning unit 77 that performs a learning process for the prediction information generation unit 75B.

[0101] <Second Study Section 77> The second learning unit 77 controls the learning process for the neural network of the prediction information generation unit 75B. A second learning data set 83, which will be described later, is used for this learning. A specific example of the learning performed by the second learning unit 77 will be described later.

[0102] (Learning process for predictive information generation model) The learning process for generating a prediction information generation model to which a neural network is applied will be described below with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the flow of learning process of the neural network included in the prediction information generating unit 75B.

[0103] First, the second learning unit 77 acquires second learning data included in the second learning data set 83 from the storage unit 8B (step S21). The second learning data includes patient images of a plurality of patients.

[0104] Next, the second learning unit 77 determines a certain patient (step S22).

[0105] Next, the second learning unit 77 inputs a patient image of a certain patient at time point A, which is included in the second learning data, to the input layer 751 (step S23). The input layer 751 may extract parameters indicating feature amounts from the input patient image.

[0106] Next, the second learning unit 77 acquires output data relating to symptoms of a target region of a certain patient from the output layer 752 (step S24). This output data includes the same content as the second training data.

[0107] Next, the second learning unit 77 acquires second training data included in the second training data set 83. Then, the second learning unit 77 compares the acquired output data with condition information that indicates the condition of a target area of ​​a certain patient at time B, which is included in the second training data, and calculates an error (step S25).

[0108] Thereafter, the second learning unit 77 adjusts the prediction information generation model so as to reduce the error (step S26).

[0109] Any known method can be applied to adjust the prediction information generation model. For example, backpropagation may be used as a method for adjusting the prediction information generation model. The adjusted prediction information generation model becomes a new prediction information generation model, and the prediction information generation unit 75B uses the new prediction information generation model in subsequent calculations. In the stage of adjusting the prediction information generation model, parameters used by the prediction information generation unit 75B may be adjusted.

[0110] The parameters include, for example, parameters used in the input layer 751 and the output layer 752. Specifically, the parameters include weighting coefficients used in the LSTM layers of the input layer 751 and the output layer 752. The parameters may also include filter coefficients.

[0111] If the error is not within a predetermined range and if patient images of all patients included in the second learning data set 83 have not been input (NO in step S27), the second learning unit 77 changes the patient (step S28) and then returns to step S23 to repeat the learning process. If the error is within a predetermined range and if patient images of all patients included in the second learning data set 83 have been input (YES in step S27), the second learning unit 77 ends the learning process.

[0112] <Second training dataset 83> The second training dataset 83 is data used in machine learning to generate a prediction information generation model. The second training dataset 83 may include patient information on patients with a disease of the target region. Here, the patient information may include condition information indicating the condition of the target region of each patient, acquired at multiple points in the past, and may be information in which the condition information for each patient is associated with information indicating the points in time at which the condition information was acquired. The second training dataset 83 includes second training data used as input data and second teacher data for calculating an error from the first prediction information output by the prediction information generation unit 75B.

[0113] The second training data may include, for example, image data showing the target region of each of the multiple patients. The image data used as the second training data may be image data of the whole body, upper body, lower body, upper limbs, or lower limbs of each of the multiple patients. The image data used as the second training data may be medical image data showing the target region of each of the multiple patients. The medical image data may include, for example, at least one of X-ray image data, CT image data, MRI image data, PET image data, and ultrasound image data showing the target region of each of the multiple subjects. The second training data is the same type of data as the subject images. The second training data is the same type of data as the first prediction information.

[0114] The second training data may include condition information indicating the condition of the target region of each patient at the time the patient image was captured, and symptom information related to the target region. Here, the condition information may include information related to the progression of symptoms of the target region. Also, here, the symptom information may include information related to the onset time of the disease in the target region.

[0115] The second training data set 83 may be data in which the second training data and the second teacher data are combined. That is, the second training data set 83 may be time-series data in which patient images acquired from each of a plurality of patients at multiple time points in the past are associated with condition information indicating the condition of the target body part at the time the patient image was captured. For example, the second training data set 83 may include parameters indicating feature amounts extracted from the following information at a certain time point and one year after the certain time point:

[0116] For example, if the disease is obesity, the second training dataset 83 may include weight, BMI, abdominal circumference, visceral fat mass, blood pressure, blood glucose level, lipids, uric acid level, or liver function value. For example, if the disease is alopecia, the second training dataset 83 may include hair count, sex hormone values, Norwood classification or Ludwig classification, etc. For example, if the disease is cataract, the second training dataset 83 may include visual acuity, visual field, degree of recommended eye opacity, or Emery-Little classification, etc. For example, if the disease is periodontal disease, the second training dataset 83 may include degree of tooth or gum pain, degree of tooth or gum swelling, number of remaining teeth, gingival index, periodontal pocket depth, etc. For example, if the disease is rheumatoid arthritis, the second training dataset 83 may include the degree of pain, degree of swelling, joint angle, range of joint motion, Larsen classification or Stein-Blocker classification, etc., of the subject's whole body, upper limbs, or lower limbs. For example, if the disease is Heberden's nodes, the second training dataset 83 may include the degree of pain, degree of swelling, or range of joint motion, etc., of the subject's hands. For example, if the disease is hallux valgus, the second training dataset 83 may include the degree of pain, degree of swelling, range of joint motion, HV angle, M1-M2 angle, etc., of the subject's feet. For example, if the disease is osteoarthritis, the second training dataset 83 may include the degree of pain, degree of swelling, joint angle, range of joint motion, degree of stiffness, joint cartilage thickness, KL classification, or presence or absence of lameness, etc., of the subject's whole body, upper limbs, or lower limbs. For example, if the disease is spondylosis deformans, the second training data set 83 may include the degree of pain, the degree of spinal curvature, the range of motion of the spine, or the KL classification, etc. For example, if the disease is compression fracture, the second training data set 83 may include the degree of pain or the range of motion of the spine, etc. For example, if the disease is sarcopenia, the second training data set 83 may include muscle mass, walking speed, grip strength, etc.

[0117] The second training data set 83 may also include parameters indicating the attributes of the subjects. Examples of the attributes of the subjects include the gender, age, height, and weight of each subject. When the second training data set 83 is time-series data, the second training unit 77 may use an image of the subject at a certain time point as the second training data, and may use an image of the subject a predetermined period after the certain time point, information about the symptoms of the target area at the time the image of the subject was captured, and information about the subject as the second training data.

[0118] The second training data set 83 may include information related to the QOL of each of the multiple subjects in the time-series data. For example, the information may include information such as SF-36 and VAS. The prediction information generation unit 75B, which has a prediction information generation model generated by machine learning using such a second training data set 83, can also output information related to the QOL of the subject at the second time point from the subject image of the subject.

[0119] Specifically, the input data used during learning by the prediction information generation unit 75B is (a) a subject image containing a target area of ​​the subject at a certain time point A, which is included in the second learning data. Based on the above-mentioned input data, the prediction information generation unit 75B outputs, as output data, first prediction information regarding the target area at time point B, a predetermined period (e.g., three years) after time point A. Specifically, the prediction information generation unit 75B outputs, as output data, information indicating, for example, the angle around the target area of ​​the subject at time point B, the degree of enlargement and shrinkage of the target area, the degree of wrinkles and blemishes on the target area, the timing and degree of pain onset in the target area, and the timing when invasive treatment for the target area will be required. The output data shown here is an example and is not limited to these.

[0120] When the prediction information generation unit 75B uses basic information as input data, the second learning unit 77 may input, when training the prediction information generation unit 75B, symptom information and attribute information of the subject as well as an image of the subject showing the target area of ​​the subject at a certain point in time A to the prediction information generation unit 75B as second learning data.

[0121] (Prediction system 100, processing performed by 100) The flow of processing performed by the prediction systems 100 and 100a including the prediction device 1B will be described below with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the flow of processing performed by the prediction systems 100 and 100a according to this embodiment.

[0122] First, in step S31, the prediction information generating unit 75B acquires a subject image (input data) and basic information (input data) (image and information acquiring step).

[0123] Next, in response to the input of the subject image and basic information, the prediction information generation unit 75B generates first prediction information in step S32 and outputs the first prediction information to the prediction information acquisition unit 71 (first prediction step).

[0124] Thereafter, the prediction information acquisition unit 71 inputs (a) the first prediction information acquired from the prediction information generation unit 75B and (b) a subject image (input data) that was acquired from the subject image management device 4 before the acquisition of the first prediction information, or that will be acquired simultaneously with or after the acquisition of the first prediction information, to the prediction image generation unit 72 (not shown).

[0125] Subsequently, in response to the input of the target person image and the first prediction information, the predicted image generating unit 72 generates a predicted image in step S33 and outputs the predicted image (predicted image generating step).

[0126] Third Embodiment Other embodiments of the present disclosure will be described below. For convenience of explanation, the same reference numerals will be used to designate components having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0127] The prediction system 100, 100a may have a function of outputting a predicted image that predicts the state of the target area at a second time point when an intervention is performed on the target area, as well as a method of intervention on the subject and the effect of the intervention. A prediction device 1C having such a function will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the configuration of the prediction system 100, 100a according to another embodiment of the present disclosure.

[0128] Here, specific examples of intervention methods will be described. For example, if the disease is obesity, the intervention method may include lifestyle guidance, dietary therapy, drug therapy, exercise therapy, surgical therapy (liposuction, gastrectomy, gastric banding, etc.), etc. For example, if the disease is alopecia, the intervention method may include lifestyle guidance, dietary therapy, drug therapy, surgical therapy (hair transplant surgery), wearing a wig, etc. For example, if the disease is cataract, the intervention method may include drug therapy, exercise therapy, surgical therapy (cataract extraction, intraocular lens placement, etc.), etc. For example, if the disease is periodontal disease, the intervention method may include oral care guidance, drug therapy, orthodontic therapy, surgical therapy (periodontal plasty, implant therapy, etc.), use of dentures, etc. For example, if the disease is rheumatoid arthritis, the intervention method may include drug therapy and surgical therapy (osteotomy, joint replacement). For example, if the disease is Heberden's nodes, the intervention method may include drug therapy, etc. For example, if the disease is hallux valgus, intervention methods may include shoe guidance, exercise therapy, orthotic therapy, drug therapy, surgical therapy (osteotomy, fusion surgery, joint replacement, etc.), etc. For example, if the disease is osteoarthritis, intervention methods may include exercise therapy, orthotic therapy, drug therapy, rehabilitation, surgical therapy (intra-articular injection, arthroscopic surgery, osteotomy, fusion surgery, joint replacement, etc.), etc. For example, if the disease is spondylosis, intervention methods may include exercise therapy, orthotic therapy, drug therapy, surgical therapy (spinal instrumentation surgery, etc.), etc. For example, if the disease is compression fracture, intervention methods may include orthotic therapy, drug therapy, surgical therapy (spinal instrumentation surgery, etc.), etc. For example, if the disease is sarcopenia, intervention methods may include lifestyle guidance, dietary therapy, drug therapy, exercise therapy, etc.

[0129] (Configuration of prediction device 1C) The prediction device 1C includes a control unit 7C that performs overall control of each unit of the prediction device 1, and a storage unit 8C that stores various data used by the control unit 7C. The control unit 7C includes a prediction information acquisition unit 71, a prediction image generation unit 72C, an output control unit 73C, a first learning unit 74, a prediction information generation unit 75B, a basic information acquisition unit 76, and a second learning unit 77, as well as an intervention effect prediction unit 78 and a third learning unit 79.

[0130] 12 shows a prediction device 1C including a first learning unit 74, a second learning unit 77, and a third learning unit 79, but is not limited to this. The prediction device 1C may include any (or all) of the first learning unit 74, the second learning unit 77, and the third learning unit 79, or may not include any (or all) of them.

[0131] For example, the prediction device 1C may not include the first learning unit 74. In this case, a trained predictive image generation model may be pre-installed in the prediction device 1C. Alternatively, the prediction device 1C may not include the second learning unit 77. In this case, a trained predictive information generation model may be pre-installed in the prediction device 1C. Alternatively, the prediction device 1C may not include the third learning unit 79. In this case, a trained intervention effect prediction model (described later) may be pre-installed in the prediction device 1C.

[0132] The memory unit 8C may store a control program 81, which is a program for performing various controls of the prediction device 1C, a first learning data set 82, and a second learning data set 83, as well as third teacher data 84 and intervention information 85, which will be described later.

[0133] 12 shows a prediction device 1C in which a control program 81, a first training data set 82, a second training data set 83, third teacher data 84, and intervention information 85 are stored in a memory unit 8C, but this is not limiting. The memory unit 8C of the prediction device 1C may store any (or all) of the control program 81, the first training data set 82, the second training data set 83, the third teacher data 84, and the intervention information 85, or none (or all) of them may be stored.

[0134] <Predicted image generation unit 72C> The predicted image generation unit 72C inputs the target person image, the first prediction information, and the second prediction information to the predicted image generation model, and outputs a predicted image. The predicted image generation unit 72C outputs the predicted image output from the predicted image generation model (i.e., generated by the predicted image generation unit 72C).

[0135] <Output control unit 73C> The output control unit 73C transmits the predicted image output from the predicted image generation unit 72C to the terminal device 2. As shown in Fig. 12 , the output control unit 73C may transmit, together with the predicted image, at least one of the subject image, the first prediction information, and the second prediction information used to generate the predicted image to the terminal device 2.

[0136] <Intervention Effect Prediction Section 78> The intervention effect prediction unit 78 outputs second prediction information indicating a method of intervention for the subject and the effect of the intervention, based on first prediction information related to the subject site at a second time point after a predetermined period has elapsed since the first time point. The intervention effect prediction unit 78 may have an intervention effect prediction model capable of estimating the second prediction information from the first prediction information.

[0137] The intervention effect prediction model is a calculation model used when the intervention effect prediction unit 78 performs calculations based on input data. The configuration of the intervention effect prediction model is not particularly limited as long as it is a calculation model that can estimate the second prediction information from the first prediction information.

[0138] The intervention effect prediction model may be a neural network, for example, a trained neural network having an input layer and an output layer. More specifically, the intervention effect prediction model may be a neural network trained using effect information as training data.

[0139] For example, the effect information includes condition information indicating the condition of the target site of each patient, which has been acquired at multiple points in time in the past, and in which the condition information for each patient is associated with intervention information 85 indicating the intervention applied to each patient. The effect information may include time-series data on the condition information of each patient, which has been acquired at multiple points in time in the past from each of multiple patients to whom an intervention has been applied in the past.

[0140] When the intervention effect prediction model is a trained neural network, the intervention effect prediction unit 78 performs calculations based on the intervention effect prediction model in response to the first prediction information being input as input data to the input layer, and outputs the second prediction information as output data from the output layer.

[0141] The second prediction information is, for example, information indicating the type of intervention and information indicating the effect of the intervention. The effect of the intervention is information indicating the symptoms of the target area of ​​the subject at the second time point when the intervention is applied. Alternatively, the effect of the intervention may be information indicating the extent to which the application of the intervention improves the symptoms of the disease related to the target area of ​​the subject at the second time point or the extent to which the progression of the symptoms is suppressed compared to when the intervention is not applied. The second prediction information may include information indicating the time when the intervention should be applied (intervention time).

[0142] For example, the intervention effect prediction unit 78 may be configured to extract feature quantities from the first prediction information and use them as input data. The extraction of feature quantities may be performed using known algorithms such as those listed below. ·Convolutional neural network (CNN) Autoencoder Recurrent neural network (RNN) ·LSTM (Long Short-Term Memory).

[0143] Hereinafter, the configuration of the intervention effect prediction unit 78 will be further explained using Fig. 7, taking as an example a case where the intervention effect prediction unit 78 uses a neural network as an intervention effect prediction model. The configuration shown in Fig. 7 is an example, and the configuration of the intervention effect prediction unit 78 is not limited to this.

[0144] As shown in FIG. 7 , the intervention effect prediction unit 78 includes an input layer 781 and an output layer 782. The intervention effect prediction unit 78 acquires first prediction information from the prediction information generation unit 75B and uses it as input data to be input to the input layer 781. The intervention effect prediction unit 78 may further acquire a subject image and use it as input data. The intervention effect prediction unit 78 may acquire basic information from the basic information acquisition unit 76 and use the basic information as input data to be input to the input layer 781. The intervention effect prediction unit 78 performs calculations based on the intervention effect prediction model on the input data input to the input layer 781 and outputs a predicted image from the output layer 782.

[0145] As shown in FIG. 7, the intervention effect prediction unit 78 includes a neural network including an input layer 781 and an output layer 782. The neural network may be any neural network suitable for handling time-series information. For example, it may be an LSTM. The neural network may be any neural network suitable for handling both time-series information and location information. For example, it may be a ConvLSTM network that combines a CNN and an LSTM. The input layer 781 can extract features of temporal changes in the input data. The output layer 782 can calculate new features based on the features extracted by the input layer 781, the temporal changes in the input data, and initial values. The input layer 781 and the output layer 782 each include multiple LSTM layers. Each of the input layer 781 and the output layer 782 may include three or more LSTM layers.

[0146] An intervention effect prediction model is generated by executing machine learning, which will be described later, on the neural network of the intervention effect prediction unit 78.

[0147] The input data input to the input layer 781 may be, for example, parameters indicating feature amounts extracted from first prediction information related to the target site at a second time point after a predetermined period has elapsed since the first time point. Alternatively, the input data may be information indicating an intervention method included in intervention information 85, which will be described later. When the intervention effect prediction unit 78 uses the intervention information 85 as input data, the intervention effect prediction unit 78 can select at least one intervention method from the intervention information 85 and output second prediction information that predicts the effect of the intervention.

[0148] In response to the input of the above-mentioned input data to the input layer 781, the output layer 782 outputs second prediction information indicating a method of intervention for the subject and the effect of the intervention.

[0149] The second prediction information may be, for example, information that indicates the degree to which the symptoms of the disease related to the target area of ​​the subject will be improved or the degree to which the progression of the symptoms will be suppressed when the intervention is applied at the second time point. More specifically, the intervention effect prediction unit 78 may output the following information as the second prediction information. How close the angles around the target area are to normal angles. How closely the area grows and shrinks to normal size. -By what percentage will wrinkles and blemishes in the target area be reduced? -How long the condition of the target area will be maintained. How much walking ability (including stair climbing) improves.

[0150] The second prediction information is the same type of data as the first prediction information. For example, if the disease is obesity, the second prediction information may be information regarding the subject's weight, BMI, abdominal circumference, visceral fat mass, blood pressure, blood glucose level, lipid level, uric acid level, or liver function index. For example, if the disease is alopecia, the second prediction information may be information regarding the subject's hair count, sex hormone levels, Norwood classification, or Ludwig classification. For example, if the disease is cataract, the second prediction information may be information regarding the subject's visual acuity, visual field, degree of opacity of the crystalline lens, or Emery-Little classification. For example, if the disease is periodontal disease, the second prediction information may be information regarding the subject's tooth or gum pain, tooth or gum swelling, number of remaining teeth, gingival index, or periodontal pocket depth. For example, if the disease is rheumatoid arthritis, the second prediction information may be information regarding the subject's whole body, upper limbs, or lower limb pain, swelling, joint angle, joint range of motion, Larsen classification, or Stein-Brocker classification. For example, if the disease is Heberden's nodes, the second prediction information may be information regarding the degree of pain, swelling, or range of joint motion of the subject's hand. For example, if the disease is hallux valgus, the second prediction information may be information regarding the degree of pain, swelling, range of joint motion, HV angle, or M1-M2 angle of the subject's foot. For example, if the disease is osteoarthritis, the second prediction information may be information regarding the degree of pain, swelling, joint angle, range of joint motion, or KL classification of the subject's entire body, upper limb, or lower limb. For example, if the disease is spondylosis osteoarthritis, the second prediction information may be information regarding the degree of spinal curvature, range of spinal motion, or KL classification of the subject. For example, if the disease is compression fracture, the second prediction information may be information regarding the degree of spinal curvature, range of spinal motion, or KL classification of the subject. For example, if the disease is sarcopenia, the second prediction information may be information regarding the subject's muscle mass.

[0151] <Intervention information 85> The intervention information 85 is information about an intervention whose effect is estimated by the intervention effect prediction unit 78. Examples of the intervention information 85 whose effect is estimated include non-invasive treatments such as weight restriction, thermotherapy, ultrasound therapy, wearing a brace, or taking supplements. The intervention information 85 may also include the estimated effect of invasive treatments such as surgical therapy.

[0152] (Learning process of neural network in predicted image generating unit 72C) The learning process for generating a predictive image generation model to which a generative adversarial network (GAN) is applied will be described below with reference to Fig. 13. Fig. 13 is a diagram showing an example of the configuration of a neural network included in a predictive image generation unit 72C. As shown in Fig. 13, the predictive image generation model to which a generative adversarial network is applied has two networks: a generator 721C and a classifier 722C.

[0153] First, the first learning unit 74 acquires the subject image and the first prediction information from the storage unit 8C and inputs them to the generator 721C. The first learning unit 74 also inputs the second prediction information generated by the intervention effect prediction unit 78 to the generator 721C.

[0154] The generator 721C generates predicted image candidates (fake images) from the subject image, the first prediction information, and the second prediction information. The generator 721C may generate predicted image candidates by referring to real images included in the first training dataset 82.

[0155] Next, the first learning unit 74 inputs the predicted image candidates generated by the generator 721C and the real images included in the first learning data set 82 to the classifier 722C.

[0156] The classifier 722C receives as input the genuine images from the first training data set 82 and the predicted image candidates generated by the generator 721C, and outputs, for each image, the probability that it is a genuine image.

[0157] The first learning unit 74 calculates a classification error that indicates how accurate the probability output by the classifier 722C is. The first learning unit 74 iteratively improves the classifier 722C and the generator 721C using the backpropagation method.

[0158] The first learning unit 74 updates the weights and biases of the classifier 722C and the weights and biases of the generator 721C until the probability output by the classifier 722C meets a predetermined standard, thereby enabling the predicted image generation unit 72C to generate a predicted image that is indistinguishable from the real thing.

[0159] <Third Study Section 79> The third learning unit 79 controls the learning process for the neural network of the intervention effect prediction unit 78. Third teacher data 84 is used for this learning.

[0160] Here, the third teacher data 84 is data used in machine learning to generate an intervention effect prediction model. The third teacher data 84 includes third learning input data used as input data and third teacher data for calculating an error from the first prediction information output by the intervention effect prediction unit 78.

[0161] The third learning input data may include, for example, information indicating the time when the intervention was applied for each of multiple patients to whom the intervention was applied, patient images showing the target area of ​​each patient, and symptom information regarding the onset or progression of symptoms in the target area at the time the patient image of each patient was taken.

[0162] The third training data may include a patient image showing the target region of the patient taken at a time later (e.g., one year later) than the time when the patient image used in the third learning input data was taken, and symptom information regarding the onset or progression of symptoms in the target region of the patient. The third training data may include symptom information regarding the target region of each patient at the time when the patient image was taken. Here, the symptom information may include information regarding the onset time of the patient's disease or the progression of symptoms.

[0163] The third teacher data 84 may be information including condition information indicating the condition of the target area of ​​each patient, acquired at multiple time points in the past, and correlating the condition information for each patient with intervention information indicating the intervention applied to each patient, i.e., effect information. The third teacher data 84 may be time-series data in which patient images acquired at multiple time points from each of multiple patients to whom interventions were applied in the past are associated with information regarding the symptoms of the target area at the time the patient images were captured.

[0164] <Learning process for intervention effect prediction model> The learning process for generating an intervention effect prediction model to which a neural network is applied will be described below using Fig. 14 while also referring to Fig. 7. Fig. 14 is a flowchart showing an example of the flow of the learning process of the neural network included in the intervention effect prediction unit 78.

[0165] First, the third learning unit 79 acquires the third learning input data included in the third teacher data 84 from the memory unit 8C (step S41). The third learning input data includes, for example, (a) information indicating the time when the intervention was applied to each of multiple patients to whom the intervention was applied, (b) pixel data of the patient image showing the target area of ​​each patient, and (c) symptom information regarding the onset or progression of symptoms of the target area at the time when the patient image of each patient was captured.

[0166] Next, the third learning unit 79 determines a certain patient (step S42).

[0167] Next, the third learning unit 79 inputs the following into the input layer 781: (a) information indicating the time when an intervention was applied to a patient to whom the intervention was applied, (b) pixel data of a patient image showing the target area of ​​a patient, and (c) symptom information regarding the onset or progression of symptoms in the target area at the time the patient image of a patient was taken (step S43).

[0168] Next, the third learning unit 79 acquires output data, which is information indicating at least one of an intervention method for a certain patient and the effect of the intervention, from the output layer 782. This output data includes the same content as the third training data.

[0169] Next, the third learning unit 79 acquires the third training data included in the third training data 84. Then, the third learning unit 79 compares the acquired output data with information included in the third training data that indicates a method of intervention for a certain patient and the effect of the intervention, and calculates an error (step S45).

[0170] Thereafter, the third learning unit 79 adjusts the intervention effect prediction model so as to reduce the error (step S46).

[0171] Any known method can be applied to adjust the intervention effect prediction model. For example, backpropagation may be used as a method for adjusting the intervention effect prediction model. The intervention effect prediction model after adjustment becomes a new intervention effect prediction model, and the intervention effect prediction unit 78 uses the new intervention effect prediction model in subsequent calculations. In the stage of adjusting the intervention effect prediction model, parameters used by the intervention effect prediction unit 78 may be adjusted.

[0172] The parameters include, for example, parameters used in the input layer 781 and the output layer 782. Specifically, the parameters include weighting coefficients used in the LSTM layers of the input layer 781 and the output layer 782. The parameters may also include filter coefficients.

[0173] If the error is not within a predetermined range and if patient images of all patients included in the third teacher data 84 have not been input (NO in step S47), the third learning unit 79 changes the patient (step S48) and then returns to step S43 to repeat the learning process. If the error is within a predetermined range and if patient images of all patients included in the third teacher data 84 have been input (YES in step S47), the third learning unit 79 ends the learning process.

[0174] <Third training data 84> The third teacher data 84 is data used in machine learning to generate an intervention effect prediction model. The third teacher data 84 includes condition information indicating the condition of the target site of each patient, obtained at multiple past time points, and may also include effect information, i.e., information in which the condition information for each patient is associated with intervention information indicating the intervention applied to each patient. The third teacher data 84 includes third learning input data used as input data and third teacher data for calculating an error from the first prediction information output by the intervention effect prediction unit 78.

[0175] The third training input data may include, for example, (a) information indicating the time when an intervention was administered to a patient to whom the intervention was administered, (b) pixel data of a patient image showing the target site of the patient, and (c) symptom information regarding the onset or progression of symptoms in the target site at the time the patient image of the patient was captured. The image data used as the third training input data may be medical image data showing the target sites of each of multiple patients. The medical image data may include, for example, at least one of X-ray image data, CT image data, MRI image data, PET image data, and ultrasound image data of the target sites of each of multiple subjects.

[0176] The third training data may include a patient image of the target area of ​​the patient taken at a time later (e.g., one year later) than the time when the patient image used for the third learning input data was taken, condition information indicating the condition of the target area of ​​the patient, and symptom information related to the target area. The third training data may be time-series data in which patient images obtained at multiple time points from each of multiple patients to whom an intervention was previously administered are associated with information related to joint symptoms at the time the patient image was taken.

[0177] Specifically, the input data used during learning by the intervention effect prediction unit 78 is (a) a subject image showing the target area of ​​the subject at a certain time point A, which is included in the third learning input data. Based on the above-mentioned input data, the intervention effect prediction unit 78 outputs, as output data, first prediction information regarding the target area at time point B, a predetermined period (e.g., three years) after time point A. Specifically, the intervention effect prediction unit 78 outputs, as output data, information indicating, for example, the angle around the target area of ​​the subject at time point B, the degree of enlargement and shrinkage of the target area, the degree of wrinkles and blemishes on the target area, the time and degree of pain that will occur in the target area, and the time when invasive treatment will be required for the target area. The output data shown here is an example and is not limited to these.

[0178] Specifically, the input data used during learning by the intervention effect prediction unit 78 is information indicating the onset or progression of a disease related to a target area of ​​the patient at a certain time point B, which is included in the third teacher data 84, and information indicating a method of intervention, which is included in the intervention information. Based on the above-mentioned input data, the intervention effect prediction unit 78 outputs, as output data, information indicating a method of intervention for the subject and the effect of the intervention. Based on the above-mentioned input data, the intervention effect prediction unit 78 outputs, as output data, information indicating, for example, the degree to which symptoms of a disease related to a target area of ​​the patient are improved or the degree to which the progression of the symptoms is suppressed when the intervention at time point B is applied. More specifically, the intervention effect prediction unit 78 may output, as output data, the above-mentioned second prediction information.

[0179] (Processing performed by prediction systems 100 and 100a) The flow of processing performed by the prediction systems 100 and 100a will be described below with reference to Fig. 15. Fig. 15 is a flowchart showing an example of the flow of processing performed by the prediction systems 100 and 100a according to this embodiment.

[0180] First, the prediction information acquisition unit 71 acquires an image of a subject, while the basic information acquisition unit 76 acquires basic information (step S51: acquisition step).

[0181] Next, the prediction information generation unit 75B generates first prediction information in response to the input of the subject image and basic information, and outputs the first prediction information to the prediction information acquisition unit 71 and the intervention effect prediction unit 78 (step S52: first information prediction step).

[0182] Next, the intervention effect prediction unit 78 refers to the intervention information 85 and selects at least one of the intervention methods included in the intervention information 85 (step S53: intervention method selection step).

[0183] Furthermore, in response to the input of the first prediction information, the intervention effect prediction unit 78 generates second prediction information for the selected intervention method and outputs the second prediction information to the predicted image generation unit 72C and the output control unit 73C (step S54: intervention effect prediction step).

[0184] Next, the predicted image generation unit 72C generates a predicted image in response to the input of the target person image, the first prediction information, and the second prediction information, and outputs the predicted image to the terminal device 2 (step S55: predicted image generation step).

[0185] The prediction systems 100, 100a can output a visually easy-to-understand predicted image that shows how the condition of the target area at the second time point differs depending on whether or not an intervention is effective. Furthermore, since the predicted image is generated from a subject image, which is an image of the subject himself, it is a realistic image that is persuasive to the subject. Therefore, for example, if the predicted image is presented to the subject by a doctor in charge of the subject, it can effectively help the subject understand the need for intervention and increase the subject's motivation for the intervention.

[0186] [Software implementation example] The control blocks of the prediction devices 1, 1A, 1B, and 1C (particularly the control units 7, 7A, 7B, and 7C) may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

[0187] In the latter case, the prediction devices 1, 1A, 1B, and 1C each include a computer that executes instructions from a program, which is software that realizes each function. The computer includes, for example, one or more processors and a computer-readable recording medium storing the program. The processor in the computer reads and executes the program from the recording medium, thereby achieving the object of the present disclosure. The processor may be, for example, a central processing unit (CPU). The recording medium may be a "non-transitory tangible medium," such as a read-only memory (ROM), a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The device may also include a random access memory (RAM) for loading the program. The program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). One aspect of the present disclosure may also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

[0188] The invention according to the present disclosure has been described above based on the drawings and examples. However, the invention according to the present disclosure is not limited to the above-described embodiments. In other words, the invention according to the present disclosure can be modified in various ways within the scope of the present disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the invention according to the present disclosure. In other words, it should be noted that a person skilled in the art can easily make various modifications or corrections based on the present disclosure. It should also be noted that these modifications or corrections are included in the scope of the present disclosure.

[0189] [Other aspects] A prediction system according to one embodiment of the present disclosure includes a prediction information acquisition unit that acquires (a) a subject image showing a target area of ​​a subject at a first time point and (b) first prediction information regarding the target area at a second time point that is a predetermined period of time after the first time point, and a prediction image generation unit that generates and outputs a prediction image that predicts the state of the target area at the second time point from the first prediction information and the subject image.

[0190] Furthermore, a control method according to one aspect of the present disclosure is a control method for a prediction system, and includes a prediction information acquisition step of acquiring (a) a subject image showing a target area of ​​a subject at a first time point and (b) first prediction information regarding the target area at a second time point a predetermined period of time has elapsed since the first time point, and a prediction image generation step of generating and outputting a prediction image that predicts the state of the target area at the second time point from the first prediction information and the subject image, wherein the prediction system has a prediction image generation model that can generate the prediction image using the subject image and the first prediction information. [Explanation of symbols]

[0191] 1, 1A, 1B, 1C Prediction Device 2, 2a, 2b Terminal equipment 3, 3a, 3b 1st prediction information management device 4, 4a, 4b Subject image management device 5, 5a, 5b Medical Facilities 6. Communication Networks 7, 7A, 7B, 7C control section 8, 8B, 8C storage section 9, 9a, 9b Electronic medical record management device 71 Prediction information acquisition unit 72, 72C Prediction image generation unit 73, 73C Output control section 74 First Study Section 75, 75B Prediction information generation unit 76 Basic information acquisition department 77 Second Study Section 78 Intervention Effect Prediction Department 79 Third Study Section 81 Control Program 82 First training dataset 83 Second training dataset 84 Third training data 85 Intervention information 100, 100a Prediction System 721, 721C generator 722, 722C discriminator 751, 781 Input layer 752, 782 output layer

Claims

1. A prediction information acquisition unit that acquires (a) a subject image showing a target area of ​​a subject having a predetermined disease at a first time point, and (b) first prediction information regarding the onset or progression of the disease in the target area at a second time point after a predetermined period has elapsed since the first time point; a prediction image generation unit having a prediction image generation model that generates a prediction image that predicts the state of the target region at the second time point from the first prediction information and a converted image obtained by converting the file format of the target image into a predetermined file format; and a prediction information generating unit that generates the first prediction information, the predictive image generation model is generated by machine learning using first training data including first data of the same type as the subject image and second data of the same type as the first prediction information, and first teacher data including third data related to the second data; The first prediction information is generated based on at least the subject image. Prediction system.

2. The predicted image is an image that simulates at least a part of the image of the subject. The prediction system of claim 1 .

3. The subject image is an appearance image showing the target part. The prediction system according to claim 1 or 2.

4. The subject image is a medical image showing the target area. The prediction system according to any one of claims 1 to 3.

5. The medical image is at least one of an X-ray image, a CT image, an MRI image, a PET image, and an ultrasound image of the subject. The prediction system of claim 4 .

6. The predicted image is an image that predicts the effect of a disease occurring in the target site on the target site. The prediction system according to any one of claims 1 to 5.

7. the predictive image generation model is a neural network trained using a plurality of image data showing a target region as the first training data; The prediction system of claim 1 .

8. The first prediction information is information related to the shape and appearance of the target site related to the disease of the target site. The prediction system of claim 6 .

9. the prediction information generation unit has a prediction information generation model that generates the first prediction information from the subject image, the prediction information generation model is generated by machine learning using second learning data including the first data and second teacher data including fourth data related to the first data. The prediction system according to any one of claims 1 to 8.

10. The fourth data includes information regarding the onset time of a disease in the target area. The prediction system of claim 9 .

11. a basic information acquiring unit that acquires basic information including at least one of the subject's sex, age, height, and weight, and information indicating the state of the target part of the subject at the first time point; the prediction information generation model is capable of estimating the first prediction information from a subject image of the subject and the basic information of the subject; The prediction system according to claim 9 or 10.

12. the prediction information generation model is a neural network trained using patient information on patients having a disease in the target region as the second training data, The patient information includes condition information indicating the condition of the target site of each patient, which is acquired at multiple points in time in the past, and the condition information for each patient is associated with information indicating the points in time at which the condition information was acquired. The prediction system according to any one of claims 9 to 11.

13. An intervention effect prediction unit that receives the first prediction information as an input and outputs second prediction information indicating a method of intervention for the subject and an effect of the intervention, The prediction system according to any one of claims 1 to 12.

14. the intervention effect prediction unit has, as an intervention effect prediction model, a neural network trained using effect information as training data; The effect information includes condition information indicating the condition of the target site of each patient, which is acquired at multiple time points in the past, and is information in which the condition information for each patient is associated with intervention information indicating an intervention applied to each patient. The prediction system of claim 13.

15. The method of intervention includes at least one of dietary therapy, exercise therapy, drug therapy, orthotic therapy, rehabilitation, and surgical therapy; The prediction system according to claim 13 or 14.

16. the first prediction information includes a prediction regarding a time when a symptom likely to occur in the target area will occur; The prediction system according to any one of claims 1 to 15.

17. 1. A method for controlling a prediction system, comprising: (a) a subject image showing a target area of ​​the subject having a predetermined disease at a first time point; and (b) a prediction information acquisition step of acquiring first prediction information regarding the onset or progression of the disease in the target area at a second time point after a predetermined period has elapsed since the first time point. a predicted image generating step of generating and outputting a predicted image that predicts the state of the target region at the second time point from the first prediction information and a converted image obtained by converting the file format of the target image into a predetermined file format; a prediction information generating step of generating the first prediction information, the prediction system includes a predicted image generation model capable of generating the predicted image using the subject image and the first prediction information; A control method, wherein the first prediction information is generated based on at least the subject image.

18. 17. A control program for causing a computer to function as the prediction system according to claim 1, the control program causing a computer to function as the prediction information acquisition unit, the prediction image generation unit, and the prediction information generation unit.

Images