Information processing system, prediction device, information processing method, control program, and recording medium

Abstract

This information processing system is provided with a prediction unit that uses a prediction model to output prediction information from a first image and first data, the first image showing at least a part of a first subject. The prediction model is generated by machine learning using, as an explanatory variable, a third image and second data, the third image showing at least a part of a second subject, and using, as an objective variable, abnormality information relating to an abnormality that has occurred in a bone of the second subject at a second time point, which is a time point different from a first time point at which the third image was captured. The prediction information is information indicating the possibility of an abnormality occurring in a bone of the first subject.

Description

Information processing system, prediction device, information processing method, control program, and recording medium

[0001] The present disclosure relates to an information processing system, a prediction device, an information processing method, a control program, and a recording medium.

[0002] Conventionally, there has been known a technique for estimating disease-related information from medical images using a neural network. For example, Patent Literature 1 discloses a configuration for estimating whether a patient has osteoporosis based on an X-ray image of the patient.

[0003] Japanese Patent Application Publication No. 2023-143875

[0004] An information processing system according to one aspect of the present disclosure includes a prediction unit that outputs prediction information using a prediction model based on a first image and first data showing at least a portion of a first subject. The prediction model is generated by machine learning using a third image and second data showing at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point different from a first time point when the third image was captured as a dependent variable. The prediction information is information indicating the possibility of an abnormality occurring in the bones of the first subject.

[0005] An information processing method according to one aspect of the present disclosure is an information processing method executed by one or more computers, and includes a prediction step of outputting predicted information using a prediction model from a first image and first data showing at least a portion of a first subject. The prediction model is generated by machine learning using a third image and second data showing at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality occurring in the bones of the second subject at a second time point different from a first time point when the third image was captured as a dependent variable. The predicted information is information indicating a possibility of an abnormality occurring in the bones of the first subject.

[0006] 1 is a block diagram showing an example of the configuration of an information processing system according to a first embodiment of the present disclosure. FIG. 1 is a block diagram for explaining the operation of a prediction model according to the first embodiment. FIG. 2 is a flowchart showing an example of a learning process flow by a learning unit of a prediction device according to the first embodiment. FIG. 3 is a flowchart showing an example of a prediction process flow by a prediction unit of a prediction device according to the first embodiment. FIG. 4 is a diagram showing an example of a first image of a first subject according to the first embodiment. FIG. 5 is a diagram showing an example of segmentation of a second image according to the first embodiment. FIG. 6 is a block diagram showing an example of the configuration of an information processing system according to the second embodiment. FIG. 7 is a block diagram for explaining the operation of an estimation model according to the second embodiment. FIG. 8 is a block diagram for explaining the operation of a prediction model according to the second embodiment. FIG. 9 is a flowchart showing an example of a learning process by a learning unit according to the second embodiment. FIG. 10 is a diagram showing a plain X-ray image of the chest of a first subject according to the second embodiment. FIG. 11 is a flowchart showing an example of a prediction process flow by a prediction device according to the second embodiment. FIG. 12 is a diagram showing a prediction result of a subject's fracture risk by a prediction device according to the second embodiment and the effects of each countermeasure.

[0007] Factors that lead to fractures include a decrease in bone strength and increased stress on bones due to muscle atrophy. In order to accurately estimate the possibility of a fracture, it is desirable to consider not only bone strength but also the condition of the muscles. However, there is room for improvement in the accuracy of techniques for estimating the occurrence of fractures from medical images.

[0008] One aspect of the present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an information processing system, an information processing method, a prediction device, a control program, and a recording medium that are capable of improving the prediction accuracy of predicting information regarding abnormalities from medical images.

[0009] According to one aspect of the present disclosure, it is possible to improve the prediction accuracy of predicting information about an abnormality from a medical image.

[0010] First Embodiment An information processing system 1 according to a first embodiment of the present disclosure will be described below with reference to FIGS.

[0011] [Configuration of Information Processing System] First, the configuration of the information processing system 1 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of the information processing system 1 in embodiment 1. Fig. 1 shows the information processing system 1 including a prediction device 10, an image management device 40, an electronic medical record management device 50, and a presentation device 60. Note that the configuration of the information processing system 1 is not limited to the configuration shown in Fig. 1.

[0012] The prediction device 10 is a device that acquires a first image G1 and first data of a first subject, which is a target for predicting information about an abnormality, and outputs prediction information from the acquired first image G1 and first data using a prediction model 32. In the first embodiment, the first data is, for example, a second image G2 of the first subject (see FIG. 2 ).

[0013] Here, the first subject is, for example, a human. The first subject may be a non-human animal, such as a dog, cat, or horse. The first image G1 may depict at least some bones in a predetermined region of the first subject. The second image G2 may depict muscles in a region of the first subject corresponding to the predetermined region. The second image G2 may depict the same region of the first subject as the region corresponding to the predetermined region, or may depict a region of the first subject different from the region corresponding to the predetermined region.

[0014] The first image G1 is, for example, a plain X-ray image showing bones of a region including at least one of the head, neck, chest, lower back, temporomandibular joint, spinal intervertebral joint, hip joint, sacroiliac joint, knee joint, ankle joint, foot, toes, shoulder joint, acromioclavicular joint, elbow joint, wrist joint, hand, fingers, and temporomandibular joint of the first subject. The first image G1 may also show a region of the first subject other than a region where an abnormality such as a fracture is expected. The plain X-ray image may include, for example, a panoramic X-ray image used for dentistry. The panoramic X-ray image is, for example, an image that includes multiple teeth or all teeth.

[0015] The first image G1 may be, for example, an image captured at a medical facility at a third time point. The first image G1 includes at least one of a front image of a predetermined region from the front (e.g., an image obtained by irradiating the predetermined region with X-rays in the front-to-back direction) and a side image of the predetermined region from the side (e.g., an image obtained by irradiating the predetermined region with X-rays in the left-to-right direction).

[0016] The first image G1 is not limited to a plain X-ray image, but may be any medical image containing information about bones, such as a computed tomography (CT) image, a magnetic resonance imaging (MRI) image, an image obtained by dual energy x-ray absorptiometry (DXA) method, an image obtained by dual energy subtraction (DES), an ultrasound image, etc. The first image G1 may be an image that includes a phantom or an image that does not include a phantom.

[0017] The second image G2 is an echo image showing muscles in regions corresponding to regions such as the head, chest, waist, feet, and hands of the first subject (see FIG. 6). The second image G2 may show only one region of the first subject, or may show multiple regions. The second image G2 may be a still image or a moving image.

[0018] Furthermore, the second image G2 is not limited to an echo image, and may be any medical image containing information about muscles, such as a CT image, an MRI image, an image obtained by DXA, an image obtained by DES, or an ultrasound image. That is, the second image G2 may be of a different or the same type of image as the first image G1. Furthermore, the second image G2 may be an image captured using a medium with a different wavelength from that of the first image G1.

[0019] The second image G2 may be, for example, an image captured at a medical facility at a third time point. The difference in capture time between the first image G1 and the second image G2 may be within a predetermined period, such as two weeks, one month, six months, or one year. The third time point may be set based on the date on which the first image G1 was captured, or may be set based on the date on which the second image G2 was captured. The reference date may be either the later or earlier of the capture dates of the first image G1 and the second image G2.

[0020] The second image G2 also includes at least one of a front image showing a specified area from the front (for example, an image obtained by irradiating a specified area with X-rays in the front-to-back direction) and a side image showing a specified area from the side (for example, an image obtained by irradiating a specified area with X-rays in the left-to-right direction).

[0021] If the first image G1 or the second image G2 is a CT image, information about the trabeculae based on a three-dimensionally constructed image may be used, or information about the trabeculae based on a two-dimensionally captured image may be used. Also, for example, at least one of a three-dimensional image, a cross-sectional image in a direction perpendicular to the body axis connecting the head and legs (e.g., horizontal section), and a cross-sectional image in a direction parallel to the body axis (e.g., sagittal section or coronal section) may be used.

[0022] The prediction information output by the prediction device 10 is information indicating the possibility that an abnormality will occur in the bone of the first subject captured in the first image G1 at the fourth time point. In this embodiment, the prediction information is a fracture risk indicating the possibility that a fracture will occur in the bone of the first subject at the fourth time point.

[0023] Fractures are an example of musculoskeletal diseases, and fragility fractures are assumed as fractures. Musculoskeletal diseases also include osteoporosis, osteoarthritis, spondylosis, neurological disorders, sarcopenia, etc.

[0024] The fourth time point is a time point different from the third time point. The third time point is the time point when the first image G1 was captured. The fourth time point refers to any time point in the future or the past of the third time point. The fourth time point may include multiple time points, such as one year before, one year after, five years after, ten years after, and thirty years after the third time point.

[0025] The image management device 40 is a computer that functions as a server for managing the third and fourth images. The third image is a plain X-ray image showing the bones of a predetermined part of the second subject. The fourth image is an echo image showing the muscles of a part of the second subject corresponding to the predetermined part.

[0026] The fourth image may depict a region of the second subject that is different from the region corresponding to the predetermined region. The third image and the fourth image may be stored in separate image management devices. The prediction device 10 may acquire the third image and the fourth image from the imaging device without using the image management device 40.

[0027] Furthermore, the third image is not limited to a simple X-ray image, but may be any medical image containing information about bones, such as a CT image, an MRI image, an image obtained by DXA, an image obtained by DES, or an ultrasound image. The fourth image is not limited to an echo image, but may be any medical image containing information about muscles, such as a CT image, an MRI image, an image obtained by DXA, an image obtained by DES, or an ultrasound image. In other words, the fourth image may be of a different or the same type of image as the third image. The third and fourth images may be of a different or the same type of combination of the first image G1 and the second image G2.

[0028] The third image may be, for example, an image captured at a medical facility at a first time point. The difference in capture time between the third image and the fourth image may be within a predetermined period, such as two weeks, one month, six months, or one year. The first time point may be set based on the date the third image was captured, or may be set based on the date the fourth image was captured. The reference date may be either the later or earlier of the capture dates of the third and fourth images.

[0029] The electronic medical record management device 50 is a computer that functions as a server for managing electronic medical record information of a first subject who has undergone a medical examination or test at a medical facility, etc. The image management device 40 and the electronic medical record management device 50 are connected to the acquisition unit 21 of the prediction device 10.

[0030] The electronic medical record information includes attribute information of the first subject, which may include at least one of the following: the first subject's age, sex, height, weight, muscle quality, race, lifestyle information, medication information, occupational information, blood test information, urine test information, saliva test information, information on existing diseases, medical history, medical history of the first subject's family, surgery information, genetic information, birth information, menopausal information, items from the Fracture Risk Assessment Tool (FRAX (registered trademark)), and information on estimated menopause based on hormone information.

[0031] The birth information includes at least one of whether or not a person has given birth, the number of children born, etc. The lifestyle habits may include, for example, sleep duration, wake-up time, sleep duration, daily exercise amount, meal contents, meal times, meal durations, and blood glucose levels. The meal contents may include, for example, at least one of the name of a dish, the ingredients consumed, and the intake amount. The meal contents may be, for example, an estimated intake of at least one of calcium, vitamin B, vitamin D, and vitamin K. The blood glucose level may be, for example, a designated value estimated from parameters acquired by a wearable device. The medication information may include, for example, information such as the name of a medication, the amount taken, and the duration of medication. The information regarding medications taken may include information regarding the steroid drug being used. The blood test information may be, for example, information regarding the results of at least one of a biochemical test, a glucose metabolism test, and an endocrine system test.

[0032] The presentation device 60 is a device for presenting information output by the prediction device 10. The presentation device 60 is a computer used by medical personnel such as doctors belonging to a medical facility. The presentation device 60 is, for example, a personal computer having a liquid crystal display or an organic EL display, a tablet terminal, a smartphone, etc. The presentation device 60 is controlled by the presentation control unit 26 to present the fracture risk and the like as prediction information. The presentation device 60 may also be a device that prints the prediction information on paper or the like and outputs it.

[0033] [Configuration of Prediction Device] Next, the configuration of the prediction device 10 will be described in detail with reference to Fig. 1. As shown in Fig. 1, the prediction device 10 includes a control unit 2 and a storage unit 3. The control unit 2 has, for example, a CPU (Central Processing Unit), and manages the operation of the prediction device 10 by comprehensively controlling each unit of the prediction device 10.

[0034] The control unit 2 of the prediction device 10 includes an acquisition unit 21, an analysis unit 22, a correction unit 23, a learning unit 24, a prediction unit 25, and a presentation control unit 26. The control unit 2 and the storage unit 3 are electrically connected to each other.

[0035] The acquiring unit 21 acquires a first image G1 and a second image G2 of the first subject from the image management device 40. The acquiring unit 21 also acquires a third image and a fourth image of the second subject from the image management device 40. The acquiring unit 21 may acquire the first image G1 and the second image G2 input via an input device (not shown). Furthermore, if attribute information is added to the acquired first image G1 and second image G2, the acquiring unit 21 may extract the attribute information from the first image G1 and the second image G2.

[0036] Here, the third and fourth images of the plurality of people are stored in the storage unit 3 as learning data 33. Furthermore, abnormality information regarding bone abnormalities that occurred in the plurality of people at the second time point is stored in the storage unit 3 as training data 34.

[0037] The analysis unit 22 performs region division by segmenting the second image G2, identifying which bone, muscle, or other region each pixel in the second image G2 corresponds to. Examples of methods that can be used for segmentation include a convolutional neural network (CNN), a fully convolutional network (FCN), a U-Net, and a V-Net. The analysis unit 22 identifies the soft tissue region. The analysis unit 22 analyzes information including at least one of the amount, thickness, amount of atrophy, and flexibility of the muscle and fat of the first subject.

[0038] The correction unit 23 performs a predetermined correction on the first image G1. Specifically, the correction unit 23 performs the correction to remove the soft tissue area identified by the analysis unit 22 from the first image G1. Here, soft tissue refers to tissue other than bone, such as muscle and fat. Note that the third image described above may also be corrected by the correction unit 23 in the same way as the first image G1.

[0039] The learning unit 24 performs a learning process to generate the prediction model 32. Note that if the prediction model 32 generated by another device is stored in advance in the storage unit 3, the learning unit 24 may be omitted.

[0040] The storage unit 3 is a computer-readable non-transitory recording medium that stores a control program 31. The storage unit 3 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

[0041] The control unit 2 controls the prediction device 10 by executing the control program 31. That is, the control program 31 is a control program for causing a computer to function as the information processing system 1, and is for causing a computer to function as the prediction unit 25.

[0042] In addition to the control program 31, the storage unit 3 also stores a prediction model 32. The prediction model 32 has AI (artificial intelligence). The prediction model 32 is generated by machine learning using the third image captured at the first time point and the second data as explanatory variables and abnormality information related to a fracture that occurred in the bone of the second subject at the second time point as a response variable. The storage unit 3 also stores the above-mentioned training data 33 and teacher data 34. In the first embodiment, the second data is, for example, the above-mentioned fourth image.

[0043] The prediction model 32 may be generated by machine learning using the third image, the fourth image, bone information obtained by inputting the third image into the first estimation model, and muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and abnormality information related to a fracture that occurred in the bone of the second subject at the second time point as a dependent variable.The prediction unit 25 may output prediction information using the prediction model from the first image G1, the second image G2, bone information obtained by inputting the first image G1 into the first estimation model, and muscle information obtained by inputting the second image G2 into the second estimation model.

[0044] [Operation of Prediction Model] Next, the operation of the prediction model will be described with reference to Fig. 2. Fig. 2 is a block diagram for explaining the operation of the prediction model 32 in embodiment 1. The prediction model 32 is, for example, a convolutional neural network (CNN). Note that the prediction model 32 may be configured using a neural network other than a convolutional neural network.

[0045] 2, the prediction model 32 includes, for example, an input layer 32a, a hidden layer 32b, and an output layer 32c. The prediction model 32 is generated by machine learning using the third image and the fourth image captured at the first time point as explanatory variables and abnormality information related to an abnormality occurring in the bones of the second subject at the second time point as a response variable.

[0046] The first time point is the time point when the third image is captured. The second time point is a time point different from the first time point. The second time point refers to, for example, any time point in the future or the past of the first time point when the third image is captured. The second time point may include multiple time points, such as one year before, one year after, five years after, ten years after, and thirty years after the first time point. Furthermore, the period between the first time point and the second time point may be the same length as or different from the period between the third time point and the fourth time point. Furthermore, the period between the third time point and the fourth time point may be shorter or longer than the period between the first time point and the second time point.

[0047] The second time point may be the same as the third time point described above. That is, for example, the prediction model 32 may be trained to predict the risk of fracture occurring five years later using a third image captured five years ago. In this case, the first time point is five years ago, the second and third time points are the present, and the fourth time point is five years later.

[0048] In the prediction model 32, the first image G1 and the second image G2 are input to the input layer 32a, and a fracture risk Y indicating the possibility of a fracture occurring in the bone of the first subject at the fourth time point is output.

[0049] The prediction model 32 may output information indicating the possibility of osteoporosis occurring in the bones of the first subject at the fourth time point. The possibility of osteoporosis may be classified based on, for example, at least one of the presence or absence of a fracture, the possibility of a fracture, and a change in bone mineral density. The possibility of osteoporosis may include "no osteoporosis," "suspected osteoporosis," or "possible osteoporosis." More specifically, the possibility of osteoporosis may be indicated when there is no disease that reduces bone mass, secondary osteoporosis is not observed, and a fracture is present or is highly likely to occur. Osteoporosis may also be determined when the YAM indicated by the bone mineral density estimate, which is information indicating the bone mineral density of the first subject, is less than 80% and the measurement results indicate a fracture other than the vertebral body or proximal femur. Osteoporosis may also be determined when the YAM indicated by the bone mineral density estimate is 70% or less.

[0050] [Learning Process Flow] Next, the flow of the learning process by the learning unit 24 will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the flow of the learning process by the learning unit 24 of the prediction device 10. The learning unit 24 executes the learning process shown in Fig. 3 and stores the prediction model 32 in the storage unit 3 before the prediction process by the prediction unit 25, which will be described later.

[0051] In the flowchart of FIG. 3 , first, the learning unit 24 acquires a third image of the second subject via the acquisition unit 21 (S1). Here, the second subject is, for example, multiple people. Note that the second subject may be a non-human subject, such as an animal such as a dog, cat, or horse. The second subject may be the same species as the first subject, or a different species. The second subject does not need to be multiple people, and may be the same person. The third image is a simple X-ray image of a predetermined bone region of the second subject captured at a first time point. Note that multiple data of the same person captured at different times may be used as the third image.

[0052] In S1, the learning unit 24 acquires attribute information of each second subject from the electronic medical record management device 50, and associates the attribute information with each second image G2.

[0053] After S1, the learning unit 24 acquires a fourth image of the second subject via the acquisition unit 21 (S2). The fourth image may be an echo image of the muscles of a region of the second subject corresponding to the predetermined region imaged at the first time point. The third image may at least capture the same region as the region imaged in the fourth image. The third image may also capture a region different from the region imaged in the fourth image. Note that S1 and S2 may be performed in reverse order or at the same timing.

[0054] The third image may be at least one of a CT image, an MRI image, an image obtained by a DXA method, an image obtained by a DES method, and an ultrasound image, and the fourth image may be at least one of a CT image, an MRI image, an image obtained by a DXA method, an image obtained by a DES method, and an ultrasound image.

[0055] After S2, the learning unit 24 acquires abnormality information regarding an abnormality occurring in the bones of the second subject at the second time point via the acquisition unit 21 (S3). Here, the abnormality may be a musculoskeletal disorder, such as a fracture. That is, the information is regarding a fracture occurring in the second subject at the second time point. Note that abnormalities include, in addition to fractures, osteoporosis, osteoarthritis, spondylosis deformans, neuropathy, sarcopenia, and the like.

[0056] After S3, the learning unit 24 generates a prediction model 32 by machine learning using the third image and the fourth image as explanatory variables and the anomaly information as a target variable. After S4, the learning unit 24 stores the generated prediction model 32 in the storage unit 3 (S5). This completes the learning process by the learning unit 24.

[0057] [Flow of Prediction Processing] Next, the flow of prediction processing by the prediction device 10 will be described with reference to Fig. 4 to Fig. 6. Fig. 4 is a flowchart showing an example of the flow of prediction processing by the prediction device 10. Fig. 5 is a diagram showing an example of a first image G1 of a first subject.

[0058] Below, we will explain the case where, in a medical facility or the like, an image is taken of the chest bones of a first subject as a first image G1, and an echo image is taken of the back muscles of the first subject as a second image G2, as shown in Figure 5.

[0059] 4, first, the acquisition unit 21 acquires a first image G1 of the first subject from the image management device 40 (S11). The first image G1 may be, for example, a plain X-ray image showing a chest bone B of the first subject, as shown in FIG.

[0060] Next, the acquisition unit 21 acquires a second image G2 of the first subject from the image management device 40 (S12). The second image G2 may be an echo image showing muscles in a region corresponding to the chest of the first subject. Note that S11 and S12 may be performed in the reverse order or at the same timing.

[0061] After S12, the analysis unit 22 segments the second image G2 and analyzes information about the muscles and fat of the first subject (S13). In S13, the analysis unit 22 segments the second image G2, i.e., divides multiple types of muscles, fat, etc. appearing in the second image G2 into regions.

[0062] Fig. 6 is a diagram showing an example of segmentation of the second image G2. Fig. 6 shows an echo image of the back of the first subject. In the example shown in Fig. 6, the back of the first subject is divided into regions of subcutaneous fat F, first muscle M1, second muscle M2, and bone B. As a segmentation method, the back may be divided into regions of muscle that combines multiple types of muscle, subcutaneous fat F, and bone B.

[0063] The analysis unit 22 can analyze the volume of the first muscle M1 by determining the thickness of the first muscle M1, as shown by the open arrow in FIG. 6 . The analysis unit 22 can also analyze the fat volume, muscle atrophy, flexibility, and the like of the first subject. For example, the analysis unit 22 can calculate the fat volume of the first subject by segmenting the second image G2 to calculate the fat area or by determining the brightness of the echo image. The analysis unit 22 can also determine the volume of muscle atrophy of the first subject by comparing the average muscle thickness measurements of people of the same age as the first subject with the muscle thickness of the first subject. The analysis unit 22 can also determine the flexibility of the first subject's muscle by capturing a video of the echo image of the first subject and analyzing the muscle movement.

[0064] After S13, the correction unit 23 performs a predetermined correction to remove the soft tissue, i.e., tissue regions other than bone, identified by the analysis unit 22 from the first image G1 (S14). Specifically, the correction unit 23 removes muscle and / or fat, etc., other than bone, identified by the analysis unit 22, from the first image G1. The second image G2 preferably shows a region corresponding to the first image G1.

[0065] After S14, the prediction unit 25 reads out the prediction model 32 from the memory unit 3, inputs the first image G1 and the second image G2 corrected by the correction unit 23 into the input layer 32a of the prediction model 32, and outputs the fracture risk Y from the output layer 32c (S15: prediction step).

[0066] Furthermore, in S15, the prediction unit 25 may output, as prediction information, influence levels indicating the degree of influence that each of the first image G1 and the second image G2 has on the fracture of the first subject from the first image G1 and the second image G2 using the prediction model 32. For example, information may be output indicating that the influence level of the bone condition shown in the first image G1 is 70% and the influence level of the muscle condition shown in the second image G2 is 30%. The prediction model 32 is generated by machine learning using the third image and the fourth image as explanatory variables and the influence level indicating the degree of influence that each of the first image G1 and the second image G2 has on the fracture of the second subject at the second time point as a response variable.

[0067] Furthermore, in S15, the prediction unit 25 may output a predicted fracture time, which is a time when a fracture is likely to occur, using the prediction model 32. In this case, the prediction model 32 is generated by machine learning using the third image and the fourth image as explanatory variables and the time when a fracture occurs in the bone of the second subject as a target variable. The predicted fracture time may be in years, for example, 5 years or 10 years, or may be in months, for example, 5 years and 6 months or 10 years and 6 months.

[0068] The fracture risk Y, the degree of influence, the predicted fracture time, etc. output by the prediction unit 25 are transmitted to the presentation control unit 26. The presentation control unit 26 then presents prediction information including at least one of the fracture risk Y, the degree of influence, the predicted fracture time, etc. on the presentation device 60 (S16).

[0069] In S16, for example, the presentation control unit 26 presents a message such as "Fracture risk Y will be 80% or more in 10 years" to the presentation device 60. The presentation control unit 26 may also display a diagram showing the fracture risk Y at multiple points in time. The presentation control unit 26 may also present a graph showing the progress of the fracture risk Y to the presentation device 60. The presentation control unit 26 may also present the fracture risk Y, which is the estimated result, and the actual measurement result to the presentation device 60. This completes the prediction process by the prediction device 10 shown in FIG. 4.

[0070] In the information processing system 1 in the first embodiment described above, the prediction unit 25 outputs the fracture risk Y, the influence degree, the predicted fracture time, etc. of the first subject as prediction information from the first image G1 showing the bones of the first subject and the second image G2 showing the muscles of the first subject, using the prediction model 32. That is, the prediction unit 25 predicts the fracture risk Y, the influence degree, the predicted fracture time, etc. using the prediction model 32, using information about the bones of the first subject as input information, as well as information about the muscles that support the bones.

[0071] According to the above configuration, the prediction unit 25 can predict with high accuracy the fracture risk Y of the first subject developing a fracture, the degree of influence, the predicted fracture timing, etc. As a result, for example, a doctor at a medical facility can use the output results of the information processing system 1, such as the fracture risk Y, to diagnose the first subject, who is a patient, and can more appropriately diagnose the patient. Furthermore, even a doctor who does not specialize in orthopedics can diagnose the patient with an accuracy close to that of an orthopedic surgeon by referring to the output results of the information processing system 1.

[0072] Furthermore, in S14 of FIG. 4, the correction unit 23 performs correction to remove tissue areas other than bone from the first image G1, thereby improving the accuracy of prediction of the fracture risk Y by the prediction unit 25.

[0073] [Embodiment 2] Next, an information processing system 1A according to embodiment 2 of the present disclosure will be described with reference to Figures 7 to 13. For ease of explanation, components having the same functions as those described in embodiment 1 above will be denoted by the same reference numerals, and their description will not be repeated.

[0074] [Configuration of Information Processing System] The configuration of the information processing system 1A of the second embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the configuration of the information processing system 1A. Fig. 7 shows the information processing system 1A including a prediction device 10A, an image management device 40, an electronic medical record management device 50, and a presentation device 60, but the configuration of the information processing system 1A is not limited to the configuration shown in Fig. 7.

[0075] The prediction device 10A is a device that acquires a first image G1a of a first subject that is an object to be predicted, and outputs prediction information from the acquired first image using a prediction model 32A.

[0076] Here, the first subject is, for example, a human. Note that the first subject may be a non-human animal, such as a dog, cat, or horse. The first image G1a may include either the bones or muscles of the first subject, or may include at least a portion of the bones and muscles of the first subject.

[0077] The first image G1a may be a medical image. The first image G1a is, for example, a plain X-ray image showing tissues of a region including at least one of the head, neck, chest, lower back, temporomandibular joint, spinal intervertebral joint, hip joint, sacroiliac joint, knee joint, ankle joint, foot, toes, shoulder joint, acromioclavicular joint, elbow joint, wrist joint, hand, fingers, and temporomandibular joint of the first subject. The tissues are, for example, bones and muscles. Note that the tissues may be either bones or muscles. The plain X-ray image may include, for example, a panoramic X-ray image used for dentistry. The panoramic X-ray image is an image that includes multiple teeth, for example, all teeth.

[0078] The first image G1a is captured at a third time point. The first image G1a includes at least one of a front image captured from the front of the first subject, e.g., an image obtained by irradiating the target region with X-rays in the front-back direction, and a lateral image captured from the side, e.g., an image obtained by irradiating the target region with X-rays in the left-right direction. The first image G1a may be, for example, a front chest X-ray image including a person's chest or a front lumbar X-ray image including a person's lumbar region. The chest X-ray image is, for example, an image showing at least one of the ribs, clavicle, and sternum. The lumbar X-ray image is, for example, an image showing at least one of the lumbar vertebrae, pelvis, and femur. The first image G1a is not limited to the chest and lumbar region, and may also be, for example, an image showing the teeth, jaw, arm, hand, shoulder joint, knee joint, heel, skull, or foot bone.

[0079] The first image G1a is not limited to a simple X-ray image, but may be any image containing information about at least one of bones and muscles. Alternatively, the first image G1a may be, for example, a magnetic resonance imaging (MRI) image, a computed tomography (CT) image, a positron emission tomography (PET) image, or an ultrasound image. When the first image G1a is a CT image, information about the trabeculae based on a three-dimensionally constructed image may be used, or information about the trabeculae based on a two-dimensionally captured image may be used. When the first image G1a is a CT image, for example, at least one of a three-dimensional image, a cross-sectional image perpendicular to the body axis connecting the head and legs (e.g., a horizontal section), and a cross-sectional image parallel to the body axis (e.g., a sagittal section or a coronal section) may be used. The first image G1a may or may not include bones.

[0080] The prediction information output by the prediction device 10A is information indicating the possibility of an abnormality, such as a musculoskeletal disorder, occurring in the area captured in the first image G1a. In the second embodiment, the prediction information is a fracture risk indicating the possibility of a fracture occurring in the bone of the first subject at a fourth time point that is different from the third time point. Note that a fracture is an example of a musculoskeletal disorder, and a fragility fracture is assumed as the fracture. In addition to fractures, musculoskeletal disorders include, for example, osteoporosis, osteoarthritis, spondylosis deformans, neuropathy, sarcopenia, etc.

[0081] The first image G1a may be, for example, an image taken at a medical facility. The first image G1a depicts a region of the first subject where an abnormality such as a fracture is suspected. Note that the first image G1a may also depict a region of the first subject other than the region where an abnormality such as a fracture is suspected.

[0082] The fourth time point refers to any time point in the future or the past of the third time point. The fourth time point may include multiple time points, such as one year, five years, ten years, and thirty years after the third time point.

[0083] The image management device 40 is a computer that functions as a server for managing the first image G1a and the third image captured in a medical facility. The first image G1a may be a plain X-ray image showing at least a portion of the bones and muscles of the first subject. The third image may be a medical image. The third image may be a plain X-ray image showing the tissues of the second subject. The first image G1a and the third image may be stored in separate image management devices. The third image may be the same type of image as the first image G1a, or may be a different type of image. For example, the first image G1a may be a plain X-ray image, and the third image may be a CT image.

[0084] The electronic medical record management device 50 is a computer that functions as a server for managing electronic medical record information of a first subject who has undergone a medical examination and / or test at a medical facility, etc. The image management device 40 and the electronic medical record management device 50 are connected to the acquisition unit 21 of the prediction device 10A.

[0085] The electronic medical record information includes attribute information of the first subject, which may include at least one of the following: the first subject's age, sex, height, weight, race, information about lifestyle habits, medication information, occupational information, blood test information, urine test information, saliva test information, information about existing diseases, medical history, medical history of the first subject's family, genetic information, birth information, menopausal information, items from the Fracture Risk Assessment Tool (FRAX (registered trademark)), and information about estimated menopause based on hormone information.

[0086] The birth information may include at least one of whether or not the subject has given birth and the number of births. The lifestyle habits may include, for example, sleep duration, wake-up time, sleep duration, daily exercise amount, dietary content, meal times, meal durations, and blood glucose levels. The dietary content may include, for example, at least one of the name of a dish, ingested ingredients, and intake amount. The dietary content may be, for example, an estimated intake of at least one of calcium, vitamin B, vitamin D, and vitamin K. The blood glucose level may be, for example, a designated value estimated from parameters acquired by a wearable device. The medication information may include, for example, information such as the name of a medication, the amount taken, and the duration of medication. The information about the medication may include information about the steroid drug being used. The blood test information may be, for example, information about at least one of a biochemical test, a glucose metabolism test, and an endocrine system test. The information processing system 1A may acquire attribute information of the first subject from the electronic medical record management device 50. Alternatively, if attribute information is linked to the first image G1a, the attribute information may be acquired from the first image G1a.

[0087] The presentation device 60 is a device for presenting information output by the prediction device 10A. The presentation device 60 is, for example, a liquid crystal display or an organic EL display. The presentation device 60 is controlled by the presentation control unit 26 to present numerical values ​​such as an estimated bone density value, an estimated bone quality value, an estimated muscle mass value, and a fracture risk, as well as support information for the first subject. The presentation device 60 may also be a device that prints the prediction information on paper or the like and outputs it.

[0088] [Configuration of Prediction Device] Next, the configuration of the prediction device 10A will be described in detail with reference to Fig. 7. As shown in Fig. 7, the prediction device 10A includes a control unit 2 and a storage unit 3. The control unit 2 has, for example, a CPU (Central Processing Unit) and manages the operation of the prediction device 10A by comprehensively controlling each unit of the prediction device 10A. The control unit 2 includes an acquisition unit 21, a learning unit 24, a prediction unit 25, an estimation unit 27, and a presentation control unit 26. The control unit 2 and the storage unit 3 are electrically connected to each other.

[0089] The acquisition unit 21 acquires a first image G1a that shows at least a portion of the bones and muscles of the first subject at the third time point from the image management device 40. The acquisition unit 21 also acquires attribute information of the first subject from the electronic medical record management device 50. Note that the acquisition unit 21 may acquire the first image G1a input by an input device (not shown).

[0090] The learning unit 24 controls a learning process for generating the estimation model 35 and the prediction model 32A. The estimation unit 27 outputs first estimated information on the bones of the first subject from the first image G1a showing at least a part of the bones and / or muscles of the first subject, using at least one of a first estimation model 351, a second estimation model 352, and a third estimation model 353, which will be described later.

[0091] The first estimation information includes at least one of a bone density estimation value, which is information indicating the bone density of the bones of the first subject output from the first estimation model 351, a bone quality estimation value, which is information indicating the bone quality of the first subject output from the second estimation model 352, and a muscle mass estimation value, which is information indicating the muscle mass of the first subject output from the third estimation model 353.

[0092] The storage unit 3 is a computer-readable non-transitory recording medium that stores a control program 31. The storage unit 3 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

[0093] The control unit 2 controls the prediction device 10A by executing a control program 31. That is, the control program 31 is a control program for causing a computer to function as the information processing system 1A, and is for causing the computer to function as the prediction unit 25 and the estimation unit 27.

[0094] In addition to the control program 31, the storage unit 3 also stores a prediction model 32A and an estimation model 35. The prediction model 32A and the estimation model 35 have AI (Artificial Intelligence). Specifically, the estimation model 35 has at least one AI from a first estimation model 351, a second estimation model 352, and a third estimation model 353.

[0095] The prediction model 32A was generated by machine learning using the third image and information about the bones of the second subject as explanatory variables and abnormality information about abnormalities that occurred in the bones of the second subject at the second time point as the objective variable.

[0096] Here, the third image is an image showing the tissue of the second subject. The second subject may be the same person as the first subject, or a different person from the first subject. The third image may be captured at the same location as the first image G1a, or at a different location from the first image G1a. Furthermore, the third image may be captured by the same imaging device as the first image G1a, or by a different imaging device from the first image G1a. Furthermore, the first image G1a and the third image may be acquired by the same or different methods. For example, the first image G1a may be acquired from the electronic medical record management device 50, and the third image may be acquired from the image management device 40.

[0097] Furthermore, the third image may be an image showing the same region as the first image G1a, or an image showing a different region from the first image G1a. For example, the third image may be an image showing at least a portion of the chest, the same as the first image G1a, or an image showing at least a portion of the waist, different from the first image G1a. The third image may be an image in the same orientation as the first image G1a, or an image in a different orientation from the first image G1a. That is, the third image may be a front image or a side image. The information about the bones of the second subject includes, for example, at least one of the possibility of osteoporosis, bone density, bone mass, bone quality, muscle mass, etc.

[0098] For example, the possibility of osteoporosis as information about bones is classified based on at least one of the presence or absence of a fracture, the possibility of a fracture, and a change in bone density, and may be no osteoporosis, suspected osteoporosis, or present osteoporosis, etc. Specifically, the possibility of osteoporosis may be indicated when there is no disease that reduces bone mass and secondary osteoporosis is not observed, and when there is a fracture or the possibility of a fracture is high, primary osteoporosis may be indicated.

[0099] The bone density can be measured by actually measuring bone density from at least one of the hand, lumbar vertebrae, proximal femur, tibia, heel, and arm (radius, etc.). Bone density can be measured by, for example, single-energy X-ray absorptiometry, dual-energy X-ray absorptiometry, ultrasound, microdensitometry (MD), or quantitative computed tomography (CT). In a DXA device that measures bone density using the DXA method, when measuring bone density of the lumbar vertebrae, X-rays are irradiated from the front of the lumbar vertebrae of the subject. In the MD method, X-rays are irradiated, for example, to the hand.

[0100] Bone mineral density (BMD) is a value related to bone density. Bone mineral density may be expressed by at least one of bone mineral density per unit area (g / cm2), bone mineral density per unit volume (g / cm3), YAM (%), T-score, and Z-score. YAM (%) stands for "Young Adult Mean" and is sometimes referred to as the young adult mean percentage. For example, bone mineral density may be expressed as bone mineral density per unit area (g / cm2) and YAM (%). Bone mineral density may be an index defined by guidelines or a unique index. Values ​​described in osteoporosis guidelines, such as the "2015 Edition of the Prevention and Treatment Guidelines of the Japan Osteoporosis Society," can be applied to bone mineral density.

[0101] The bone mass may be information measured by a bone density measuring device such as DXA, or information obtained by estimating bone density from X-ray images using a first estimation model. Furthermore, the bone quality information may include, but is not limited to, at least one of bone formation markers, bone resorption markers, bone quality markers (e.g., vitamin K levels), cortical bone thickness, trabecular density, trabecular orientation, and trabecular bone structure index (trabecular bone score). Bone mass is the sum of bone mineral and bone matrix protein. In the present disclosure, bone mass is an index related to bone density and is the amount of bone tissue in the skeleton.

[0102] For example, muscle mass can be measured by a body composition monitor (muscle mass [kg] for each body part). For example, muscle mass can be measured by the area [cm2] and / or width [cm] of a muscle region imaged by MRI or DXA. For example, muscle mass can be measured by the muscle thickness [cm] of an ultrasound image. For example, muscle mass can be measured by a dynamometer (measurements [kg] of back muscles and / or grip strength, etc.).

[0103] The second time point refers to any time point in the future or the past of the first time point at which the third image was captured, and may include multiple time points, such as one year, five years, ten years, and thirty years after the first time point.

[0104] Note that if the machine learning period of the prediction model 32A is, for example, five years, the time of the prediction information predicted by the prediction unit 25 may be five years from now, the same as the learning period, or may be shorter than the learning period, for example, three years from now, or longer than the learning period, for example, eight years from now. The prediction unit 25 may also predict, for example, the time when the fracture risk will be 80% or more. The prediction unit 25 may also predict the time when the YAM will be 80% or less. In this case, it is sufficient for the prediction model 32A to have learned the time when the fracture risk will be 80% or more. In this case, it is sufficient for the prediction model 32A to have learned the time when the YAM will be 80% or less.

[0105] The estimation unit 27 uses at least one of the estimation models 35, which are a first estimation model 351, a second estimation model 352, and a third estimation model 353. The first estimation model 351 and the second estimation model 352 are examples of bone strength estimation models. The third estimation model 353 is an example of a bone load estimation model. The bone strength estimation model is generated by machine learning using bone strength information indicating at least one measurement result of the bone mineral density, bone mass, and bone quality of the bone of the second subject as a dependent variable.

[0106] The first estimation model 351 is generated by machine learning using the third image as an explanatory variable and bone strength information indicating the measurement results of the bone density of the second subject as a dependent variable. The first estimation model 351 outputs information indicating the bone density of the bone of the first subject from the first image G1a. Here, the bone strength information includes information regarding bone density and information regarding bone quality. Note that the information regarding bone density and the information regarding bone quality may be handled separately.

[0107] The bone density of the second subject can be measured using dual-energy X-ray absorptiometry (DXA). In a DXA device that measures bone density using DXA, for example, when measuring the bone density of the lumbar vertebrae, the lumbar vertebrae are irradiated with X-rays, specifically, two types of X-rays, from the front. Alternatively, the DXA device may irradiate the measurement location with X-rays from the side to measure the bone density of the lumbar vertebrae. Furthermore, the measurement location only needs to capture at least a portion of the chest, proximal femur, knee joint, etc.

[0108] For example, when bone mineral density of the proximal femur is measured using a DXA device, X-rays are irradiated from the front of the proximal femur of the second subject. Here, "front of the proximal femur" refers to a direction that correctly faces the imaging site, such as the proximal femur, and may be from the ventral side of the body of the second subject or from the back side of the second subject. The proximal femur includes, for example, at least one of the neck, trochanter, shaft, and the entire proximal femur (neck, trochanter, shaft, etc.).

[0109] The bone density of the second subject may be measured using an ultrasound method. In an apparatus for measuring bone density using an ultrasound method, for example, ultrasound is applied to the calcaneus to measure the bone density of the chest.

[0110] The second estimation model 352 is generated by machine learning using the third image as an explanatory variable and bone strength information indicating the measurement results of the bone quality of the second subject as a dependent variable. The second estimation model 352 outputs information indicating the bone quality of the first subject from the first image G1a.

[0111] The third estimation model 353 is generated by machine learning using the third image as an explanatory variable and bone load information indicating the measurement results of the muscle mass of the second subject as a dependent variable. The third estimation model 353 outputs information indicating the muscle mass of the first subject from the first image G1a. The bone load information is information indicating the results of measuring at least one of the muscle mass of the second subject and the posture of the second subject. Note that there is a relationship between muscle mass and bone load such that, for example, when the mass of muscles involved in maintaining posture, such as the rectus abdominis and / or erector spinae muscles, decreases, the load on the bone increases in order to maintain posture. Here, posture is indicated by the second subject's reference state, such as the degree of inclination of the body from an upright position.

[0112] In addition, the third estimation model 353 may be generated by machine learning using the third image as an explanatory variable and a fall risk objective variable indicating the possibility of the second subject falling, and may output information indicating the fall risk of the first subject from the first image G1a.

[0113] Muscle mass can be measured using at least one of the following methods: physical function measurement, measurement using a body composition scale, locomotive syndrome test, sarcopenia diagnosis, center of gravity sway measurement, lower limb muscle strength measurement, standing speed measurement, MRI, DXA, or muscle thickness measurement using ultrasound imaging diagnosis.

[0114] [Operation of Estimation Model] Next, the operation of the estimation model 35 will be described with reference to Fig. 8. Fig. 8 is a block diagram for explaining the operation of the estimation model 35. The first estimation model 351, the second estimation model 352, and the third estimation model 353 constituting the estimation model 35 are, for example, convolutional neural networks (CNN). Note that the estimation model 35 may be configured using a neural network other than a convolutional neural network.

[0115] 8 , the first estimation model 351 includes, for example, an input layer 351 a, a hidden layer 351 b, and an output layer 351 c. The first estimation model 351 includes first learned parameters that use the third image as an explanatory variable and bone strength information indicating the measurement result of the bone density of the second subject as a response variable.

[0116] In the first estimation model 351, the first image G1a is input to the input layer 351a, and the bone mineral density estimate E1 is output from the output layer 351c. Note that the hidden layer 351b may include, for example, multiple convolutional layers, multiple pooling layers, and a fully connected layer.

[0117] The bone mineral density estimate E1 is the bone mineral density per unit area [g / cm ２ ], bone mineral density per unit volume [g / cm ３ ], YAM (Young Adult Mean), T-score, and / or Z-score.

[0118] The second estimation model 352 includes, for example, an input layer 352 a, a hidden layer 352 b, and an output layer 352 c. The second estimation model 352 includes second learned parameters that use the third image as an explanatory variable and bone strength information indicating the measurement result of the bone quality of the second subject as a response variable.

[0119] In the second estimation model 352, the first image G1a is input to the input layer 352a, and the bone quality estimate E2 is output from the output layer 352c. Note that the hidden layer 352b may include, for example, multiple convolutional layers, multiple pooling layers, and a fully connected layer.

[0120] The third estimation model 353 has, for example, an input layer 353 a, a hidden layer 353 b, and an output layer 353 c. The third estimation model 353 includes third learned parameters that use the third image as an explanatory variable and bone load information indicating the measurement results of the muscle mass of the second subject as a target variable. The first learned parameter, the second learned parameter, and the third learned parameter correspond to learned parameters for estimation.

[0121] In the third estimation model 353, the first image G1a is input to the input layer 353a, and a muscle mass estimate E3, which is information indicating the muscle mass of the first subject, is output from the output layer 353c. Note that the hidden layer 353b may include, for example, multiple convolutional layers, multiple pooling layers, and a fully connected layer.

[0122] [Operation of Prediction Model] Next, the operation of the prediction model 32A will be described with reference to Fig. 9. Fig. 9 is a block diagram for explaining the operation of the prediction model 32A in embodiment 2. The prediction model 32A is, for example, a convolutional neural network. Note that the prediction model 32A may be configured using a neural network other than a convolutional neural network.

[0123] 9 , the prediction model 32A includes, for example, an input layer 32a, a hidden layer 32b, and an output layer 32c. The prediction model 32A includes trained prediction parameters that use the third image captured at the first time point and information about the bones of the second subject as explanatory variables, and abnormality information about abnormalities occurring in the bones of the second subject at the second time point as a response variable. Note that bone abnormalities may include fractures, bone loss, primary osteoporosis, secondary osteoporosis, osteophyte formation, osteomalacia, bone metastasis of malignant tumors, multiple myeloma, vertebral hemangioma, spinal caries, pyogenic spondylitis, Paget's disease of bone, fibrous dysplasia, ankylosing spondylitis, and the like.

[0124] In the prediction model 32A, a first image G1a captured at a third time point, an estimated bone density E1, an estimated bone quality E2, and an estimated muscle mass E3 are input to the input layer 32a, and a fracture risk Y indicating the probability of a fracture occurring in the bone of the first subject at a fourth time point is output. Note that the first image G1a does not necessarily have to be input to the input layer 32a. The estimated bone density E1, estimated bone quality E2, and estimated muscle mass E3, which are the first estimated information, are examples of first data.

[0125] At this time, the bone density estimate E1, bone quality estimate E2, and muscle mass estimate E3 input to the input layer 32a may be weighted based on the strength of the causal relationship with the occurrence of a fracture. For example, considering that bone strength is more influenced by bone density than by bone quality, the bone density estimate E1 may be weighted more heavily than the bone quality estimate E2. Note that weighting may be performed according to a predetermined standard such as a guideline, or an original standard.

[0126] The muscle mass estimate E3 is input into the prediction model 32A because it takes into account that a person's muscle mass is correlated with the strength that supports the bones, and affects the risk of fracture when a person falls, for example.

[0127] [Learning Process Flow] Next, the flow of the learning process by the learning unit 24 will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the learning process by the learning unit 24. The learning unit 24 executes the learning process before the prediction device 10A performs a prediction process (to be described later) and stores the prediction model 32A and the estimation model 35 in the storage unit 3.

[0128] In the flowchart shown in FIG. 10 , first, the learning unit 24 acquires a third image of the second subject via the acquisition unit 21 (S21). Here, the second subjects are multiple different people. The third image is an image of the bones and muscles of the second subject captured at a first time point. Note that in S1, the learning unit 24 acquires attribute information of each second subject from the electronic medical record management device 50 and associates the attribute information with each third image. However, if attribute information has been associated with the third image in advance, the learning unit 24 extracts the attribute information from the third image.

[0129] After S21, the learning unit 24 acquires information about the bones of the second subject via the acquisition unit 21 (S22). The information about the bones of the second subject includes information indicating the measurement results of the bone density and bone quality of the bones of the second subject, and information indicating the measurement results of the muscle mass of the second subject.

[0130] After S22, the learning unit 24 generates an estimated model 35 by machine learning using the third image acquired in S21 as an explanatory variable and information about the bones of the second subject acquired in S22 as a target variable (S23).

[0131] In S23, the learning unit 24 performs machine learning using the third image as an explanatory variable and bone strength information indicating the measurement results of the bone density of the bones of the second subject as a target variable, to generate a first estimation model 351. The learning unit 24 may input a plurality of third images to the first estimation model 351, compare the output bone density estimates with the bone density measurement results, and adjust the first estimation model 351 using an error backpropagation method or the like so as to reduce the error therebetween.

[0132] Furthermore, the learning unit 24 performs machine learning using the third image as an explanatory variable and bone strength information indicating the bone quality measurement results of the second subject as a target variable, to generate a second estimation model 352. The learning unit 24 then inputs a plurality of third images to the second estimation model 352, compares the output bone quality estimates with the bone quality measurement results, and adjusts the second estimation model 352 using an error backpropagation method or the like so as to reduce the errors between them.

[0133] The learning unit 24 also performs machine learning using the third image as an explanatory variable and bone load information indicating the measurement results of the muscle mass of the second subject as a target variable, to generate a third estimation model 353. The learning unit 24 then inputs multiple third images to the third estimation model 353, compares the output muscle mass estimates with the muscle mass measurement results, and adjusts the third estimation model 353 using an error backpropagation method or the like to reduce the error between them.

[0134] After S23, the learning unit 24 generates a prediction model 32A by machine learning using the third image captured at the first time point and information about the bones of the second subject as explanatory variables and abnormality information about abnormalities occurring in the bones of the second subject at the second time point as a response variable (S24). Here, the abnormality information about abnormalities occurring in the bones is information about fractures. Note that the abnormality information may include bone loss, osteophyte formation, bone atrophy, bone sclerosis, and the like in addition to fractures.

[0135] After S24, the learning unit 24 stores the generated estimation model 35 and prediction model 32A in the storage unit 3 (S25).

[0136] [Flow of Prediction Processing] Next, the flow of prediction processing by the prediction device 10A will be described with reference to Fig. 11 to Fig. 13. Fig. 11 is a diagram showing a plain X-ray image of the chest of a first subject. Fig. 12 is a flowchart showing an example of the flow of prediction processing by the prediction device 10A.

[0137] Hereinafter, a case will be described in which a plain X-ray image is taken of the chest of a first subject from the front as a first image G1a in a medical facility or the like, as shown in Fig. 11. Fig. 11 shows the bone B and muscle M of the first subject. In the plain X-ray image, the bone B appears white and the muscle M appears gray, making it possible to distinguish the bone B from the muscle M based on differences in color and / or brightness of the plain X-ray image. This allows the size, shape, etc. of the bone B and the muscle M to be estimated.

[0138] 12, first, the acquisition unit 21 acquires a first image G1a of the first subject from the image management device 40 (S31). The first image G1a is a plain X-ray image showing the bones B and muscles M of the first subject.

[0139] After S31, the estimation unit 27 reads out the first estimation model 351 from the storage unit 3, inputs the acquired first image G1a to the first estimation model 351, and outputs the bone mineral density estimate E1 (S32: first estimation step). The output bone mineral density estimate E1 is transmitted to the prediction unit 25. The estimation unit 27 may output the bone mineral density estimate E1 of the first subject at multiple future and past time points. The estimation unit 27 may also output the transition of the bone mineral density estimate E1 of the first subject from a past time point to a future time point.

[0140] Next, the estimation unit 27 reads out the second estimation model 352 from the storage unit 3, inputs the acquired first image G1a to the second estimation model 352, and outputs the bone quality estimation value E2 (S33: second estimation step). The output bone quality estimation value E2 is transmitted to the prediction unit 25.

[0141] Next, the estimation unit 27 reads out the third estimation model 353 from the storage unit 3, inputs the acquired first image G1a to the third estimation model 353, and outputs a muscle mass estimated value E3 (S34: third estimation step). The output muscle mass estimated value E3 is sent to the prediction unit 25. Here, the first estimation step S32, the second estimation step S33, and the third estimation step S34 correspond to estimation steps.

[0142] The estimation unit 27 may use at least one of the first estimation model 351, the second estimation model 352, and the third estimation model 353 to output at least one of the bone density estimation value E1, the bone quality estimation value E2, and the muscle mass estimation value E3.

[0143] Furthermore, the estimation unit 27 may output a bone mass estimate using a fourth estimation model that outputs information indicating the bone mass of the first subject from the first image G1 a. The fourth estimation model is generated by machine learning using the third image as an explanatory variable and bone strength information indicating the measurement result of the bone mass of the second subject as a dependent variable.

[0144] The estimation unit 27 may also output a posture estimation value using a fifth estimation model that outputs information indicating the posture of the first subject from the first image G1a. The fifth estimation model is generated by machine learning using the third image as an explanatory variable and bone load information indicating the measurement result of the posture of the second subject as a target variable. The posture estimation value may be the thoracic spine kyphotic angle (TKA), the lumbar loadosis angle (LLA), the sacral inclination angle (SIA), or a combined value of these.

[0145] After S34, the prediction unit 25 reads out the prediction model 32A from the storage unit 3, inputs the first image G1a, the estimated bone mineral density E1, the estimated bone quality E2, and the estimated muscle mass E3 into the prediction model 32A, and outputs a fracture risk Y (S35: prediction step). The fracture risk Y corresponds to prediction information indicating the possibility of a fracture occurring in the first subject's bone at a fourth time point, which is different from the third time point when the first image G1a was captured. Note that the prediction unit 25 may output the fracture risk Y for each case, taking into account whether or not the subject has undergone menopause and / or whether or not the subject has given birth.

[0146] The bone density estimate E1, bone quality estimate E2, and muscle mass estimate E3 output by the estimation unit 27, and the fracture risk Y output by the prediction unit 25 are transmitted to the presentation control unit 26. The presentation control unit 26 then presents the bone density estimate E1, bone quality estimate E2, muscle mass estimate E3, and fracture risk Y on the presentation device 60 (S36).

[0147] In S36, the presentation control unit 26 may present support information for supporting the first subject to the presentation device 60. In this case, the prediction unit 25 outputs support information for supporting the first subject to the presentation control unit 26 from the first image and the first estimated information, using a prediction model 32A corresponding to the attribute information of the first subject.

[0148] The prediction model 32A is generated by machine learning using a third image corresponding to the attribute information of the second subject and information about the bones of the second subject, namely, an estimated bone density value, an estimated bone quality value, and an estimated muscle mass value, as explanatory variables, and support information that supports reducing the fracture risk of the second subject as a target variable.

[0149] For example, when the estimated bone mineral density E1 is lower than the average bone mineral density of people with the same or similar attribute information, such as age and sex, as the first subject, the presentation control unit 26 displays on the presentation device 60 a message encouraging the first subject to take in more calcium, get more sunlight, exercise, etc. Furthermore, when the estimated muscle mass E3 is lower than the average muscle mass of people with the same or similar attribute information, such as age, sex, height, and weight, as the first subject, the presentation control unit 26 displays on the presentation device 60 a message encouraging the first subject to increase the amount of exercise.

[0150] The support information may be output by the prediction unit 25. For example, when the fracture risk Y output by the prediction unit 25 is high, the presentation control unit 26 causes the presentation device 60 to present support information urging the first subject to avoid strenuous exercise.

[0151] In S36, the presentation control unit 26 may present, to the presentation device 60, information indicating a time when a fracture is likely to occur in the bone B of the first subject. In S36, the presentation control unit 26 may present, to the presentation device 60, information indicating a probability that a fracture will occur in the bone B of the first subject within a predetermined period. In this case, the prediction unit 25 may predict a fracture prediction time when a fracture is likely to occur or a probability that a fracture will occur in the bone B of the first subject within a predetermined period, taking into account the fracture risk Y, etc. The fracture prediction time may be in units of years, for example, 5 years or 10 years, or may be in units of months, for example, 5 years and 6 months, or 10 years and 6 months. The predetermined period may be in units of years, for example, 3 years or 5 years, or may be in units of months, for example, 3 years and 6 months, or 5 years and 6 months. In S36, the presentation control unit 26 may present, to the presentation device 60, a message such as, "Fracture risk Y will be 80% or more in 10 years." In addition, in S36, the presentation control unit 26 may present on the presentation device 60 a message saying, "The fracture risk Y in three years' time is 60%."

[0152] In addition, in S36, the presentation control unit 26 may present the fracture risk Y at multiple points in time or a graph showing the progress of the fracture risk Y on the presentation device 60.

[0153] In the information processing system 1A in the above-described embodiment 2, the estimation unit 27 can estimate the bone density estimate value E1, bone quality estimate value E2, and muscle mass estimate value E3 of the bone B of the first subject as first estimated information from the first image G1a showing the bone B and muscle M of the first subject using three estimation models, namely, the first estimation model 351, the second estimation model 352, and the third estimation model 353.

[0154] The prediction model 32A uses the first image G1a and at least one of the estimated bone density E1, estimated bone quality E2, and estimated muscle mass E3 of the bone B to output the fracture risk Y of the first subject as prediction information. Specifically, the prediction unit 25 can accurately predict the fracture risk Y of a fracture occurring in the chest, which is a region captured in the first image G1a.

[0155] As a result, for example, a doctor at a medical facility can use the output results of the information processing system 1A, such as the fracture risk Y, to diagnose the first subject, who is a patient, and can diagnose the patient more appropriately and provide the patient with more accurate support information. Furthermore, even a doctor who does not specialize in orthopedics can diagnose the patient with an accuracy close to that of an orthopedic surgeon by referring to the output results of the information processing system 1A.

[0156] As support information, the doctor or the like may propose, for example, a treatment plan to increase bone density to a first subject whose bone density estimate E1 has a large impact on fracture risk Y. Furthermore, the doctor or the like may propose, as support information, a treatment plan to improve bone quality to a first subject whose bone quality estimate E2 has a large impact on fracture risk Y. Furthermore, the doctor or the like may propose, as support information, measures to increase muscle mass related to posture maintenance or measures to improve posture to a first subject whose muscle mass estimate E3 has a large impact on fracture risk Y. Furthermore, based on the support information, the doctor or the like can decide whether to recommend a diet or exercise therapy to the first subject, whether to administer medication, or the type of medication to use.

[0157] FIG. 13 shows the results of the prediction of the fracture risk Y of the first subject by the prediction device 10A and the effects of each countermeasure. FIG. 13 shows an example in which the estimation unit 27 estimates the bone density estimate E1 of the first subject to be "0.985," the bone quality estimate E2 to be "1.123," and the muscle mass estimate E3 to be "20.54." Note that higher estimates indicate greater strength of the bone B of the first subject and greater muscle mass. FIG. 13 also shows an example in which the prediction unit 25 estimates the fracture risk Y of the first subject to be "0.42." The higher the value of the fracture risk Y, the higher the likelihood of a fracture occurring.

[0158] In this way, the bone density estimate E1, bone quality estimate E2, muscle mass estimate E3, and fracture risk Y are quantified and presented on the presentation device 60, allowing doctors and others at medical facilities to communicate more specific diagnostic results to the first subject.

[0159] 13 also shows that when "Measure 1" of increasing bone density by 5% is implemented for the first subject, the fracture risk Y is reduced by "-12%." It also shows that when "Measure 2" of increasing bone quality by 5% is implemented for the first subject, the fracture risk Y is reduced by "-7%." It also shows that when "Measure 3" of increasing muscle mass by 7% is implemented for the first subject, the fracture risk Y is reduced by "-10%."

[0160] As shown in the above-mentioned measures 1 to 3, by appropriately changing the values ​​of the estimated bone density value E1, the estimated bone quality value E2, and the estimated muscle mass value E3, it is possible to identify the factor that increases the fracture risk Y. In this case, it is possible to identify that the bone density of the first subject is the factor that increases the fracture risk Y. Therefore, by recommending that the first subject increase his or her bone density, a doctor or the like can be expected to effectively reduce the fracture risk Y of the first subject.

[0161] Other Embodiment 1 In the information processing system 1 of the above-described embodiment 1, the learning unit 24 of the prediction device 10 generates the prediction model 32. However, this is not limited to this, and the prediction model 32 may be generated by a device other than the prediction device 10. In this case, the prediction model 32 generated by the other device may be stored in the storage unit 3. Note that the prediction model 32 generated by the other device may be received by a communication unit (not shown) via a communication network, and the control unit 2 may store the received prediction model 32 in the storage unit 3. With this configuration, there is no need to store the learning data 33 and the teacher data 34 in the storage unit 3.

[0162] Furthermore, in the information processing system 1 of the first embodiment described above, the control unit 2 and the storage unit 3 are provided in the prediction device 10, but this is not limited to this. The prediction device 10 may be a cloud-based device installed on a cloud. In this case, the first image G1 and the second image G2 are transmitted to the prediction device 10 on the cloud via a communication network, and the prediction information predicted by the prediction device 10 is received by the presentation device 60 via the communication network. Furthermore, the prediction device 10 may be an on-premise device installed in a medical facility or a company that provides analysis services.

[0163] Furthermore, although the information processing system 1 of the first embodiment predicts the fracture risk Y of a bone fracture in the first subject, the present invention is not limited to this. The information processing system 1 may also predict the risk of osteoporosis, scoliosis, spinal canal stenosis, intervertebral disc degeneration, ankylosing spondylitis, spinal cord injury, cartilage damage, osteomyelitis, osteophytes, muscular atrophy, spinal muscular atrophy, osteoarthritis, bone and soft tissue tumors, etc., in the first subject.

[0164] In the information processing system 1 of the first embodiment described above, the prediction device 10 outputs the fracture risk Y, which is the possibility of an abnormality occurring in a part of the body that is shown in the first image G1, as the prediction information, but this is not limited to this. The prediction information may also indicate the possibility of an abnormality occurring in a part of the body that is not shown in the first image G1.

[0165] For example, the prediction unit 25 may predict the risk Y of a lumbar or femur fracture from a first image G1 showing the chest of a first subject and a second image G2 of a region corresponding to the chest, using the prediction model 32. In this case, the prediction model 32 is generated by machine learning using a third image showing the chest of a second subject and a fourth image showing a region corresponding to the chest, both taken at a first time point, as explanatory variables, and abnormality information related to a fracture that occurred in the lumbar or femur of the second subject at a second time point as a response variable.

[0166] Furthermore, in the information processing system 1 of the first embodiment described above, the first image G1 and the second image G2 are input to the input layer 32a of the prediction model 32. However, this is not limiting, and the results of muscle strength measurements of the first subject may be input instead of the second image G2. In this case, the prediction model 32 may be generated by machine learning using the third image of the second subject captured at the first time point and the results of muscle strength measurements of the second subject at the first time point as explanatory variables, and abnormality information regarding an abnormality occurring in the bones of the second subject at the second time point as a response variable.

[0167] Furthermore, in the information processing system 1 of the first embodiment described above, the prediction device 10 inputs the first image G1 corrected in S14 of Fig. 4 to the prediction model 32 in S15, but this is not limiting. The prediction device 10 may input the first image that has not been corrected in S15 to the prediction model 32 without performing S13 and S14 of Fig. 4.

[0168] Furthermore, in the information processing system 1 of the first embodiment described above, the prediction device 10 uses a neural network as the prediction model 32, but this is not limited to this, and other models such as a linear regression model may also be used.

[0169] [Other Embodiment 2] In the prediction device 10 of the above-described embodiment 1, the prediction model 32 having one AI is stored in the storage unit 3. However, this is not limiting, and multiple AIs may be stored in the storage unit 3. For example, the storage unit 3 may store a first estimation model that outputs information indicating the bone density and / or bone quality of the first subject, and a second estimation model that outputs information indicating the muscle mass of the first subject.

[0170] The first estimation model is generated by machine learning using the first image G1 of the second subject as an explanatory variable and bone density information indicating the measurement results of the bone density and / or bone quality of the second subject as a dependent variable, and the second estimation model is generated by machine learning using the second image G2 of the second subject as an explanatory variable and muscle mass information indicating the measurement results of the muscle mass of the second subject as a dependent variable.

[0171] In S15 of Figure 4, the prediction unit 25 inputs the first image G1 of the first subject into the first estimation model, and outputs a bone density estimation value, which is information indicating the bone density of the first subject, and a bone quality estimation value, which is information indicating the bone quality of the first subject.

[0172] Here, the bone quality of the bone is a property based on at least one of statistical properties of the bone, geometric properties of the bone, mechanical properties of the bone, and chemical properties of the bone, and may include information on the attribute information of the first subject.

[0173] The bone quality can be based on at least one of bone metabolism markers, sex, race, whether or not the patient has undergone menopause, age, cortical bone condition, cancellous bone condition, cancellous bone trabecular condition, disease information, bone evaluation information, medication information, presence or absence of fracture, number of fractures, location of fracture, and fracture history. More specifically, the bone quality can be based on at least one of bone formation markers, bone resorption markers, bone quality markers (e.g., vitamin K level), cortical bone thickness, trabecular density, trabecular orientation, and trabecular bone score.

[0174] The disease information may include at least one of osteoporosis, rheumatism, osteonecrosis (e.g., femoral head necrosis, etc.), systemic sclerosis, kidney disease, and osteopetrosis. The bone evaluation information may include information evaluated by a Fracture Risk Assessment Tool (FRAX (registered trademark)). The drug information may include at least one of the trade name, generic name, dosage, administration period, and administration method (e.g., oral, intravenous injection, intramuscular injection, subcutaneous injection, etc.) of drugs including at least one of drugs that suppress bone resorption, drugs that promote bone formation, and other drugs (e.g., calcium preparations, vitamin preparations, female hormone preparations, etc.).

[0175] Furthermore, bone quality may include, for example, the type of medullary cavity shape. For example, the Dorr classification can be used for the medullary cavity shape. For example, the medullary cavity shape can be classified as follows using at least one of the thickness of the cortical bone and the shape of the medullary cavity: Type A: A type in which the cortical bone is thick and the medullary cavity is narrow. Type B: A type between Type A and Type C in which the medullary cavity is neither narrow nor wide. Type C: A type in which the cortical bone is thin and the medullary cavity is wide.

[0176] The estimated bone mineral density may be a value related to bone density. The estimated bone mineral density may be expressed by at least one of bone mineral density per unit area (g / cm2), bone mineral density per unit volume (g / cm3), YAM (%), T-score, and Z-score. YAM (%) stands for "Young Adult Mean" and is sometimes referred to as the young adult mean percentage. The estimated bone mineral density may be an index used in osteoporosis guidelines, such as the "2015 Edition of the Prevention and Treatment Guidelines of the Japan Osteoporosis Society," or may be an original index.

[0177] The prediction unit 25 also inputs the second image G2 of the first subject into the second estimation model, thereby outputting an estimated muscle mass value, which is information indicating the muscle mass of the first subject. Then, in S16 of Fig. 4, the presentation control unit 26 presents the estimated bone density value, the estimated bone quality value, and the estimated muscle mass value, in addition to the fracture risk Y, on the presentation device 60.

[0178] The first estimation model was generated by machine learning using the first image G1 of the second subject as an explanatory variable and bone density information indicating the measurement results of the bone density and bone quality of the second subject as a dependent variable. The second estimation model was generated by machine learning using the second image G2 of the second subject as an explanatory variable and muscle mass information indicating the measurement results of the muscle mass of the second subject as a dependent variable.

[0179] The bone mineral density of the second subject can be measured using, for example, DXA, ultrasound, microdensitometry (MD), and quantitative computed tomography (CT). In a DXA device that measures bone mineral density using DXA, when measuring bone mineral density of the lumbar vertebrae, X-rays are irradiated from the front of the subject's lumbar vertebrae. In a DXA device that measures bone mineral density of the proximal femur, X-rays are irradiated from the front of the subject's proximal femur. Here, "front of the lumbar vertebrae" and "front of the proximal femur" refer to the direction that correctly faces the imaging site, such as the lumbar vertebrae and the proximal femur, and may be on the ventral side of the subject's body or on the back side of the subject. The proximal femur includes, for example, at least one of the neck, trochanter, diaphysis, and the entire proximal femur (neck, trochanter, and diaphysis). In the MD method, for example, the hand is irradiated with X-rays.

[0180] The bone quality of the second subject can be measured by calculating the concentration of a bone metabolism marker in the urine or blood of the second subject, such as type I collagen cross-linked N-telopeptide (NTX), type I collagen cross-linked C-telopeptide (CTX), tartrate-resistant acid phosphatase (TRACP-5b), or deoxypyridinoline (DPD).

[0181] In addition, the muscle mass of the second subject can be measured, for example, by physical function measurement, measurement using a body composition scale, locomotive syndrome test, sarcopenia diagnosis, center of gravity sway measurement, lower limb muscle strength measurement, standing speed measurement, muscle thickness measurement using ultrasound imaging diagnosis, etc.

[0182] According to the information processing system 1 of the second alternative embodiment described above, it is possible to output the estimated bone density, estimated bone quality, and estimated muscle mass of the first subject from the first image G1 and the second image G2 of the first subject. This allows a doctor or the like at a medical facility to provide the first subject, who is a patient, with a more specific diagnosis result that takes into account each estimated value by referring to the estimated bone density, estimated bone quality, and estimated muscle mass presented on the presentation device 60.

[0183] [Other embodiment 3] In the information processing system 1 described above, in S16 of FIG. 4 , the presentation control unit 26 presents the bone density estimate, the bone quality estimate, and the muscle mass estimate to the presentation device 60 in addition to the fracture risk Y. However, support information for supporting the first subject may also be presented to the presentation device 60.

[0184] In this case, the prediction unit 25 may output support information for supporting the first subject by comparing the fracture risk Y, the estimated bone density, the estimated bone quality, and the estimated muscle mass with reference information indicating the bone density, the bone quality, and the muscle mass according to the age and / or sex of the first subject. Here, the estimated bone density, the estimated bone quality, and the estimated muscle mass correspond to the estimated information of the first subject.

[0185] For example, when the estimated bone density is low compared to the average bone density of people of the same or similar age and sex as the first subject, the prediction unit 25 outputs support information to the presentation control unit 26 that encourages the first subject to take in more calcium, get more sunlight, exercise, etc. Furthermore, when the estimated muscle mass is low compared to the average muscle mass of people of the same or similar age and sex as the first subject, the prediction unit 25 outputs support information to the presentation control unit 26 that encourages the first subject to increase the amount of exercise.

[0186] The predictor 25 may predict the attribute information of the first subject by analyzing the brightness of the echo image, which is the second image G2, using the analyzer 22. The attribute information is information including at least one of the age, sex, and muscle quality of the first subject.

[0187] In the information processing system 1 of the first embodiment described above, the prediction unit 25 outputs the fracture risk Y of the bones in the chest, which is a specific region, but this is not limiting. The prediction unit 25 may output the fracture risk for each of multiple regions of the bones of the first subject from the first image G1 and the second image G2 using the prediction model 32.

[0188] In this case, the first image G1 shows bones at multiple locations of the first subject. The second image G2 shows muscles at multiple locations of the first subject. The prediction model 32 is generated by machine learning using a third image showing bones at multiple locations of the second subject and a fourth image showing muscles at multiple locations of the second subject as explanatory variables, and abnormality occurrence information for each location that occurred within a predetermined period after the third image and / or the fourth image were captured as a target variable. Note that the third image and the fourth image may be captured at different times or the same time. If the third image and the fourth image are captured at different times, the date of capture of either image may be used as the reference date. Alternatively, the date midway between the date of capture of the third image and the date of capture of the fourth image may be used as the reference date.

[0189] According to the above-described configuration, a doctor or the like can provide the first subject with the fracture risk Y for each part of the bone B, enabling more detailed diagnosis. Note that the prediction unit 25 may combine the fracture risks Y for each part of the bone B of the first subject to predict the fracture risk Y for all of the bones of the first subject.

[0190] The prediction unit 25 may also output the fracture risk Y, the estimated bone mineral density, the estimated bone quality, and the estimated muscle mass for each of a plurality of bone sites of the first subject from the first image G1 and the second image G2 using the prediction model 32. For example, each site may be divided into regions such as the cervical vertebrae, the thoracic vertebrae, and the lumbar vertebrae, or may be divided into vertebral bodies such as the lumbar vertebrae L1, L2, L3, and L4.

[0191] The prediction unit 25 may then identify a region of interest that is highly related to fracture from among multiple regions of the bones of the first subject based on the fracture risk Y, the estimated bone density value, the estimated bone quality value, and the estimated muscle mass value, and present the region of interest via the presentation control unit 26. This allows a doctor or the like to provide more appropriate treatment by focusing on examining the region of interest when diagnosing the first subject.

[0192] Other Embodiment 5 In the information processing system 1A of the second embodiment described above, the learning unit 24 of the prediction device 10A generates the prediction model 32A and the estimation model 35. However, this is not limiting, and a device other than the prediction device 10A may generate the prediction model 32A and the estimation model 35. In this case, the prediction model 32A and the estimation model 35 generated by the other device may be stored in the storage unit 3, and the learning unit 24 may be omitted. Note that the prediction model 32A and the estimation model 35 generated by the other device may be received by a communication unit (not shown) via a communication network, and the control unit 2 may store the received prediction model 32A and the estimation model 35 in the storage unit 3. Alternatively, the prediction model 32A and the estimation model 35 generated by the other device may be recorded on a recording medium such as a USB memory or a DVD, and then the prediction model 32A and the estimation model 35 may be stored in the storage unit 3 via the recording medium.

[0193] Furthermore, in the information processing system 1A of the second embodiment described above, the control unit 2 and the storage unit 3 are included in the prediction device 10A, but this is not limiting. The prediction device 10A may be a cloud-based device installed on a cloud. In this case, estimated information regarding the future bone condition of the first subject is transmitted to the prediction device 10A on the cloud via a communication network, and the prediction information predicted by the prediction device 10A is received by the presentation device 60 via the communication network. The prediction device 10A may also be an on-premise device installed in a medical facility or a company that provides analysis services.

[0194] Furthermore, the information processing system 1A may be configured such that the imaging device that captures the first image G1 and the third image and the prediction device 10A are integrated together. In this case, the image management device 40 and the electronic medical record management device 50 are not required.

[0195] In the information processing system 1A of the second embodiment described above, the prediction information indicates the possibility of an abnormality occurring in a part that appears in the first image G1a, but is not limited to this. The prediction information may also indicate the possibility of an abnormality occurring in a part that does not appear in the first image G1a.

[0196] For example, the prediction unit 25 may predict the fracture risk Y of the lumbar vertebrae or the femur from the first image G1a showing the chest of the first subject using the prediction model 32A. In this case, the prediction model 32A is generated by machine learning using the second image showing the chest of the second subject and information about the bones of the second subject as explanatory variables and abnormality information about the fracture that has occurred in the lumbar vertebrae or the femur of the second subject as a response variable.

[0197] In the information processing system 1A of the second embodiment described above, the information indicating the bone density of the first subject's bones, the information indicating the bone quality, the information indicating the muscle mass, and the information regarding the fracture risk Y are presented in numerical form on the presentation device 60. However, this is not limiting, and each piece of information may be presented in a heat map format, for example. Furthermore, the information indicating the bone quality may be a feature amount obtained by texture analysis of at least a part of the third image.

[0198] [Other Embodiment 6] In the information processing system 1A of the above-described embodiment 2, a neural network is used as the prediction model 32A and the estimation model 35, but this is not limited thereto, and other models such as a linear regression model may also be used.

[0199] In the above-described second embodiment, the prediction unit 25 outputs the fracture risk Y of the chest bone B using the first image G1a of the chest, which is a specific region, but this is not limited to this. The prediction unit 25 may also output an abnormality for each region of the bone B of the first subject using the prediction model 32A. Here, outputting an abnormality for each region means outputting the fracture risk Y, etc., for each region, such as the cervical vertebrae, thoracic vertebrae, and lumbar vertebrae, or for each vertebra in each region (e.g., thoracic vertebrae T1 to T12, lumbar vertebrae L1 to L5, etc.).

[0200] The prediction model 32A is generated by machine learning using the third image and information about each part of the bone B of the second subject as explanatory variables, and abnormality information about fractures that occurred in each part of the bone B of the second subject at the second time point as a response variable. This configuration allows a doctor or other medical professional to inform the first subject of the fracture risk Y for each part of the bone B, enabling a more detailed diagnosis. The prediction unit 25 may combine the fracture risks Y for each part of the bone B of the first subject to predict the fracture risk Y of all bones of the first subject.

[0201] [Example of implementation using software] The functions of the prediction device 10, 10A can be realized by a program that causes a computer to function as the prediction device 10, 10A, and a program that causes a computer to function as each control block (particularly, the prediction unit 25 and the presentation control unit 26) of the prediction device 10, 10A.

[0202] In this case, the prediction device 10, 10A includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using the control device and storage device.

[0203] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0204] In addition, some or all of the functions of each of the control blocks can be realized by logic circuits. For example, integrated circuits in which logic circuits that function as each of the control blocks are formed are also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can also be realized by, for example, a quantum computer.

[0205] The invention according to the present disclosure has been described above based on the drawings and examples. However, the invention according to the present disclosure is not limited to the above-described embodiments. In other words, the invention according to the present disclosure can be modified in various ways within the scope of the present disclosure, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the invention according to the present disclosure. In other words, it should be noted that a person skilled in the art could easily make various modifications or corrections based on the present disclosure. It should also be noted that these modifications or corrections are included in the scope of the present disclosure.

[0206] [Summary 1] An information processing system according to aspect 1 of the present disclosure includes a prediction unit that outputs prediction information using a prediction model from a first image and first data showing at least a portion of a first subject. The prediction model is generated by machine learning using a third image and second data showing at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point different from the first time point when the third image was captured as a dependent variable. The prediction information is information indicating the possibility of an abnormality occurring in the bones of the first subject.

[0207] In the information processing system according to Aspect 2 of the present disclosure, in the above-mentioned Aspect 1, the first data may be data including a second image. The second data may be data including a fourth image.

[0208] In an information processing system according to Aspect 3 of the present disclosure, in Aspect 2 above, the first image may be an image depicting a predetermined region of the first subject, the second image may be an image depicting a region of the first subject corresponding to the predetermined region, the third image may be an image depicting a predetermined region of the second subject, and the fourth image may be an image depicting a region of the second subject corresponding to the predetermined region.

[0209] In an information processing system according to aspect 4 of the present disclosure, in any of aspects 1 to 3 above, the predictive information may be information indicating the possibility of an abnormality occurring in the bone of the first subject at a fourth time point, which is different from the third time point at which the first image was captured.

[0210] An information processing system according to aspect 5 of the present disclosure may be, in any one of aspects 2 to 4 above, such that the first image shows bones at multiple locations of the first subject, the second image shows muscles at multiple locations of the first subject, the third image shows bones at multiple locations of the second subject, and the fourth image shows muscles at multiple locations of the second subject.

[0211] In an information processing system according to aspect 6 of the present disclosure, in any of aspects 2 to 5 above, the prediction model may be generated by machine learning using the third image and the fourth image as explanatory variables and the abnormality information for each part that has occurred within a predetermined period since the third image and / or the fourth image was captured as a target variable, and the prediction unit may use the prediction model to output the prediction information for each of the multiple parts of the bone of the first subject from the first image and the second image.

[0212] In the information processing system according to aspect 7 of the present disclosure, in aspect 5 or 6 above, the part may include at least one of the chest, the waist, the feet, and the hands.

[0213] In an information processing system according to aspect 8 of the present disclosure, in any one of aspects 2 to 7 above, the second image may depict one or more regions of the first subject.

[0214] In an information processing system according to aspect 9 of the present disclosure, in any one of aspects 2 to 8 above, the second image may include at least one of a still image and a video.

[0215] In an information processing system according to aspect 10 of the present disclosure, in any one of aspects 2 to 9 above, the first image and the second image may include at least one of a plain X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a DXA (Dual Energy X-ray Absorptiometry) image, an echo image, and an image obtained by DES (Dual Energy Subtraction).

[0216] In an information processing system according to an eleventh aspect of the present disclosure, in any one of the second to tenth aspects, the second image may be a different image type from the first image. The fourth image may be a different image type from the third image.

[0217] In an information processing system according to aspect 12 of the present disclosure, in aspects 3 to 11 above, the prediction information may be information indicating the possibility that the abnormality will occur in the specified area shown in the first image.

[0218] In an information processing system according to aspect 13 of the present disclosure, in any of aspects 3 to 12 above, the predictive information may be information indicating the possibility that the abnormality will occur in a location other than the specified location that is not captured in the first image.

[0219] In the information processing system according to aspect 14 of the present disclosure, in any one of aspects 2 to 13 above, the abnormality may be a musculoskeletal disorder.

[0220] In an information processing system according to aspect 15 of the present disclosure, in any of aspects 2 to 14 above, the prediction unit may output bone density and / or bone quality of the first subject from the first image of the first subject using a first estimation model, and output muscle mass of the first subject from the second image of the first subject using a second estimation model, wherein the first estimation model is generated by machine learning using the third image of the second subject as an explanatory variable and bone information indicating the bone density and / or bone quality of the second subject as an objective variable, and the second estimation model may be generated by machine learning using the fourth image of the second subject as an explanatory variable and muscle information indicating the muscle mass of the second subject as an objective variable.

[0221] In an information processing system according to aspect 16 of the present disclosure, in aspect 15 above, the prediction model may be generated by machine learning using at least one of the third image, the fourth image, the bone information obtained by inputting the third image into the first estimation model, and the muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and the abnormality information as a target variable.

[0222] According to Aspect 17 of the present disclosure, the information processing system of any one of Aspects 2 to 16 further includes an analysis unit configured to analyze information including at least one of muscle and fat mass, thickness, amount of atrophy, and flexibility of the first subject using the second image, and a correction unit configured to perform a predetermined correction on the first image. The analysis unit may identify a soft tissue region by segmenting the second image, the correction unit may perform a correction on the first image to remove the soft tissue region identified by the analysis unit, and the prediction unit may output the prediction information from the first image corrected by the correction unit and the second image using the prediction model.

[0223] In an information processing system according to aspect 18 of the present disclosure, in the above-described aspect 17, the second image may include an echo image, and the prediction unit may predict attribute information of the first subject by analyzing the brightness of the echo image using the analysis unit.

[0224] In an information processing system according to aspect 19 of the present disclosure, in aspect 18 above, the attribute information may be information including at least one of the age, sex, and muscle quality of the first subject.

[0225] In an information processing system according to aspect 20 of the present disclosure, in any of aspects 2 to 19 above, the prediction unit outputs support information to support the first subject using at least one of the prediction information, estimated information of the first subject, and reference information, wherein the estimated information is at least one of bone density, bone quality, and muscle mass of the first subject, and the reference information may be at least one of bone density, bone quality, and muscle mass according to the age and / or gender of the first subject.

[0226] In an information processing system according to aspect 21 of the present disclosure, in aspect 20 above, the prediction unit may identify a region of interest that is highly related to the abnormality based on the prediction information and / or the estimated information.

[0227] In an information processing system relating to aspect 22 of the present disclosure, in any of aspects 2 to 21 above, the prediction information may include an influence degree indicating the degree to which each of the first image and the second image affects the abnormality.

[0228] In an information processing system according to aspect 23 of the present disclosure, in any of aspects 1 to 22 above, the prediction information may include information indicating a time when the abnormality is likely to occur in the first subject.

[0229] An information processing system according to aspect 24 of the present disclosure may be in any one of aspects 1 to 23 above, and may further include a presentation control unit that causes a presentation device to present the prediction information.

[0230] According to Aspect 25 of the present disclosure, the information processing system of Aspect 1 further includes an estimation unit configured to output, from the first image, the first data including first estimated information related to the bones of the first subject using an estimation model, wherein the estimation model is generated by machine learning using the third image as an explanatory variable and the second data including information related to the bones of the second subject as a target variable.

[0231] According to Aspect 26 of the present disclosure, the information processing system of Aspect 1 further includes an estimation unit configured to output, from the first image, the first data including a plurality of first estimated information pieces related to the bones of the first subject using a plurality of estimation models, the plurality of estimation models being generated by machine learning using the third image as an explanatory variable and the second data including a plurality of information pieces related to the bones of the second subject as a response variable.

[0232] In the information processing system according to aspect 27 of the present disclosure, in the above aspect 26, the predictive information may be information indicating the possibility of an abnormality occurring in the tissue of the first subject at a fourth time point, which is different from the third time point at which the first image was captured.

[0233] In an information processing system according to aspect 28 of the present disclosure, in any of aspects 25 to 27 above, the predictive information may be information indicating the possibility that the abnormality will occur in the area shown in the first image.

[0234] In an information processing system according to aspect 29 of the present disclosure, in any of aspects 25 to 27 above, the predictive information may be information indicating the possibility that the abnormality will occur in an area not captured in the first image.

[0235] In an information processing system according to aspect 30 of the present disclosure, in any of aspects 25 to 29 above, the first image may be a simple X-ray image showing at least a portion of the bones and / or muscles of the first subject, and the third image may be a simple X-ray image showing at least a portion of the bones and / or muscles of the second subject.

[0236] In an information processing system according to aspect 31 of the present disclosure, in any of aspects 25 to 30 above, the first image may be a front image or a side image, and the third image may be an image oriented in the same direction as the first image.

[0237] In the information processing system according to aspect 32 of the present disclosure, in any one of aspects 25 to 31 above, the estimation unit may use at least one of a bone strength estimation model generated by machine learning using the third image as an explanatory variable and bone strength information indicating at least one measurement result of the bone density, bone mass, and bone quality of the bones of the second subject as an objective variable, and a bone load estimation model generated by machine learning using the third image as an explanatory variable and bone load information indicating at least one measurement result of the muscle mass and posture of the second subject as an objective variable.

[0238] In an information processing system according to Aspect 33 of the present disclosure, in the information processing system according to Aspect 32, the bone strength estimation model includes a first estimation model that outputs information indicating the bone density of the bone of the first subject from the first image, and a second estimation model that outputs information indicating the bone quality of the first subject from the first image. The bone load estimation model may include a third estimation model that outputs information indicating the muscle mass of the first subject from the first image.

[0239] In an information processing system according to aspect 34 of the present disclosure, in the above-mentioned aspect 33, the first estimated information includes two or more pieces of information: information indicating the bone density of the bone of the first subject output from the first estimation model, information indicating the bone quality of the first subject output from the second estimation model, and information indicating the muscle mass of the first subject output from the third estimation model; and the prediction unit may weight each of the two or more pieces of information based on the strength of the causal relationship with the occurrence of the abnormality and input the weighted information into the prediction model.

[0240] In the information processing system according to aspect 35 of the present disclosure, in aspect 33 above, the information indicating the bone density of the first subject may be expressed by at least one of bone mineral density per unit area, bone mineral density per unit volume, YAM (Young Adult Mean), T-score, and Z-score.

[0241] In the information processing system according to aspect 36 of the present disclosure, in the above-mentioned aspect 32, the bone strength information is information measured using a method including at least one of a DXA (Dual-energy X-ray Absorptiometry) method, an ultrasound method, and a method for calculating the concentration of a bone metabolic marker in the urine or blood of the second subject, and the bone load information may be information indicating the results of measuring at least one of the muscle mass of the second subject and the posture of the second subject.

[0242] In the information processing system according to aspect 37 of the present disclosure, in any one of aspects 25 to 36 above, the abnormality may be a musculoskeletal disorder.

[0243] In the information processing system according to aspect 38 of the present disclosure, in aspect 27 above, the prediction model may be generated by machine learning using the third image and / or information for each bone part of the second subject as explanatory variables and the abnormality information regarding the abnormality that occurred for each bone part of the second subject at the second time point as a target variable, and the prediction information may be information indicating the possibility of the abnormality occurring for each tissue part of the first subject at the fourth time point.

[0244] In an information processing system according to aspect 39 of the present disclosure, in aspect 27 above, the prediction information may include information indicating a time when the abnormality is likely to occur in the tissue of the first subject.

[0245] In the information processing system according to aspect 40 of the present disclosure, in any of aspects 25 to 39 above, the prediction unit may output support information for supporting the first subject from the first image and / or the first estimated information using the prediction model corresponding to attribute information of the first subject.

[0246] A prediction device according to aspect 41 of the present disclosure may be configured in any one of aspects 25 to 40 above, further comprising a presentation control unit that causes the presentation device to present the prediction information.

[0247] A prediction device according to aspect 42 of the present disclosure includes the prediction unit in the information processing system of any one of aspects 1 to 24 above.

[0248] A prediction device according to aspect 43 of the present disclosure includes the estimation unit and the prediction unit in the information processing system of any one of aspects 25 to 41 above.

[0249] An information processing method according to aspect 44 of the present disclosure is an information processing method executed by one or more computers, and includes a prediction step of outputting predicted information using a prediction model from a first image and first data showing at least a portion of a first subject. The prediction model is generated by machine learning using a third image and second data showing at least a portion of a second subject as explanatory variables, and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point different from the first time point when the third image was captured as a dependent variable. The predicted information is information indicating the possibility of an abnormality occurring in the bones of the first subject.

[0250] In the information processing method according to Aspect 45 of the present disclosure, in Aspect 44, the first data is data including a second image. The second data may be data including a fourth image.

[0251] The information processing method according to Aspect 46 of the present disclosure is the information processing method according to Aspect 44, further including an estimation step of outputting, from the first image, the first data including first estimated information on the bones of the first subject using an estimation model. The estimation model may be generated by machine learning using the third image as an explanatory variable and the second data including information on the bones of the second subject as a target variable.

[0252] The information processing method according to Aspect 47 of the present disclosure is the information processing method according to Aspect 44, further including an estimation step of outputting, from the first image, the first data including a plurality of first estimated information related to the bones of the first subject using a plurality of estimation models. The plurality of estimation models may be generated by machine learning using the third image as an explanatory variable and the second data including a plurality of pieces of information related to the bones of the second subject as an objective variable, and the prediction model may be generated by machine learning using the second data as an explanatory variable and abnormality information related to an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point at which the third image was captured as an objective variable.

[0253] The control program according to aspect 48 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects 1 to 24 above, and may also be a control program for causing the computer to function as the prediction unit.

[0254] The control program according to aspect 49 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects 25 to 41 described above, and may be a control program for causing the computer to function as the estimation unit and the prediction unit.

[0255] The recording medium according to aspect 50 of the present disclosure may be a computer-readable non-transitory recording medium on which the control program according to aspect 48 is recorded.

[0256] The recording medium according to aspect 51 of the present disclosure may be a computer-readable non-transitory recording medium on which the control program according to aspect 49 is recorded.

[0257] [Summary 2] An information processing system according to aspect A1 of the present disclosure includes a prediction unit that outputs predictive information from a first image and a second image that show at least a portion of a first subject using a predictive model, wherein the predictive model is generated by machine learning using a third image and a fourth image that show at least a portion of a second subject as explanatory variables and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from a first time point when the third image was captured as a dependent variable, and the predictive information is information that indicates the possibility of an abnormality occurring in the bones of the first subject.

[0258] In the information processing system according to aspect A2 of the present disclosure, in the above aspect A1, the first image may be an image depicting a predetermined part of the first subject, the second image may be an image depicting a part of the first subject corresponding to the predetermined part, the third image may be an image depicting a predetermined part of the second subject, and the fourth image may be an image depicting a part of the second subject corresponding to the predetermined part.

[0259] In the information processing system according to aspect A3 of the present disclosure, in the above-mentioned aspect A1 or A2, the predictive information may be information indicating the possibility of an abnormality occurring in the bone of the first subject at a fourth time point that is different from the third time point at which the first image was captured.

[0260] The information processing system according to aspect A4 of the present disclosure may be any of aspects A1 to A3, wherein the first image shows bones at multiple locations of the first subject, the second image shows muscles at multiple locations of the first subject, the third image shows bones at multiple locations of the second subject, and the fourth image shows muscles at multiple locations of the second subject.

[0261] In the information processing system according to aspect A5 of the present disclosure, in aspect A4 above, the prediction model may be generated by machine learning using the third image and the fourth image as explanatory variables and the abnormality information for each part that has occurred within a predetermined period since the third image and / or the fourth image was captured as a target variable, and the prediction unit may use the prediction model to output the prediction information for each of the multiple parts of the bone of the first subject from the first image and the second image.

[0262] In the information processing system according to aspect A6 of the present disclosure, in aspect A4 or A5, the region may include at least one of a chest, a waist, a foot, and a hand.

[0263] In the information processing system according to aspect A7 of the present disclosure, in any one of aspects A1 to A6, the second image may show one region or multiple regions of the first subject.

[0264] In the information processing system according to aspect A8 of the present disclosure, in any one of aspects A1 to A7, the second image may include at least one of a still image and a video.

[0265] In an information processing system according to aspect A9 of the present disclosure, in any of aspects A1 to A8 above, the first image and the second image may include at least one of a plain X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a DXA (Dual Energy X-ray Absorptiometry) image, an echo image, and an image obtained by DES (Dual Energy Subtraction).

[0266] In an information processing system according to aspect A10 of the present disclosure, in any of aspects A1 to A9 above, the second image may be of a different image type from the first image, and the fourth image may be of a different image type from the third image.

[0267] In the information processing system according to aspect A11 of the present disclosure, in aspect A2, the prediction information may be information indicating a possibility that the abnormality will occur in the specified part shown in the first image.

[0268] In an information processing system according to aspect A12 of the present disclosure, in aspect A2 above, the predictive information may be information indicating the possibility that the abnormality will occur in a part other than the specified part that is not captured in the first image.

[0269] In the information processing system according to aspect A13 of the present disclosure, in any one of aspects A1 to A12, the abnormality may be a musculoskeletal disorder.

[0270] In an information processing system according to aspect A14 of the present disclosure, in any of aspects A1 to A13 above, the prediction unit may output bone density and / or bone quality of the first subject from the first image of the first subject using a first estimation model, and output muscle mass of the first subject from the second image of the first subject using a second estimation model, wherein the first estimation model is generated by machine learning using the third image of the second subject as an explanatory variable and bone information indicating the bone density and / or bone quality of the second subject as an objective variable, and the second estimation model may be generated by machine learning using the fourth image of the second subject as an explanatory variable and muscle information indicating the muscle mass of the second subject as an objective variable.

[0271] In an information processing system according to aspect A15 of the present disclosure, in any of aspects A1 to A14, the prediction model may be generated by machine learning using at least one of the third image, the fourth image, the bone information obtained by inputting the third image into the first estimation model, and the muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and the abnormality information as a target variable.

[0272] The information processing system according to Aspect A16 of the present disclosure is any of Aspects A1 to A15 described above, further comprising an analysis unit that uses the second image to analyze information including at least one of muscle and fat mass, thickness, amount of atrophy, and flexibility of the first subject, and a correction unit that performs a predetermined correction on the first image. The analysis unit identifies a soft tissue region by segmenting the second image. The correction unit performs correction to remove the soft tissue region identified by the analysis unit from the first image. The prediction unit may output the prediction information using the prediction model from the first image corrected by the correction unit and the second image.

[0273] In accordance with Aspect A17 of the present disclosure, there is provided an information processing system in accordance with Aspect A16, wherein the second image includes an echo image, and the prediction unit may predict attribute information of the first subject by analyzing brightness of the echo image using the analysis unit.

[0274] In the information processing system according to aspect A18 of the present disclosure, in the above aspect A17, the attribute information may be information including at least one of the age, sex, and muscle quality of the first subject.

[0275] In an information processing system according to Aspect A19 of the present disclosure, in any of Aspects A1 to A18, the prediction unit outputs support information for supporting the first subject using at least one of the predicted information, estimated information of the first subject, and reference information. The estimated information may be at least one of bone mineral density, bone quality, and muscle mass of the first subject, and the reference information may be at least one of bone mineral density, bone quality, and muscle mass according to the age and / or sex of the first subject.

[0276] In the information processing system according to aspect A20 of the present disclosure, in the above aspect A19, the prediction unit may identify a region of interest that is highly related to the abnormality based on the prediction information and / or the estimation information.

[0277] In an information processing system according to aspect A21 of the present disclosure, in any of aspects A1 to A20, the prediction information may include an influence degree indicating the degree to which each of the first image and the second image affects the abnormality.

[0278] In an information processing system according to aspect A22 of the present disclosure, in any one of aspects A1 to A21, the prediction information may include information indicating a time when the abnormality is likely to occur in the first subject.

[0279] The information processing system according to aspect A23 of the present disclosure is in any one of aspects A1 to A22, and may further include a presentation control unit that causes a presentation device to present the prediction information.

[0280] A prediction device according to aspect A24 of the present disclosure includes the prediction unit in the information processing system of any one of aspects A1 to A23.

[0281] An information processing method according to aspect A25 of the present disclosure is an information processing method executed by one or more computers, and includes a prediction step of outputting predictive information using a predictive model from a first image and a second image that show at least a portion of a first subject, wherein the predictive model is generated by machine learning using a third image and a fourth image that show at least a portion of a second subject as explanatory variables and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the third image was captured as a dependent variable, and the predictive information is information that indicates the possibility of an abnormality occurring in the bones of the first subject.

[0282] The control program according to aspect A26 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects A1 to A23, and may also be a control program for causing the computer to function as the prediction unit.

[0283] The recording medium according to aspect A27 of the present disclosure may be a computer-readable non-transitory recording medium on which the control program of aspect A26 described above is recorded.

[0284] [Summary 3] An information processing system according to aspect B1 of the present disclosure includes an estimation unit that outputs first estimated information regarding a bone of a first subject from a first image capturing at least a portion of the tissue of the first subject using an estimation model, and a prediction unit that outputs predicted information from the first image and the first estimated information using a prediction model. The estimation model is generated by machine learning using a second image capturing the tissue of a second subject as an explanatory variable and information about the bone of the second subject as a dependent variable. The prediction model is generated by machine learning using the second image and information about the bone of the second subject as explanatory variables and anomaly information regarding an abnormality that occurred in the bone of the second subject at a second time point different from the first time point when the second image was captured as a dependent variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

[0285] An information processing system according to aspect B2 of the present disclosure includes an estimation unit that uses a plurality of estimation models to output a plurality of first estimated information items related to the bones of a first subject from a first image capturing at least a portion of the tissue of the first subject, and a prediction unit that uses a prediction model to output predicted information from the plurality of first estimated information items. The plurality of estimation models are generated by machine learning using a second image capturing the tissue of a second subject as an explanatory variable and a plurality of pieces of information related to the bones of the second subject as a dependent variable. The prediction models are each generated by machine learning using a plurality of pieces of information related to the bones of the second subject as an explanatory variable and anomaly information related to an abnormality that occurred in the bones of the second subject at a second time point different from the first time point when the second image was captured as a dependent variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

[0286] In the information processing system according to aspect B3 of the present disclosure, in the above-mentioned aspects B1 or B2, the prediction information may be information indicating the possibility of an abnormality occurring in the tissue of the first subject at a fourth time point that is different from the third time point at which the first image was captured.

[0287] In the information processing system according to aspect B4 of the present invention, in any one of aspects B1 to B3, the prediction information may be information indicating the possibility that the abnormality will occur in the area shown in the first image.

[0288] In the information processing system according to aspect B5 of the present invention, in any of aspects B1 to B3 above, the prediction information may be information indicating the possibility that the abnormality will occur in an area not captured in the first image.

[0289] In the information processing system according to aspect B6 of the present disclosure, in any of the above aspects B1 to B5, the first image may be a plain X-ray image showing at least a portion of the bones and / or muscles of the first subject, and the second image may be a plain X-ray image showing at least a portion of the bones and / or muscles of the second subject.

[0290] In an information processing system according to aspect B7 of the present disclosure, in any of aspects B1 to B6 above, the first image may be a front image or a side image, and the second image may be an image oriented in the same direction as the first image.

[0291] In the information processing system according to aspect B8 of the present disclosure, in any of the above aspects B1 to B7, the estimation unit may use at least one of a bone strength estimation model generated by machine learning using the second image as an explanatory variable and bone strength information indicating at least one measurement result of the bone density, bone mass, and bone quality of the bones of the second subject as an objective variable, and a bone load estimation model generated by machine learning using the second image as an explanatory variable and bone load information indicating at least one measurement result of the muscle mass and posture of the second subject as an objective variable.

[0292] In the information processing system according to Aspect B9 of the present disclosure, in the above-mentioned Aspect B8, the bone strength estimation model includes a first estimation model that outputs information indicating the bone density of the bone of the first subject from the first image, and a second estimation model that outputs information indicating the bone quality of the first subject from the first image. The bone load estimation model may include a third estimation model that outputs information indicating the muscle mass of the first subject from the first image.

[0293] In an information processing system according to Aspect B10 of the present disclosure, in accordance with Aspect B9 above, the first estimated information includes two or more pieces of information among information indicating a bone mineral density of the bone of the first subject output from the first estimation model, information indicating the bone quality of the first subject output from the second estimation model, and information indicating the muscle mass of the first subject output from the third estimation model. The prediction unit may weight each of the two or more pieces of information based on the strength of a causal relationship with the occurrence of the abnormality and input the weighted information to the prediction model.

[0294] In the information processing system according to aspect B11 of the present disclosure, in the above aspect B9, the information indicating the bone density of the first subject may be expressed by at least one of bone mineral density per unit area, bone mineral density per unit volume, YAM (Young Adult Mean), T-score, and Z-score.

[0295] In an information processing system according to Aspect B12 of the present disclosure, in the above-described Aspect B8, the bone strength information is information measured using a method including at least one of a dual-energy X-ray absorptiometry (DXA) method, an ultrasound method, and a method for calculating a concentration of a bone metabolic marker in the urine or blood of the second subject, and the bone load information may be information indicating a result of measuring at least one of a muscle mass of the second subject and a posture of the second subject.

[0296] In the information processing system according to aspect B13 of the present disclosure, in any one of aspects B1 to B12, the abnormality may be a musculoskeletal disorder.

[0297] In an information processing system according to aspect B14 of the present disclosure, in any one of aspects B3 to B13, the prediction model is generated by machine learning using the second image and / or information on each of the bone regions of the second subject as explanatory variables and the abnormality information regarding the abnormality occurring in each of the bone regions of the second subject at the second time point as a response variable. The prediction information may be information indicating a possibility of the abnormality occurring in each of the tissue regions of the first subject at the fourth time point.

[0298] In an information processing system according to aspect B15 of the present disclosure, in any of aspects B1 to B14 above, the prediction information may include information indicating a time when the abnormality is likely to occur in the tissue of the first subject.

[0299] In the information processing system according to aspect B16 of the present disclosure, in any of the above aspects B1 to B15, the prediction unit may output support information for supporting the first subject from the first image and / or the first estimated information using the prediction model corresponding to attribute information of the first subject.

[0300] The information processing system according to aspect B17 of the present disclosure is in any one of aspects B1 to B16, and may further include a presentation control unit that causes a presentation device to present the prediction information.

[0301] A prediction device according to aspect B18 of the present disclosure is the prediction device in any one of aspects B1 to B17, including the estimation unit and the prediction unit in the information processing system.

[0302] An information processing method according to aspect B19 of the present disclosure is an information processing method executed by one or more computers, and includes: an estimation step of outputting first estimated information regarding a bone of a first subject from a first image capturing at least a portion of the tissue of the first subject using an estimation model; and a prediction step of outputting predicted information from the first image and the first estimated information using a prediction model. The estimation model is generated by machine learning using a second image capturing the tissue of a second subject as an explanatory variable and information about the bone of the second subject as a dependent variable. The prediction model is generated by machine learning using the second image and information about the bone of the second subject as explanatory variables and anomaly information regarding an abnormality that occurred in the bone of the second subject at a second time point, which is different from the first time point when the second image was captured, as a dependent variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

[0303] An information processing method according to aspect B20 of the present disclosure is an information processing method executed by one or more computers, and includes: an estimation step of outputting, from a first image capturing at least a portion of a tissue of a first subject, a plurality of first estimated information items related to the bones of the first subject using a plurality of estimation models; and a prediction step of outputting, from the plurality of first estimated information items, a prediction model using a prediction model. The plurality of estimation models are each generated by machine learning using a second image capturing a tissue of a second subject as an explanatory variable and a plurality of pieces of information related to the bones of the second subject as a dependent variable. The prediction model is generated by machine learning using a plurality of pieces of information related to the bones of the second subject as an explanatory variable and anomaly information related to an abnormality that occurred in the bones of the second subject at a second time point different from the first time point when the second image was captured as a dependent variable. The predicted information is information indicating the possibility of an abnormality occurring in the tissue of the first subject.

[0304] The control program according to aspect B21 of the present disclosure may be a control program for causing a computer to function as any of the information processing systems of aspects B1 to B17, and may be a control program for causing the computer to function as the estimation unit and the prediction unit.

[0305] The recording medium according to aspect B22 of the present disclosure may be a computer-readable non-transitory recording medium on which the control program of aspect B21 is recorded.

[0306] REFERENCE SIGNS LIST 1, 1A Information processing system 2 Control unit 3 Storage unit 10, 10A Prediction device 21 Acquisition unit 22 Analysis unit 23 Correction unit 24 Learning unit 25 Prediction unit 26 Presentation control unit 27 Estimation unit 31 Control program 32, 32A Prediction model 35 Estimation model 60 Presentation device 351 First estimation model 352 Second estimation model 353 Third estimation model E1 Bone density estimated value E2 Bone quality estimated value E3 Muscle mass estimated value G1, G1a First image G2 Second image

Claims

1. An information processing system comprising: a prediction unit that outputs predictive information using a prediction model from a first image and first data that show at least a portion of a first subject; wherein the predictive model is generated by machine learning using a third image and second data that show at least a portion of a second subject as explanatory variables, and abnormality information regarding abnormalities that occurred in the bones of the second subject at a second time point that is different from the first time point when the third image was taken as a dependent variable; and wherein the predictive information is information that indicates the possibility of abnormalities occurring in the bones of the first subject.

2. The information processing system according to claim 1, wherein the first data is data including a second image, and the second data is data including a fourth image.

3. An information processing system as described in claim 2, wherein the first image is an image of a predetermined part of the first subject, the second image is an image of a part of the first subject corresponding to the predetermined part, the third image is an image of a predetermined part of the second subject, and the fourth image is an image of a part of the second subject corresponding to the predetermined part.

4. An information processing system as described in claim 2 or 3, wherein the prediction information is information indicating the possibility of an abnormality occurring in the bone of the first subject at a fourth time point, which is different from the third time point at which the first image was captured.

5. An information processing system according to any one of claims 2 to 4, wherein the first image shows bones at multiple locations on the first subject, the second image shows muscles at multiple locations on the first subject, the third image shows bones at multiple locations on the second subject, and the fourth image shows muscles at multiple locations on the second subject.

6. The information processing system of claim 5, wherein the prediction model is generated by machine learning using the third image and the fourth image as explanatory variables and the abnormality information for each of the parts that has occurred within a predetermined period since the third image and / or the fourth image was captured as a target variable, and the prediction unit uses the prediction model to output the prediction information for each of the multiple parts of the bone of the first subject from the first image and the second image.

7. The information processing system according to claim 5 or 6, wherein the body parts include at least one of a chest, a waist, a foot, and a hand.

8. An information processing system according to any one of claims 2 to 7, wherein the second image shows one or more regions of the first subject.

9. The information processing system according to any one of claims 2 to 8, wherein the second image includes at least one of a still image and a video image.

10. An information processing system according to any one of claims 2 to 9, wherein the first image and the second image include at least one of a plain X-ray image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a DXA (Dual Energy X-ray Absorptiometry) image, an echo image, and an image obtained by DES (Dual Energy Subtraction).

11. An information processing system according to any one of claims 2 to 10, wherein the second image is of a different image type from the first image, and the fourth image is of a different image type from the third image.

12. An information processing system according to claim 3, wherein the prediction information is information indicating the possibility that the abnormality will occur in the specified area shown in the first image.

13. An information processing system as described in claim 3, wherein the prediction information is information indicating the possibility that the abnormality will occur in a part other than the specified part that is not captured in the first image.

14. The information processing system according to any one of claims 2 to 13, wherein the abnormality is a musculoskeletal disorder.

15. The information processing system described in any one of claims 2 to 14, wherein the prediction unit outputs the bone density and / or bone quality of the first subject from the first image of the first subject using a first estimation model, and outputs the muscle mass of the first subject from the second image of the first subject using a second estimation model, the first estimation model being generated by machine learning using the third image of the second subject as an explanatory variable and bone information indicating the bone density and / or bone quality of the second subject as an objective variable, and the second estimation model being generated by machine learning using the fourth image of the second subject as an explanatory variable and muscle information indicating the muscle mass of the second subject as an objective variable.

16. The information processing system of claim 15, wherein the prediction model is generated by machine learning using at least one of the third image, the fourth image, the bone information obtained by inputting the third image into the first estimation model, and the muscle information obtained by inputting the fourth image into the second estimation model as explanatory variables, and the abnormality information as a target variable.

17. An information processing system as described in any one of claims 2 to 16, further comprising: an analysis unit that uses the second image to analyze information including at least one of the amount, thickness, amount of atrophy, and flexibility of muscle and fat of the first subject; and a correction unit that makes a predetermined correction to the first image, wherein the analysis unit identifies a soft tissue area by segmenting the second image; the correction unit makes a correction from the first image to remove the soft tissue area identified by the analysis unit; and the prediction unit uses the prediction model to output the prediction information from the first image corrected by the correction unit and the second image.

18. The information processing system according to claim 17, wherein the second image includes an echo image, and the prediction unit predicts attribute information of the first subject by analyzing the brightness of the echo image using the analysis unit.

19. The information processing system according to claim 18, wherein the attribute information includes at least one of the age, sex, and muscle quality of the first subject.

20. An information processing system as described in any one of claims 2 to 19, wherein the prediction unit outputs support information to support the first subject using at least one of the prediction information, estimated information of the first subject, and reference information, wherein the estimated information is at least one of the bone density, bone quality, and muscle mass of the first subject, and the reference information is at least one of the bone density, bone quality, and muscle mass according to the age and / or sex of the first subject.

21. The information processing system according to claim 20, wherein the prediction unit identifies a region of interest that is highly related to the abnormality based on the prediction information and / or the estimation information.

22. An information processing system according to any one of claims 2 to 21, wherein the prediction information includes an influence degree indicating the degree of influence that each of the first image and the second image has on the abnormality.

23. An information processing system according to any one of claims 1 to 22, wherein the prediction information includes information indicating a time when the abnormality is likely to occur in the first subject.

24. An information processing system according to any one of claims 1 to 23, further comprising a presentation control unit that causes a presentation device to present the prediction information.

25. The information processing system of claim 1, further comprising an estimation unit that outputs the first data including first estimated information regarding the bones of the first subject from the first image using an estimation model, wherein the estimation model is generated by machine learning using the third image as an explanatory variable and the second data including information regarding the bones of the second subject as a target variable.

26. The information processing system of claim 1, further comprising an estimation unit that outputs the first data including a plurality of first estimated information regarding the bones of the first subject from the first image using a plurality of estimation models, wherein the plurality of estimation models are generated by machine learning using the third image as an explanatory variable and the second data including a plurality of information regarding the bones of the second subject as a target variable.

27. The information processing system of claim 26, wherein the prediction information is information indicating the possibility of an abnormality occurring in the tissue of the first subject at a fourth time point that is different from the third time point at which the first image was captured.

28. An information processing system according to any one of claims 25 to 27, wherein the prediction information is information indicating the possibility that the abnormality will occur in the area shown in the first image.

29. An information processing system according to any one of claims 25 to 27, wherein the prediction information is information indicating the possibility that the abnormality will occur in an area not captured in the first image.

30. An information processing system described in any one of claims 25 to 29, wherein the first image is a plain X-ray image showing at least a portion of the bones and / or muscles of the first subject, and the third image is a plain X-ray image showing at least a portion of the bones and / or muscles of the second subject.

31. An information processing system according to any one of claims 25 to 30, wherein the first image is a front image or a side image, and the third image is an image oriented in the same direction as the first image.

32. An information processing system according to any one of claims 25 to 31, wherein the estimation unit uses at least one of: a bone strength estimation model generated by machine learning using the third image as an explanatory variable and bone strength information indicating at least one measurement result of the bone density, bone mass, and bone quality of the bones of the second subject as an objective variable; and a bone load estimation model generated by machine learning using the third image as an explanatory variable and bone load information indicating at least one measurement result of the muscle mass and posture of the second subject as an objective variable.

33. The information processing system of claim 32, wherein the bone strength estimation model includes a first estimation model that outputs information indicating the bone density of the bone of the first subject from the first image, and a second estimation model that outputs information indicating the bone quality of the first subject from the first image, and the bone load estimation model includes a third estimation model that outputs information indicating the muscle mass of the first subject from the first image.

34. The information processing system described in claim 33, wherein the first estimated information includes two or more pieces of information: information indicating the bone density of the bone of the first subject output from the first estimation model, information indicating the bone quality of the first subject output from the second estimation model, and information indicating the muscle mass of the first subject output from the third estimation model; and the prediction unit weights each of the two or more pieces of information based on the strength of the causal relationship with the occurrence of the abnormality and inputs the weighted information into the prediction model.

35. The information processing system of claim 33, wherein the information indicating the bone density of the first subject is expressed by at least one of bone mineral density per unit area, bone mineral density per unit volume, YAM (Young Adult Mean), T-score, and Z-score.

36. The information processing system of claim 32, wherein the bone strength information is information measured using a method including at least one of a DXA (Dual-energy X-ray Absorptiometry) method, an ultrasound method, and a method for calculating the concentration of a bone metabolic marker in the urine or blood of the second subject, and the bone load information is information indicating the results of measuring at least one of the muscle mass of the second subject and the posture of the second subject.

37. An information processing system according to any one of claims 25 to 36, wherein the abnormality is a musculoskeletal disorder.

38. The information processing system of claim 27, wherein the prediction model is generated by machine learning using the third image and / or information for each bone part of the second subject as explanatory variables and the abnormality information regarding the abnormality that occurred for each bone part of the second subject at the second time point as a target variable, and the prediction information is information indicating the possibility of the abnormality occurring for each tissue part of the first subject at the fourth time point.

39. The information processing system according to claim 27, wherein the prediction information includes information indicating a time when the abnormality is likely to occur in the tissue of the first subject.

40. An information processing system described in any one of claims 25 to 39, wherein the prediction unit outputs support information for supporting the first subject from the first image and / or the first estimated information using the prediction model corresponding to attribute information of the first subject.

41. An information processing system according to any one of claims 25 to 40, further comprising a presentation control unit that causes a presentation device to present the prediction information.

42. A prediction device comprising the prediction unit in the information processing system according to any one of claims 1 to 24.

43. A prediction device comprising the estimation unit and the prediction unit in the information processing system according to any one of claims 25 to 41.

44. An information processing method executed by one or more computers, comprising a prediction step of outputting predicted information using a prediction model from a first image and first data that show at least a portion of a first subject, wherein the prediction model is generated by machine learning using a third image and second data that show at least a portion of a second subject as explanatory variables and abnormality information regarding an abnormality that occurred in the bones of the second subject at a second time point that is different from the first time point when the third image was captured as a dependent variable, and the predicted information is information that indicates the possibility of an abnormality occurring in the bones of the first subject.

45. The information processing method according to claim 44, wherein the first data is data including a second image, and the second data is data including a fourth image.

46. ​​An information processing method as described in claim 44, further comprising an estimation step of outputting the first data including first estimated information regarding the bones of the first subject from the first image using an estimation model, wherein the estimation model is generated by machine learning using the third image as an explanatory variable and the second data including information regarding the bones of the second subject as a target variable.

47. The information processing method of claim 44, further comprising an estimation step of outputting the first data including a plurality of first estimated information related to the bones of the first subject from the first image using a plurality of estimation models, wherein the plurality of estimation models are each generated by machine learning using the third image as an explanatory variable and the second data including a plurality of information related to the bones of the second subject as an objective variable, and the prediction model is generated by machine learning using the second data as an explanatory variable and abnormality information related to an abnormality that occurred in the bones of the second subject at a second time point different from the first time point when the third image was captured as an objective variable.

48. A control program for causing a computer to function as the information processing system according to any one of claims 1 to 24, the control program causing the computer to function as the prediction unit.

49. A control program for causing a computer to function as the information processing system according to any one of claims 25 to 41, the control program causing the computer to function as the estimation unit and the prediction unit.

50. A computer-readable non-transitory recording medium having the control program according to claim 48 recorded thereon.

51. A computer-readable non-transitory recording medium having the control program according to claim 49 recorded thereon.

Images