Program, information processing method, and information processing device

Abstract

Provided is a program, for example, with which it is possible to acquire information pertaining to a bodily tissue amount from an X-ray image accurately on the basis of a small number of cases. A computer acquires training data which includes an X-ray image of a site of interest and information pertaining to a bodily tissue amount obtained from a computed tomography (CT) image of the site of interest. Then, the computer generates a learning model that, when an X-ray image is input thereto, uses the acquired training data to output information pertaining to the bodily tissue amount of the site of interest in the X-ray image.

Description

Program, information processing method, and information processing device

[0001] The present disclosure relates to a program, an information processing method, and an information processing device.

[0002] Fractures in the elderly can lead to a decline in daily living functions and potentially lead to a state of need for nursing care. Therefore, it is considered important to prevent fractures by measuring bone mineral density to diagnose osteoporosis and provide appropriate treatment early. For bone mineral density measurement, dual-energy X-ray absorptiometry (DXA) devices are recommended, which measure bone mineral density (bone mass) based on the difference in the transmittance of X-rays at two energy levels. However, DXA devices are bed-type devices that take images of subjects in a supine position, requiring installation space. Furthermore, despite their high cost, their limited applications mean that they are not widely used.

[0003] On the other hand, X-ray machines (X-ray machines) are installed in many medical institutions. Therefore, Non-Patent Documents 1 to 3 propose a technique for generating a model that predicts bone density from X-ray images by learning using pairs of X-ray images and measurement results (bone density) obtained by a DXA machine as training data.

[0004] Chen-I Hsieh et al., “Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning”, NATURE COMMUNICATIONS, 12:5472 (2021) Ryoungwoo Jang et al., “Prediction of osteoporosis from simple hip radiography using deep learning algorithm”, Scientific Reports, 11:19997 (2021) Norio Yamamoto et al., “Deep Learning for “Osteoporosis Classification Using Hip Radiographs and Patient Clinical Covariates”, Biomolecules 2020,10,1534

[0005] However, the techniques disclosed in Non-Patent Documents 1 to 3 require learning from a large amount of training data, which imposes a heavy processing load on the training data collection process and learning process.

[0006] The present disclosure aims to provide a program etc. that can acquire information regarding body tissue volume from X-ray images with high accuracy using a small number of cases.

[0007] A program according to one aspect of the present disclosure causes a computer to acquire training data including an X-ray image of a target area and information regarding the amount of body tissue obtained from a CT (Computed Tomography) image of the target area, and to use the acquired training data to generate a learning model that, when an X-ray image is input, outputs information regarding the amount of body tissue of the target area in the X-ray image.

[0008] According to one aspect of the present disclosure, information regarding body tissue volume can be obtained from X-ray images with high accuracy using a small number of cases.

[0009] 1 is a block diagram showing an example of the configuration of an information processing device. FIG. 1 is an explanatory diagram showing an overview of a learning model. FIG. 1 is an explanatory diagram showing an overview of a learning model. FIG. 2 is an explanatory diagram showing an overview of a learning model. FIG. 3 is a flowchart showing an example of a training data generation processing procedure. FIG. 4 is an explanatory diagram showing a training DRR image generation processing procedure. FIG. 5 is a flowchart showing an example of a learning model generation processing procedure. FIG. 6 is a flowchart showing an example of a bone density estimation processing procedure. FIG. 7 is an explanatory diagram showing an example of a screen. FIG. 8 is a diagram showing a relationship between bone density estimated from a predicted DRR image and bone density measured by a DXA device. FIG. 9 is a diagram showing a relationship between bone density estimated from a predicted DRR image and bone density measured by a DXA device. FIG. 10 is an explanatory diagram showing an overview of a learning model of embodiment 2. FIG. 11 is a flowchart showing an example of a training data generation processing procedure of embodiment 2. FIG. 12 is a flowchart showing an example of a bone density estimation processing procedure of embodiment 2. FIG. 13 is an explanatory diagram showing an example of a screen. FIG. 14 is an explanatory diagram showing an overview of a learning model of embodiment 3. FIG. 15 is a flowchart showing an example of a training data generation processing procedure of embodiment 3. FIG. 16 is a flowchart showing an example of a muscle mass estimation processing procedure. FIG. 17 is an explanatory diagram showing an example of a screen. FIG. 18 is a flowchart showing an example of a bone density estimation processing procedure of embodiment 4. FIG. 10 is a flowchart showing an example of a bone density and muscle mass estimation processing procedure according to embodiment 5. FIG. 11 is an explanatory diagram showing an example of a screen. FIG. 12 is a flowchart showing an example of a registration processing procedure. FIG. 13 is an explanatory diagram showing the registration processing. FIG. 14 is an explanatory diagram showing the registration processing. FIG. 15 is an explanatory diagram showing the registration processing. FIG. 16 is a flowchart showing an example of a training data generation processing procedure according to embodiment 7. FIG. 17 is an explanatory diagram showing the effect of registration processing based on a bone region. FIG. 18 is an explanatory diagram showing the effect of registration processing based on a bone region. FIG. 19 is an explanatory diagram showing an example of the configuration of a learning model according to embodiment 8. FIG. 19 is an explanatory diagram showing the effect of feedback processing of loss related to muscle mass. FIG. 19 is an explanatory diagram showing the effect of feedback processing of loss related to muscle mass.

[0010] Hereinafter, a program, an information processing method, and an information processing device according to the present disclosure will be described with reference to the drawings illustrating embodiments thereof.

[0011] (Embodiment 1) This embodiment describes an information processing device that estimates bone density (information about body tissue mass) of a target region based on an X-ray image of the target region captured by an X-ray device. The bone density of the lumbar vertebrae or the proximal femur is generally used to diagnose bone loss and osteoporosis. Therefore, in this embodiment, the target region is the proximal femur, and a configuration is described in which the bone density of the proximal femur is estimated from an X-ray image (a frontal X-ray image of the hip joint) that includes the proximal femur within the imaging range. However, the target region is not limited to the proximal femur, and may be other regions such as the lumbar vertebrae or the thoracic vertebrae.

[0012] FIG. 1 is a block diagram showing an example configuration of an information processing device. The information processing device 10 is a device capable of various information processing and transmitting and receiving information, such as a personal computer, a server computer, or a workstation. The information processing device 10 is installed and used in a medical institution, a testing institution, a research institution, or the like. The information processing device 10 may be a multi-computer including multiple computers, or may be realized by a virtual machine virtually constructed within a single device. When the information processing device 10 is configured as a server computer, the information processing device 10 may be a local server installed in a medical institution, or may be a cloud server connected to the information processing device via a network such as the Internet. The following description will be given assuming that the information processing device 10 is a single computer.

[0013] The information processing device 10 estimates the bone density of the proximal femur based on, for example, a frontal hip joint X-ray image. Specifically, as described below, the information processing device 10 performs machine learning in advance to learn from predetermined training data, and prepares a learning model 12M that inputs a frontal hip joint X-ray image and outputs information about the bone density (information about body tissue mass) of the proximal femur in the frontal hip joint X-ray image. The information processing device 10 then inputs the frontal hip joint X-ray image into the learning model 12M to acquire information about the bone density of the proximal femur from the learning model 12M. In this embodiment, the bone density information is a DRR image (Digital Reconstructed Radiograph: an X-ray image obtained by projection simulation from a three-dimensional region (region of interest) of a specific part of the CT image), which is a projection image of a three-dimensional region of the proximal femur in a CT image. Since each pixel value of a CT image is a CT value corresponding to bone density, a DRR image generated from a specific bone region in a CT image can show the distribution of bone density; for example, the higher the bone density, the higher the brightness value (pixel value). Therefore, the learning model 12M of this embodiment is configured to predict and output a DRR image (an image representing the amount of body tissue) of the proximal femur included in a frontal X-ray image of the hip joint when the image is input. Furthermore, the information processing device 10 can obtain bone density from the X-ray image by calculating the bone density of the proximal femur from the DRR image predicted using the learning model 12M. Because a decrease in bone density changes the imaging state in the X-ray image, this embodiment can predict a DRR image corresponding to the bone density state of the bone region in the X-ray image using the learning model 12M.

[0014] The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are interconnected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), an AI chip (AI semiconductor), etc. The control unit 11 executes a program 12P stored in the storage unit 12 as needed to perform the processing that the information processing device 10 should perform.

[0015] The storage unit 12 includes RAM (Random Access Memory), flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 12 pre-stores a program 12P (program product) executed by the control unit 11 and various data necessary for executing the program 12P. The storage unit 12 also temporarily stores data generated when the control unit 11 executes the program 12P. The storage unit 12 also stores a learning model 12M (described below). The learning model 12M is a model trained to output a DRR image of the proximal femur contained in a frontal X-ray image of a hip joint when the X-ray image is input. The learning model 12M is expected to be used as a program module constituting artificial intelligence software. The storage unit 12 stores information defining the learning model 12M, such as information on the layers included in the learning model 12M, information on the nodes constituting each layer, and weights (coupling coefficients) between nodes.

[0016] The storage unit 12 also stores a medical image DB 12a and a training DB 12b. The medical image DB 12a stores, in association with each other, hip joint frontal X-ray images and CT images prepared for training the learning model 12M. The medical images used for training include hip joint frontal X-ray images and CT images of subjects diagnosed by a DXA device as having normal bone density, osteoporosis, or bone loss. The training DB 12b stores training data used in the training process of the learning model 12M. The training data is stored in the training DB 12b when the information processing device 10 performs a training data generation process described below. The learning model 12M, the medical image DB 12a, and the training DB 12b may be stored in another storage device connected to the information processing device 10 or in another storage device with which the information processing device 10 can communicate.

[0017] The communication unit 13 is a communication module for connecting to a network N such as the Internet or a LAN (Local Area Network) via wired or wireless communication, and transmits and receives information to other devices via the network N. The input unit 14 accepts operation input by a user and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information in accordance with instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be an integrated touch panel. Note that the input unit 14 and the display unit 15 are not essential, and the information processing device 10 may be configured to accept operations via a connected computer and output information to be displayed to an external display device.

[0018] The reading unit 16 reads information stored in the portable storage medium 10a, such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD card, a micro SD card, or a CompactFlash (registered trademark). The program 12P (program product) and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12, or the control unit 11 may download the data from another device via the communication unit 13 and store it in the storage unit 12.

[0019] 2 to 3B are explanatory diagrams showing an overview of the learning model 12M. Note that FIG. 2 conceptually illustrates how a DRR image of a region of interest in the proximal femur is predicted from a half-section image of a frontal hip joint X-ray image, and FIGS. 3A and 3B conceptually illustrate the learning process of the learning model 12M. Also, FIG. 3A illustrates the learning process of the classifier, and FIG. 3B illustrates the learning process of the generator. As shown in FIG. 2, the learning model 12M is a model that has been trained to predict a DRR image of the region of interest using a half-section image of a frontal hip joint X-ray image including the proximal femur on the side for which bone density is to be estimated as input. Note that the half-section X-ray image input to the learning model 12M may be an X-ray image including the proximal left femur or an X-ray image including the proximal right femur. When an X-ray image including the proximal right femur is input to the learning model 12M, the half-section X-ray image is flipped left and right before being input to the learning model 12M. This allows measurement of not only the proximal left femur but also the proximal right femur.

[0020] In this embodiment, a GAN (Generative Adversarial Network) is used as the learning model 12M. The learning model 12M shown in FIGS. 2 to 3B is configured as pix2pix. The GAN includes a generator that generates output data from input data and a discriminator that identifies the authenticity of the data generated by the generator, and the generator and discriminator compete with each other to perform adversarial learning to construct a network. The generator is a module that includes an encoder that extracts latent variables from input data and a decoder that generates output data from the extracted latent variables.

[0021] The learning model 12M is generated by preparing training data that associates training X-ray images (half images of frontal hip joint X-ray images) with training DRR images, and training an untrained learning model using this training data. The training X-ray images and DRR images are preferably frontal hip joint X-ray images and DRR images of subjects diagnosed by a DXA device as having normal bone density, osteoporosis, or bone loss. The information processing device 10 of this embodiment performs training using the X-ray images and DRR images prepared for training, and generates a learning model 12M that predicts DRR images from X-ray images.

[0022] In the learning process, the information processing device 10 alternately updates the generator parameters (weights, etc.) shown in FIG. 3B and the classifier parameters shown in FIG. 3A , and ends learning when the change in the error function converges. In updating the classifier parameters, the information processing device 10 fixes the generator parameters and then inputs training X-ray images to the generator. The generator accepts the training X-ray images as input and generates DRR images (information regarding the body tissue volume of the target area) as output data. The information processing device 10 then sets pairs of X-ray images (training X-ray images) and DRR images (DRR images generated by the generator) corresponding to the input and output of the generator as false data, and provides pairs of X-ray images and DRR images included in the training data to the classifier as true data, causing the classifier to discriminate between true and false. The information processing device 10 updates the classifier parameters so that it outputs a false value when false data is input and outputs a true value when true data is input. The parameters to be updated are weights (coupling coefficients) between nodes in the classifier, and the parameter optimization method can be the backpropagation method, steepest descent method, or the like.

[0023] In updating the generator parameters, the classifier parameters are fixed and learning is performed as shown in FIG. 3B . Here, the information processing device 10 inputs training X-ray images into the generator and updates the generator parameters so that when a DRR image generated by the generator is input to the classifier, the classifier will erroneously determine whether the image is true or false (i.e., will determine the image as true) and will generate an image with similar features to the training X-ray image (e.g., similar image gradients, similar statistics of the output distribution of the intermediate layer of the classifier). Again, the updated parameters include weights (coupling coefficients) between nodes in the generator, and parameter optimization methods such as backpropagation and steepest descent can be used. As a result, a learning model 12M is generated that outputs a DRR image of the proximal femur in the X-ray image when an X-ray image is input, as shown in FIG. 2 .

[0024] The information processing device 10 prepares such a learning model 12M in advance and uses it when generating (predicting) a DRR image from an X-ray image. When actually predicting a DRR image from an X-ray image using the learning model 12M, the information processing device 10 uses only the generator as shown in FIG. 2. The learning of the learning model 12M may be performed by another learning device. The trained learning model 12M generated by training on the other learning device is downloaded from the learning device to the information processing device 10 via the network N or a portable storage medium 10a, for example, and stored in the storage unit 12. Note that, of the trained learning model 12M, only the generator that generates a DRR image from an X-ray image may be downloaded from the learning device to the information processing device 10.

[0025] The learning model 12M may be a GAN such as CycleGAN or StarGAN, in addition to pix2pix. Furthermore, the learning model 12M is not limited to a GAN, and may be a neural network such as a Variational Autoencoder (VAE) or a Convolutional Neural Network (CNN) (e.g., U-net), or a model based on other learning algorithms, or may be configured by combining multiple learning algorithms.

[0026] Here, the process of generating training DRR images used for learning the learning model 12M will be described. FIG. 4 is a flowchart showing an example of the procedure for generating training data, and FIG. 5 is an explanatory diagram of the process of generating training DRR images. The following process is executed by the control unit 11 of the information processing device 10 in accordance with the program 12P stored in the storage unit 12, but may also be executed by another information processing device or learning device. In the following process, it is assumed that the X-ray images and CT images used to generate the training data are pairs of frontal X-ray images and CT images of the hip joints, which are images of the subject's pelvis and areas including the left and right femurs, associated with each other and stored in the medical image DB 12a.

[0027] The control unit 11 of the information processing device 10 reads one pair of a hip joint frontal X-ray image and a CT image stored in the medical image DB 12a (S11). First, the control unit 11 performs a brightness value calibration process on the read CT image (S12) to correct each brightness value (CT value) in the CT image. The brightness value (CT value) of each pixel measured by CT varies due to individual differences in the X-ray CT device, the installation environment, and imaging conditions. Therefore, calibration is required to correct the deviation in the measured value. The CT image calibration process is performed using calibration data acquired, for example, when the X-ray CT device is installed, when components such as the tube are replaced, at the start of imaging, or periodically. The calibration data is generated based on the radiation density (CT value expressed in Hounsfield units (HU)) obtained by imaging a phantom made of a material with known characteristics using the X-ray CT device and the tissue density of the material. Specifically, a conversion formula for converting the density of radiation obtained by passing through the phantom into the tissue density of the substance is used in the calibration data.

[0028] The calibration process can also be performed using the method described in the paper "Automated segmentation of an intensity calibration phantom in clinical CT images using a convolutional neural network" by the inventor, Keisuke Uemura et al., published online in the International Journal of Computer-Assisted Radiology and Surgery (IJCARS) on March 17, 2021. This paper discloses a technique for automatically extracting the imaging area of ​​each material in the phantom from the resulting CT image by using an X-ray CT scanner, along with the subject. Using the technology disclosed in this paper, calibration data can be generated to convert the CT value (radiation density) of each material's imaging area extracted from the CT image into the tissue density of each material. By using this generated calibration data to perform a calibration process on the CT value of the subject's imaging area, accurate tissue density can be obtained.

[0029] Next, the control unit 11 performs a process on the CT image calibrated in step S12 using a segmentation deep neural network (DNN) to classify each pixel in the CT image into one of multiple regions (musculoskeletal regions), including bone, muscle, and other regions (S13). The process of classifying each pixel in the CT image into musculoskeletal regions can be performed using, for example, the method described in the paper "Automated Muscle Segmentation from Clinical CT Using Bayesian U-Net for Personalized Musculoskeletal Modeling" by the present inventor, Yoshito Otake et al., published on pages 1030-1040 of IEEE Transactions on Medical Imaging, Vol. 39, No. 4, April 2020. This paper discloses a musculoskeletal segmentation model that inputs a CT image, classifies each pixel in the input CT image into bone, muscle, or other regions, and outputs a classified CT image (musculoskeletal labeled image) in which each pixel is associated with a label for each region. The musculoskeletal segmentation model disclosed in this paper is constructed using a Bayesian U-net. As a result, as shown in (1) in Figure 5, each pixel in a CT image is classified into one of three regions, and a musculoskeletal labeled image can be obtained in which each region is associated with a label. In Figure 5, each pixel in the musculoskeletal labeled image is schematically represented by a color (shade) corresponding to the classified region and muscle type, etc.

[0030] As shown in (2) of FIG. 5 , the control unit 11 extracts data of a bone region from the CT image based on the musculoskeletal label image generated from the CT image (S14). Then, as shown in (3) of FIG. 5 , the control unit 11 extracts data of a region of interest (here, the proximal left femur) from the extracted bone region data (CT image) (S15). The process of extracting a region of interest from a bone region can be performed, for example, by pattern matching using a template. In this case, a template representing the shape of the proximal left femur is stored in the storage unit 12 in advance. The control unit 11 determines whether or not a region of interest in the CT image of the bone region matches the template, and extracts the region of interest from the bone region, thereby extracting data of the region of interest in the bone region. The process of extracting a region of interest from a bone region can also be performed, for example, using a machine-learned learning model that, when a CT image of a bone region is input, outputs a region of interest in the bone region (the region of the proximal left femur). In this case, the control unit 11 can input the CT image of the bone region into the learning model, and identify and extract the region of interest in the bone region based on the output information from the learning model.

[0031] Next, the control unit 11 aligns the imaging target (here, the proximal left femur) in the X-ray image (frontal hip X-ray image) acquired in step S11 with the region of interest in the CT image extracted in step S15 (S16). The control unit 11 detects the brightness gradient (edge) of the X-ray image based on the pixel values ​​of each pixel, and identifies the imaging target in the X-ray image based on the detected brightness gradient. The control unit 11 may identify the imaging target in the X-ray image by pattern matching using a pre-prepared template or by using a pre-trained learning model. The control unit 11 then identifies the imaging direction of the region of interest in the CT image (the proximal left femur, which is the imaging target) that matches the imaging target in the X-ray image, and generates a CT image of the region of interest viewed from the identified direction. This allows a CT image of the region of interest aligned with the imaging target in the X-ray image to be acquired, as shown in (4) in FIG. 5 .

[0032] The registration of the subject in the X-ray image with the subject in the CT image can be performed using, for example, the method described in the paper titled "Can Anatomic Measurements of Stem Anteversion Angle Be Considered as the Functional Anteversion Angle?" by the present inventor, Keisuke Uemura et al., published on pages 595-600 of The Journal of Arthroplasty 33 (2018). This paper discloses a technique for performing segmentation using a hierarchical statistical shape model on CT images of the pelvis and femur to identify the pelvis and femur in the CT image, and then aligning (associating) the pelvis and femur in the CT image with the pelvis and femur in the X-ray image. Furthermore, the registration of X-ray images and CT images can be performed using the method described in the paper "3D-2D registration in mobile radiographs: algorithm development and preliminary clinical evaluation" by the present inventor, Yoshito Otake, et al., published on pages 2075-2090 in Physics in Medicine and Biology 60 (2015). This paper discloses a technique for translating and rotating CT images to generate CT images in which the subject is viewed from the same direction as the X-ray image.

[0033] Then, the control unit 11 generates a DRR image of the region of interest (proximal left femur) by projecting each pixel of the CT image in the same direction as the imaging direction of the X-ray image from the CT image of the region of interest (proximal left femur) that has been aligned with the imaging target (proximal left femur) in the X-ray image (S17). The control unit 11 calculates the integrated value of each pixel value (brightness value, voxel value) aligned in the same direction as the imaging direction of the X-ray image in the CT image, and sets the calculated integrated value as each pixel value of the DRR image of the region of interest. As a result, a DRR image of the region of interest such as that shown in (5) in Figure 5 is obtained, and each pixel value in the DRR image corresponds to the bone density at each position.

[0034] The control unit 11 extracts a half image including the proximal left femur from the frontal hip joint X-ray image acquired in step S11 (S18). Specifically, the control unit 11 extracts the right half (the region including the left femur) by dividing the frontal hip joint X-ray image in half horizontally. The control unit 11 associates the extracted X-ray image (half image of the frontal hip joint X-ray image) with the DRR image of the region of interest generated in step S17 and stores them as training data in the training DB 12b (S19). The control unit 11 determines whether there are any unprocessed images among the X-ray images and CT images stored in the medical image DB 12a that have not been subjected to the above-described training data generation process (S20). If it is determined that there are unprocessed images (S20: YES), the control unit 11 returns to the process of step S11 and performs the processes of steps S11 to S19 for the X-ray images and CT images for which training data generation has not been performed. If it is determined that there are no unprocessed images (S20: NO), the control unit 11 terminates the series of processes. Through the above-described processes, training data used to train the learning model 12M can be generated based on the X-ray images and CT images stored in the medical image DB 12a and stored in the training DB 12b. In the above-described processes, the X-ray images and CT images used to generate the training data are stored in the medical image DB 12a. However, the control unit 11 may also be configured to acquire X-ray images and CT images stored in another device, for example, via the network N or a portable storage medium 10a. For example, the control unit 11 may also be configured to acquire X-ray images and CT images from electronic medical record data stored in an electronic medical record server. Furthermore, in the above-described processes, a region of interest (ROI) of a bone region is extracted from a CT image and then aligned with the X-ray image. However, a region of interest may also be extracted from a CT image after alignment with the X-ray image.

[0035] Next, a process for generating a learning model 12M by learning the training data generated by the above-described process will be described. Fig. 6 is a flowchart showing an example of the process procedure for generating the learning model 12M. The following process is executed by the control unit 11 of the information processing device 10 according to the program 12P stored in the storage unit 12, but may also be executed by another learning device. Furthermore, the process for generating the training data shown in Fig. 4 and the process for generating the learning model 12M shown in Fig. 6 may each be executed by a separate device.

[0036] The control unit 11 of the information processing device 10 acquires one piece of training data from the training DB 12b (S31). Specifically, the control unit 11 reads one pair of a half-image of a frontal hip joint X-ray image (specifically, an X-ray image including the region of the proximal left femur) and a DRR image stored in the training DB 12b. The control unit 11 performs a training process for the learning model 12M using the read training data (S32). Here, the control unit 11 updates the parameters of the generator and the classifier of the learning model 12M according to the above-described procedure, and generates the learning model 12M that generates and outputs a DRR image of the proximal femur in the X-ray image when an X-ray image included in the training data is input.

[0037] The control unit 11 determines whether there is unprocessed training data that has not yet been subjected to the learning process among the training data stored in the training DB 12b (S33). If it is determined that there is unprocessed training data (S33: YES), the control unit 11 returns to the process of step S31 and executes the processes of steps S31 to S32 for the training data that has not yet been subjected to the learning process. If it is determined that there is no unprocessed training data (S33: NO), the control unit 11 ends the series of processes. Through the above-described learning process, a learning model 12M that inputs an X-ray image including a region of the proximal femur and outputs a DRR image of the proximal femur is generated.

[0038] By repeatedly performing the learning process using the training data described above, the learning model 12M can be further optimized. Furthermore, a learning model 12M that has already been trained can be retrained by performing the learning process described above, in which case a learning model 12M with higher accuracy can be generated. The learning model 12M of this embodiment is trained using training data that allows for the spatial correspondence between the abundant three-dimensional data obtained by CT and the X-ray image with high accuracy, using a segmentation technology that accurately classifies musculoskeletal regions from CT images and a technology that accurately aligns target areas in X-ray images with target areas in CT images. Therefore, a learning model 12M capable of generating highly accurate DRR images can be realized without requiring a large number of cases (training data).

[0039] Next, a process for estimating the bone mineral density of a subject's proximal femur from a frontal X-ray image of the hip joint, including the proximal femur, using the learning model 12M generated as described above will be described. Figure 7 is a flowchart showing an example of the bone mineral density estimation process, and Figure 8 is an explanatory diagram showing an example screen. The control unit 11 of the information processing device 10 executes the following process in accordance with the program 12P stored in the storage unit 12. Hereinafter, a DRR image generated from an X-ray image using the learning model 12M will be referred to as a predicted DRR image.

[0040] The control unit 11 of the information processing device 10 acquires a frontal hip joint X-ray image of a region including the pelvis and left and right femurs of a subject, such as a patient, captured with an X-ray device (S41). The control unit 11 acquires the frontal hip joint X-ray image of the patient whose bone density is to be estimated, for example, from electronic medical record data stored in an electronic medical record server. If the frontal hip joint X-ray image of the subject is stored in a portable storage medium 10a, the control unit 11 may read the frontal hip joint X-ray image from the portable storage medium 10a using the reading unit 16.

[0041] The control unit 11 extracts a half image including the proximal femur on the side where bone density is to be measured from the acquired hip joint frontal X-ray image (S42). Specifically, the control unit 11 extracts the right half region (the region including the left femur) by dividing the hip joint frontal X-ray image in half horizontally. Note that, when measuring the bone density of the subject's right proximal femur, the control unit 11 performs a process of extracting the left half region (the region including the right femur) of the hip joint frontal X-ray image and then flipping the image horizontally. Based on the half image of the X-ray image extracted in step S42, the control unit 11 generates a predicted DRR image of the proximal femur in the X-ray image (S43). Specifically, the control unit 11 inputs the X-ray image including the proximal femur (the half image of the hip joint frontal X-ray image) to the learning model 12M and obtains the predicted DRR image of the proximal femur in the X-ray image as output information from the learning model 12M.

[0042] The control unit 11 calculates the bone mineral density of the proximal femur from the generated predicted DRR image of the proximal femur (S44). Each pixel value in the predicted DRR image corresponds to bone mineral density, and the control unit 11 calculates the bone mineral density of the proximal femur, for example, by calculating the average value of each pixel in the predicted DRR image. In addition to the bone mineral density (BMD), the control unit 11 also calculates the results of a young adult comparison (YAM) and a same-age comparison calculated from the BMD. The control unit 11 stores the calculated test results in, for example, electronic medical record data in an electronic medical record server (S45).

[0043] The control unit 11 generates a screen displaying the test results, outputs the screen to, for example, the display unit 15 (S46), and displays the screen on the display unit 15, thereby completing the process. For example, the control unit 11 generates a test result screen as shown in FIG. 8 . The screen shown in FIG. 8 displays the subject's identification information (e.g., patient ID, patient name, etc.), a frontal hip joint X-ray image, and the date and time of its capture. Furthermore, the screen shown in FIG. 8 displays, as bone density test results based on the frontal hip joint X-ray image, a predicted DRR image, the name of the target site in the predicted DRR image (the proximal left femur in FIG. 8 ), bone density estimated from the predicted DRR image, a young adult comparison, and a same-age comparison. Furthermore, if comments to be presented to a physician or the like are stored in the memory unit 12 in association with the respective values ​​of bone density, young adult comparison, or same-age comparison, the control unit 11 may read the comments corresponding to the calculated test results (bone density, young adult comparison, or same-age comparison) from the memory unit 12 and display them on the test result screen as shown in FIG. 8 .

[0044] The above-described processing makes it possible to estimate the bone density of the proximal femur in a frontal hip joint X-ray image from the frontal hip joint X-ray image taken using an X-ray device commonly used in medical institutions, etc. Furthermore, in this embodiment, a predicted DRR image of the proximal femur generated from the frontal hip joint X-ray image and a bone density estimated from the predicted DRR image can be presented to a doctor, etc. Therefore, the doctor can assess the condition of the patient's proximal femur based on the predicted DRR image and the estimated bone density.

[0045] In this embodiment, the learning model 12M automatically extracts features of the imaging state of the proximal femur in X-ray images to generate a predicted DRR image. This enables bone density estimation simply by capturing images with an X-ray device, without the need for an examination using a DXA device or the like. Therefore, bone density of the target area can be estimated from X-ray images taken during health checkups or at small clinics, making it easy to perform bone density measurement tests. This enables early diagnosis and early therapeutic intervention of bone loss or osteoporosis, which is expected to prevent fractures associated with bone loss or osteoporosis and contribute to extending healthy life expectancy.

[0046] As described above, this embodiment achieves highly accurate spatial alignment between the target region (proximal femur) in an X-ray image and the target region in a CT image. Therefore, by using training data based on highly accurately aligned X-ray images and CT images (DRR images), highly accurate predicted DRR images can be generated without training a large amount of training data. For example, a predicted DRR image was generated from an X-ray image using a learning model 12M trained using 200 pairs of training data generated from X-ray images and CT images collected from a patient with osteoarthritis of the hip. The results of a comparison between the bone mineral density of the proximal femur estimated from the predicted DRR image and the bone mineral density of the proximal femur measured using a DXA device are shown in Figures 9A and 9B. Figures 9A and 9B are graphs showing the relationship between bone mineral density estimated from the predicted DRR image and bone mineral density measured using a DXA device. The diagrams shown in FIG. 9A and FIG. 9B respectively show the results of verification using X-ray images and bone mineral density (BMD) collected by X-ray devices and DXA devices at different medical institutions.

[0047] The graph in Figure 9A shows the bone mineral density of the proximal femur estimated from the predicted DRR image on the horizontal axis and the bone mineral density of the proximal femur measured by a DXA device on the vertical axis. The graph in Figure 9B shows the bone mineral density of the proximal femur measured by a DXA device on the horizontal axis and the bone mineral density of the proximal femur estimated from the predicted DRR image on the vertical axis. As can be seen from the graphs in Figures 9A and 9B, a high linear correlation was observed between the bone mineral density of the proximal femur estimated from the predicted DRR image and the bone mineral density of the proximal femur measured by a DXA device. Specifically, at the medical institution shown in Figure 9A, a correlation coefficient of 0.861 was obtained, and the average error (mean absolute error) between the bone mineral density estimated from the predicted DRR image and the bone mineral density measured by a DXA device was 0.06 g / cm2. Furthermore, at the medical institution shown in Figure 9B, a correlation coefficient of 0.869 was obtained, and the average error between bone mineral density estimated from predicted DRR images and bone mineral density measured by a DXA device was 0.07 g / cm2. Thus, even with the learning model 12M trained using a small amount of training data, it is possible to predict bone mineral density at a level comparable to that measured by a DXA device. This reduces the workload involved in collecting and learning training data.

[0048] In this embodiment, a configuration has been described in which a predicted DRR image of the proximal femur is generated from an X-ray image including the proximal femur using the learning model 12M, and the bone density of the proximal femur is estimated from the predicted DRR image. However, the bone density estimation target region may be the lumbar vertebrae, thoracic vertebrae, cervical vertebrae, clavicle, ribs, hand bones, foot bones, or specific regions thereof, in addition to the proximal femur. Similar processing is performed to generate training data and learning models for other regions, and bone density can be estimated using the learning model.

[0049] In this embodiment, the training data generation process, the learning process of the learning model 12M using the training data, and the bone mineral density estimation process using the learning model 12M are not limited to being performed locally by the information processing device 10. For example, an information processing device may be provided to perform each of the above processes. Alternatively, a server may be provided to perform the training data generation process and the learning process of the learning model 12M. In this case, the information processing device 10 is configured to transmit X-ray images and CT images used for the training data to the server, and the server generates training data from the X-ray images and CT images, generates the learning model 12M by a learning process using the generated training data, and transmits it to the information processing device 10. Thus, the information processing device 10 can realize the bone mineral density estimation process of the target area using the learning model 12M acquired from the server. Alternatively, a server may be provided to perform the bone mineral density estimation process using the learning model 12M. In this case, the information processing device 10 is configured to transmit an X-ray image of the subject to a server, and the server performs processing to generate a predicted DRR image and to estimate bone density using the learning model 12M, and transmits the generated predicted DRR image and the estimated bone density to the information processing device 10. Even with such a configuration, processing similar to that of the present embodiment described above is possible, and similar effects can be obtained.

[0050] (Embodiment 2) In the above-described embodiment 1, a configuration was described in which a predicted DRR image of a target region (e.g., a proximal femur) is generated from an X-ray image of the target region and bone density of the target region is estimated from the predicted DRR image. In this embodiment, an information processing device is described that estimates bone density of a target region from an X-ray image of a region other than the target region from which bone density is to be estimated. The information processing device of this embodiment has a configuration similar to that of the information processing device 10 of embodiment 1, and therefore a description of the configuration will be omitted. In addition to the configuration shown in FIG. 1 , the information processing device 10 of this embodiment stores a bone density estimation learning model 12M1 (see FIG. 10 ) in the storage unit 12. Furthermore, the medical image DB 12a of this embodiment stores X-ray images and CT images of a region other than the target region from which bone density is to be estimated, and bone density measured by a DXA device for the target region from which bone density is to be estimated, in association with each other. In this embodiment, the bone density of the proximal femur is estimated from a chest X-ray image, and therefore the medical image DB 12a stores X-ray and CT images of the subject's chest, as well as the bone density of the proximal femur of the subject measured by a DXA device. The bone density stored in the medical image DB 12a may be the bone density of a site used in diagnosing bone loss and osteoporosis, such as the lumbar vertebrae, pelvis, or femur, or may be the average or median bone density of the entire body.

[0051] FIG. 10 is an explanatory diagram illustrating an overview of learning models 12M and 12M1 according to the second embodiment. When estimating bone density of the proximal femur, the information processing device 10 of this embodiment uses learning model 12M to generate a predicted DRR image of a region other than the proximal femur, such as the ribs, clavicle, or thoracic vertebrae captured in a chest X-ray. The information processing device 10 then estimates the bone density of the proximal femur from the generated predicted DRR image using bone density estimation learning model 12M1 (second learning model). Note that the captured regions of the X-ray image used to estimate bone density of the proximal femur are not limited to the ribs, clavicle, or thoracic vertebrae, and each bone in an X-ray image in which the proximal femur is not captured can also be used.

[0052] The bone mineral density estimation learning model 12M1 is a machine learning model that has been trained using predetermined training data. It is trained to input a DRR image generated from a CT image of a subject's chest and output the bone mineral density of the subject's proximal femur. The bone mineral density estimation learning model 12M1 is intended for use as a program module constituting artificial intelligence software. The bone mineral density estimation learning model 12M1 is, for example, composed of a CNN, but may also be composed using other algorithms such as logistic regression or linear regression, or a combination of multiple algorithms. Since the bone mineral density of human bones is thought to decrease similarly in each bone of the body, it is thought that there is a strong correlation between the bone mineral density of the bones captured in the CT image and the bone mineral density of bones other than the captured bone. Therefore, in this embodiment, the bone mineral density estimation learning model 12M1 can be used to estimate the bone mineral density of areas other than the captured bone, as predicted from the CT image (DRR image).

[0053] The bone mineral density estimation learning model 12M1 is generated by preparing training data that associates training DRR images with training (correct) bone densities, and using this training data to train the untrained learning model 12M1. The training DRR images can be, for example, DRR images generated from CT images of the chest of a subject diagnosed with normal bone density, osteoporosis, or bone loss using a DXA device, and the training bone density can be the bone density of the subject's proximal femur measured using a DXA device. The information processing device 10 of this embodiment performs training using the DRR images and bone densities prepared for training, and generates the bone mineral density estimation learning model 12M1 that outputs the bone densities of areas other than the target of the DRR image from the DRR image.

[0054] The bone mineral density estimation learning model 12M1 learns to output a correct bone mineral density when a training DRR image is input. During the learning process, the learning model 12M1 performs calculations based on the input DRR image and calculates an output value as the calculation result. The learning model 12M1 then compares the calculated output value with the correct bone mineral density and optimizes parameters used in the calculation process so that the two values ​​approximate each other. The parameters include weights (coupling coefficients) between nodes in the bone mineral density estimation learning model 12M1, and parameter optimization methods such as backpropagation and steepest descent can be used. As a result, the learning model 12M1 is generated to output the bone mineral density of the proximal femur when a DRR image generated from a chest CT image is input. The training of the bone mineral density estimation learning model 12M1 may also be performed by another learning device.

[0055] Fig. 11 is a flowchart showing an example of a training data generation process procedure according to the second embodiment. The process shown in Fig. 11 is the same as the process shown in Fig. 4 except that step S51 is added instead of step S11, step S18 is deleted, and step S52 is added between steps S19 and S20. Explanations of the same steps as in Fig. 4 will be omitted. In this embodiment, it is assumed that chest X-ray images and chest CT images of each subject's chest are stored in association with the bone mineral density of the proximal femur of each subject measured by a DXA device for use in training the learning models 12M and 12M1 in the medical image DB 12a.

[0056] In this embodiment, the control unit 11 of the information processing device 10 retrieves one set of a chest X-ray image, a chest CT image, and a bone density of the proximal femur from the medical image DB 12a (S51). The control unit 11 then executes the same processes as steps S12 to S17 and S19 in FIG. 4 . Specifically, the control unit 11 performs a calibration process on the chest CT image (S12), classifies each pixel in the chest CT image into a bone region, a muscle region, or an other region (S13), and extracts data on the bone region from the chest CT image (S14). The control unit 11 then extracts data on a region of interest (e.g., ribs) from the CT image of the extracted bone region (S15), and aligns the position of the imaging target (here, ribs) between the chest X-ray image acquired in step S51 and the region of interest in the CT image extracted in step S15 (S16). Furthermore, the control unit 11 generates a DRR image of the region of interest (ribs) from the CT image of the region of interest that has been aligned (S17), associates the X-ray image acquired in step S51 with the DRR image of the region of interest generated in step S17, and stores the associated image in the training DB 12b as training data for learning the learning model 12M (S19). Note that in this embodiment, the chest X-ray image read from the medical image DB 12a is used as is as training data, but a process of extracting a region to be used as training data from the read X-ray image may also be performed, as in step S18 in FIG.

[0057] The control unit 11 also associates the DRR image of the region of interest generated in step S17 with the bone mineral density of the proximal femur acquired in step S51, and stores the associated data in the memory unit 12 as training data for training the bone mineral density estimation learning model 12M1 (S52). The control unit 11 may also prepare a database in the memory unit 12 for storing training data for training the bone mineral density estimation learning model 12M1 and store the generated training data in this database. The control unit 11 then proceeds to step S20. Through the above-described process, training data for training the learning model 12M can be generated and stored in the training database 12b based on the X-ray images, CT images, and bone mineral density stored in the medical image database 12a. Training data for training the bone mineral density estimation learning model 12M1 can also be generated and stored in the database.

[0058] In this embodiment, the generation of the bone mineral density estimation learning model 12M1 can be achieved by a process similar to the learning process shown in FIG. 6 . In the process of generating the bone mineral density estimation learning model 12M1, in step S31 in FIG. 6 , the control unit 11 reads pairs of DRR images of a region of interest (here, ribs) and bone mineral density of the proximal femur stored in the memory unit 12. In step S32, the control unit 11 inputs the DRR images included in the training data to the bone mineral density estimation learning model 12M1 and acquires output information from the bone mineral density estimation learning model 12M1. The control unit 11 compares the acquired output information with the bone mineral density of the proximal femur included in the training data and optimizes parameters such as weights between nodes in the bone mineral density estimation learning model 12M1 using, for example, backpropagation algorithm, so that the two are similar. Through the above-described process, the bone mineral density estimation learning model 12M1 is generated, which outputs the bone mineral density of the proximal femur by inputting DRR images generated from CT images of the ribs.

[0059] Next, a process for estimating the bone mineral density of a subject's proximal femur from a chest X-ray image of the subject using learning models 12M and 12M1 will be described. Fig. 12 is a flowchart showing an example of the bone mineral density estimation process procedure of the second embodiment, and Fig. 13 is an explanatory diagram showing an example screen. The control unit 11 of the information processing device 10 acquires a chest X-ray image of the subject's chest taken with an X-ray device (S61). Again, the control unit 11 acquires the chest X-ray image of the subject for which bone mineral density estimation is desired from, for example, electronic medical record data.

[0060] The control unit 11 inputs the acquired chest X-ray image into the learning model 12M and generates a predicted DRR image of the target region (here, the ribs) in the X-ray image (S62). The control unit 11 acquires the predicted DRR image of the ribs as output information of the learning model 12M. In this embodiment, the acquired chest X-ray image is input directly to the learning model 12M. However, as in step S42 in FIG. 7 , a process may be performed to extract a region of the target region used to estimate the bone density of the proximal femur from the acquired X-ray image, and an X-ray image of the extracted region may be input to the learning model 12M. In addition to the ribs, the target region used to estimate the bone density of the proximal femur may also be the clavicle, thoracic vertebrae, etc.

[0061] The control unit 11 inputs the predicted DRR image of the ribs generated in step S62 into the bone mineral density estimation learning model 12M1 and estimates the bone mineral density of the subject's proximal femur (S63). The control unit 11 acquires the bone mineral density of the proximal femur as output information from the bone mineral density estimation learning model 12M1. The control unit 11 calculates values ​​indicating the results of a young adult comparison (YAM) and a same-age comparison from the estimated bone mineral density (BMD) (S64). The control unit 11 then stores the test results, for example, in electronic medical record data (S65), and generates a screen displaying the test results and outputs it to the display unit 15 (S66). Here, the control unit 11 generates, for example, the test result screen shown in FIG. 13. 13 displays the subject's identification information, a chest X-ray image, and the date and time of its capture, and also displays, as bone density test results based on the chest X-ray image, a predicted DRR image of the ribs, the name of the target area in the predicted DRR image (ribs in FIG. 13), the bone density of the proximal femur estimated from the predicted DRR image, a young adult comparison, and a same-age comparison. Here, too, if a comment corresponding to the test result (bone density, young adult comparison, or same-age comparison) is stored in the memory unit 12, the control unit 11 may read the comment corresponding to the test result from the memory unit 12 and display it on the test result screen.

[0062] This embodiment also provides the same effects as those of the first embodiment. Furthermore, this embodiment allows for estimation of the bone density of the subject's proximal femur from a chest X-ray image taken using an X-ray device. Therefore, in this embodiment, for example, when a chest X-ray image is taken during a health checkup, it is possible to estimate the bone density of the subject's proximal femur from the chest X-ray image, making it possible to more easily perform bone density measurement tests during other examinations. Therefore, even for subjects for whom bone density measurement is not the purpose of the examination, it is possible to diagnose the possibility of bone loss or osteoporosis, enabling early diagnosis and early therapeutic intervention.

[0063] In addition, in this embodiment, a predicted DRR image of a specific region (e.g., a rib) generated from a chest X-ray image and a bone density of the proximal femur estimated from the predicted DRR image can be presented to a physician or other professional. Therefore, a physician can assess the condition of the patient's ribs and proximal femur based on the predicted DRR image of the specific region and the estimated bone density of the proximal femur, and diagnose bone loss or osteoporosis. Note that the bone density estimation target is not limited to the proximal femur, but can be any other region. The X-ray image used to estimate bone density may be any region other than the region for which bone density is to be estimated.

[0064] In this embodiment, the learning model 12M and the bone mineral density estimation learning model 12M1 may be configured as a single learning model. Specifically, for example, the learning model may be configured to input a chest X-ray image and output the bone mineral density of the proximal femur of the subject in the chest X-ray image. Furthermore, in this embodiment, instead of the bone mineral density estimation learning model 12M1, a learning model trained to input a predicted DRR image of a specific region (e.g., ribs) generated from a chest X-ray image using the learning model 12M and generate and output a predicted DRR image of the proximal femur from the predicted DRR image may be used. In this case, the DRR image of the proximal femur of the subject can be estimated from the predicted DRR image of the ribs, and the bone mineral density of the proximal femur can be estimated based on the estimated DRR image of the proximal femur. Furthermore, for example, a learning model trained to input a chest X-ray image and generate and output a predicted DRR image of the proximal femur of the subject in the chest X-ray image may be used. In this case, too, a DRR image of the subject's proximal femur can be estimated from a chest X-ray image, and the bone mineral density of the proximal femur can be estimated based on the DRR image of the proximal femur. In this embodiment, the modifications described in the first embodiment as appropriate can also be applied.

[0065] (Embodiment 3) In the above-described embodiment 1, a configuration was described in which a predicted DRR image of a target region (a bone region such as the proximal femur) is generated from an X-ray image of the target region, and the bone density of the target region is estimated from the predicted DRR image. In this embodiment, an information processing device is described in which a predicted DRR image of a muscle region of a target region (e.g., a muscle region such as the lower abdomen or buttocks) is generated from an X-ray image of the target region, and the muscle mass of the target region is estimated from the generated predicted DRR image. The information processing device of this embodiment has a configuration similar to that of the information processing device 10 of embodiment 1, and therefore a description of the configuration will be omitted. In this embodiment, X-ray images and CT images of the region (muscle region) from which muscle mass is to be estimated are stored in association with each other.

[0066] FIG. 14 is an explanatory diagram showing an overview of the learning model 12Ma of embodiment 3. The learning model 12Ma of this embodiment has a configuration similar to that of the learning model 12M of embodiment 1 shown in FIG. 2 and is generated by a similar learning process. Note that the learning model 12Ma of this embodiment is trained to output a DRR image of the muscle region of the buttocks (predicted DRR image, information on muscle mass) when an X-ray image of the buttocks is input, using, for example, a frontal hip joint X-ray image (X-ray image of the buttocks) and a DRR image of the muscle region of the buttocks generated from a CT image of the buttocks (lower abdomen) taken with an X-ray CT device as training data. In the example shown in FIG. 14, the learning model 12Ma generates DRR images of multiple muscle regions, such as the gluteus maximus, gluteus medius, and hamstrings, but may be configured to generate a DRR image of any one muscle region. For example, if there is a muscle region that can be used to diagnose sarcopenia, the learning model 12Ma may be generated to generate a DRR image of that muscle region.

[0067] The following describes the process of generating training data used for learning the learning model 12Ma of this embodiment. Fig. 15 is a flowchart showing an example of the training data generation process procedure of embodiment 3. The process shown in Fig. 15 is the process shown in Fig. 4 with step S71 added instead of step S14, steps S15 to S16 and S18 deleted, and steps S72 to S73 added instead of steps S17 and S19. Explanations of the same steps as in Fig. 4 will be omitted.

[0068] The control unit 11 of the information processing device 10 of this embodiment performs the same processes as steps S11 to S13 in FIG. 4 . That is, the control unit 11 reads a pair of an X-ray image (a frontal X-ray image of a hip joint) and a CT image from the medical image DB 12a, performs a brightness calibration process on the read CT image, and then classifies each pixel in the CT image into a bone region, a muscle region, and other regions (musculoskeletal regions). As a result, a musculoskeletal labeled image can be obtained in which each pixel in the CT image is classified into each region, as shown in (1) in FIG. 5 . In this embodiment, the control unit 11 extracts muscle region data from the CT image based on the musculoskeletal labeled image (S71). The control unit 11 extracts muscle region data (CT image) for each muscle type.

[0069] The control unit 11 generates a DRR image of the muscle region by projecting each pixel of the CT image in a predetermined direction from the extracted muscle region data (CT image) (S72). For example, if the X-ray image was obtained by receiving radiation passing through the subject's back to the front, the control unit 11 generates a DRR image by projecting each pixel of the CT image of the muscle region in a direction from the back to the front of the subject. Here, too, the control unit 11 determines each pixel value of the DRR image to be the integrated value of each pixel value (brightness value, voxel value) aligned in a predetermined direction in the CT image. Therefore, in this embodiment, each pixel value in the DRR image of the muscle region corresponds to the muscle density (muscle mass) at each position. Note that the control unit 11 may also align the positions of the imaging targets (here, each muscle) in the X-ray image acquired in step S11 and the CT image of the muscle region extracted in step S71, and generate a DRR image by projecting each pixel of the aligned CT image in the same direction as the X-ray image.

[0070] The control unit 11 associates the X-ray image acquired in step S11 with the DRR image of the muscle region generated in step S72 and stores the associated data in the training DB 12b as training data (S73). The control unit 11 then proceeds to step S20. Through the above-described process, training data used for training the learning model 12Ma of this embodiment can be generated based on the X-ray images and CT images stored in the medical image DB 12a and stored in the training DB 12b. By using the training data generated in this manner, the learning model 12Ma can be generated in this embodiment through a process similar to the learning process shown in FIG. 6.

[0071] Next, a process for estimating the muscle mass of a subject's buttocks from a frontal X-ray image of the subject's hip joint using the learning model 12Ma will be described. Fig. 16 is a flowchart showing an example of the muscle mass estimation process, and Fig. 17 is an explanatory diagram showing an example screen. The process shown in Fig. 16 is the process shown in Fig. 7 with step S42 deleted and steps S81 to S82 added instead of steps S43 to S44. Explanation of the same steps as in Fig. 7 will be omitted.

[0072] The control unit 11 of the information processing device 10 acquires a frontal X-ray image of the hip joint of a subject (S41). Based on the frontal X-ray image of the hip joint, the control unit 11 generates a predicted DRR image of the muscle region in the X-ray image (S81). Specifically, the control unit 11 inputs the frontal X-ray image of the hip joint into the learning model 12Ma and acquires a predicted DRR image of the muscle region of the buttocks in the X-ray image as output information from the learning model 12Ma. The control unit 11 then calculates the muscle mass of each muscle in the buttocks from the generated predicted DRR image of the muscle region (S82). In this embodiment, each pixel value in the predicted DRR image corresponds to muscle density, and the control unit 11 calculates the muscle density of the muscle region by calculating the average value of each pixel value in the predicted DRR image. The control unit 11 also calculates the muscle mass of the muscle region based on the calculated muscle density and the volume of the muscle region. The control unit 11 may calculate the muscle mass of each pixel based on the pixel value of each pixel in the predicted DRR image and add up the muscle mass corresponding to each pixel to calculate the muscle mass in that muscle region. Furthermore, the control unit 11 may predict the muscle mass of the subject's entire body based on the muscle mass in each muscle region. For example, by registering the muscle mass of each muscle of the subject, such as the gluteus maximus, gluteus medius, and hamstrings, in association with the subject's entire body muscle mass, the muscle mass of the subject's entire body can be predicted from the muscle mass of each muscle estimated from the predicted DRR image. The control unit 11 may also predict the muscle mass of the subject's upper limbs or lower limbs based on the muscle mass in each muscle region.

[0073] The control unit 11 stores the test results, including the calculated muscle density and muscle mass of each muscle, in, for example, electronic medical record data (S45), and generates a test result screen as shown in FIG. 17 and outputs it to the display unit 15 (S46). The screen shown in FIG. 17 displays the subject's identification information, a frontal hip joint X-ray image, and the date and time of its capture. The screen shown in FIG. 17 also displays, as muscle mass test results based on the frontal hip joint X-ray image, a predicted DRR image of each muscle, the name of the target part in the predicted DRR image (gluteus maximus, gluteus medius, and hamstrings in FIG. 17), and the muscle mass estimated from the predicted DRR image. Furthermore, if comments to be presented to a physician or other person associated with muscle density or muscle mass are stored in the storage unit 12, the control unit 11 may read the comments corresponding to the calculated test results (muscle density or muscle mass) from the storage unit 12 and display them on the test result screen as shown in FIG. 17.

[0074] Through the above-described processing, in this embodiment, muscle mass of the muscle region in a frontal hip joint X-ray image taken using an X-ray device can be estimated from the hip joint X-ray frontal image. Furthermore, in this embodiment, a predicted DRR image of the muscle region generated from the hip joint X-ray frontal image and the muscle mass estimated from the predicted DRR image can be presented to a physician or other professional. Therefore, the physician can assess the patient's muscle condition based on the predicted DRR image and the estimated muscle mass. In this embodiment, the learning model 12Ma automatically extracts features of the imaging state of the muscle region in the X-ray image to generate a predicted DRR image of the muscle region. This enables muscle mass estimation simply by capturing images with an X-ray device, without the need for an examination using a DXA device or the like. Therefore, muscle mass estimation of a target area can be performed from X-ray images taken during health checkups or at small clinics, making it easy to measure muscle mass. This enables early diagnosis of sarcopenia, a condition in which muscle mass and strength decrease with age, and is expected to prevent and suppress the onset and progression of sarcopenia and contribute to extending healthy lifespan.

[0075] In this embodiment, too, a learning model 12Ma can be realized that can generate, with high accuracy, a predicted DRR image of a muscle region in an X-ray image from the X-ray image by learning using a small amount of training data. Therefore, in this embodiment, it is possible to predict muscle mass and muscle density from an X-ray image to the same extent as the measurement results using a DXA device, and the workload in collecting and learning training data can be reduced.

[0076] In this embodiment, a configuration has been described in which a predicted DRR image of the muscle region of the buttocks (lower abdomen) is generated from an X-ray image of the buttocks using the learning model 12Ma, and muscle density and muscle mass of the buttocks are estimated from the predicted DRR image. The target body part for muscle density and muscle mass estimation may be the upper limbs, lower limbs, chest, abdomen, whole body, etc., in addition to the buttocks. Similar processing is used to generate training data and learning models for other body parts, making it possible to estimate muscle density and muscle mass using the learning model.

[0077] This embodiment also achieves the same effects as those of the first and second embodiments described above. Furthermore, this embodiment allows for estimation of muscle mass at a target site from an X-ray image of the target site captured using an X-ray device. Therefore, muscle mass measurement tests can be more easily performed, enabling the diagnosis of possible diseases such as sarcopenia even in subjects who are not concerned about their muscle mass, thereby enabling early diagnosis of sarcopenia and early therapeutic intervention. Furthermore, the modified examples described in the first and second embodiments described above can also be applied to this embodiment.

[0078] (Embodiment 4) An embodiment that combines the configurations of Embodiment 1 and Embodiment 2 will be described. That is, in this embodiment, an information processing device will be described that generates a predicted DRR image of a target region from an X-ray image of the target region, estimates the bone density of the target region based on the generated predicted DRR image, and estimates the bone density of a region other than the target region. The information processing device of this embodiment has a configuration similar to that of the information processing device 10 of Embodiments 1 and 2, and therefore a description of the configuration will be omitted. That is, the information processing device 10 of this embodiment includes a learning model 12M that generates a predicted DRR image of the target region from an X-ray image of the target region, and a bone density estimation learning model 12M1 that estimates the bone density of a region other than the target region from the DRR image of the target region. In the following example, the learning model 12M is a model that generates a predicted DRR image of the proximal femur from an X-ray image of the proximal femur, and the bone density estimation learning model 12M1 is a model that estimates the bone density of the ribs from the DRR image of the proximal femur. However, the learning models 12M and 12M1 are not limited to such configurations, and multiple types of models may be prepared for each of the learning models 12M and 12M1. For example, the learning model 12M may further include a model that generates (predicts) a DRR image of the pelvis from an X-ray image of the pelvis, a model that generates DRR images of the ribs, clavicle, or thoracic vertebrae from an X-ray image of the chest, etc. Furthermore, the bone density estimation learning model 12M1 may include a model that estimates the bone density of the ribs, clavicle, thoracic vertebrae, lumbar vertebrae, cervical vertebrae, hand bones, or foot bones from a DRR image of the proximal femur, pelvis, or femur, or a model that estimates the bone density of the proximal femur, lumbar vertebrae, cervical vertebrae, hand bones, or foot bones from a DRR image of the ribs, clavicle, or thoracic vertebrae.

[0079] The information processing device 10 of this embodiment is capable of executing the processes shown in Figures 11 and 6, generating training data for learning the learning models 12M and 12M1 by the process shown in Figure 11, and executing the learning process for the learning models 12M and 12M1 by the process shown in Figure 6.

[0080] Next, a process for estimating the bone mineral density of the proximal femur and the bone mineral density of the ribs of a subject from a frontal X-ray image of the hip joint of the subject using the learning models 12M and 12M1 will be described. FIG. 18 is a flowchart showing an example of the bone mineral density estimation process procedure of the fourth embodiment, and FIG. 19 is an explanatory diagram showing an example screen. The process shown in FIG. 18 is the process shown in FIG. 7 with steps S91 to S96 added after step S46. Description of the same steps as in FIG. 7 will be omitted. The control unit 11 of the information processing device 10 executes the same processes as steps S41 to S46 in FIG. 7. As a result, the information processing device 10 estimates the bone mineral density of the proximal femur of the subject from the frontal X-ray image of the hip joint of the subject, and displays an examination result screen as shown in FIG. 8.

[0081] The control unit 11 determines whether or not to estimate bone density of a region other than the region to be X-rayed (S91). For example, as shown in FIG. 19 , the examination result screen has an input field for selecting a region other than the region to be X-rayed. When a region other than the region to be X-rayed is input via the input field, the control unit 11 determines that the bone density of the input region should be estimated. In the example shown in FIG. 19 , a pull-down menu for selecting an arbitrary region is provided in the input field. When a doctor or other person wants to check the bone density of another region, the doctor or other person selects the desired region via the pull-down menu. When the control unit 11 determines not to estimate bone density of another region (S91: NO), i.e., when no other region is input in the input field, the control unit 11 terminates the process.

[0082] If the control unit 11 determines to estimate bone mineral density of another body part (S91: YES), it selects a bone mineral density estimation learning model 12M1 for the target body part based on the body part entered in the input field (S92). Here, since a predicted DRR image of the proximal femur was generated in step S43 and the ribs were selected as another body part for bone mineral density estimation, the control unit 11 selects a bone mineral density estimation learning model 12M1 for estimating the bone mineral density of the ribs from the DRR image of the proximal femur. The control unit 11 then inputs the predicted DRR image of the proximal femur generated in step S43 into the selected bone mineral density estimation learning model 12M1 and estimates the bone mineral density of the subject's ribs (another body part) based on the output information from the bone mineral density estimation learning model 12M1 (S93). The control unit 11 then executes the same processes as steps S64 to S65 in FIG. 12 (S94 to S95) and outputs the obtained test results to the currently displayed test result screen (S96). This makes it possible to display the bone density of the subject's ribs in addition to the bone density of the proximal femur estimated from the frontal X-ray image of the hip joint, as shown in FIG.

[0083] This embodiment also achieves the same effects as the above-described embodiments. Furthermore, this embodiment can estimate and present not only the bone density of a target region but also the bone density of other regions from an X-ray image of the target region. Therefore, this embodiment can estimate not only the bone density of the target region of the X-ray image but also the bone density of other regions based on an X-ray image taken during a health checkup or hospital visit. Therefore, even if an X-ray image is taken of a region other than the region for which bone density is to be confirmed, it is possible to predict the bone density of the region for which bone density is to be confirmed, which may lead to early diagnosis and early treatment of bone loss or osteoporosis. Furthermore, this embodiment can also apply the modifications described in the above-described embodiments as appropriate.

[0084] (Embodiment 5) An embodiment that combines the configurations of Embodiment 1 and Embodiment 3 will be described. That is, in this embodiment, an information processing device that estimates bone density in a bone region of a target region from an X-ray image of the target region and estimates muscle mass in a muscle region of the target region will be described. The information processing device of this embodiment has a configuration similar to that of the information processing device 10 of Embodiments 1 and 3, and therefore a description of the configuration will be omitted. That is, the information processing device 10 of this embodiment has a learning model 12M that generates a predicted DRR image of the bone region of the target region from an X-ray image of the target region, and a learning model 12Ma that generates a predicted DRR image of the muscle region of the target region from an X-ray image of the target region. In the following example, the learning model 12M is a model that generates a predicted DRR image of the proximal femur from a frontal X-ray image of the hip joint, and the learning model 12Ma is a model that generates a predicted DRR image of the muscle region of the buttocks from a frontal X-ray image of the hip joint. However, the learning models 12M and 12Ma are not limited to such configurations, and multiple types of models may be prepared for each of the learning models 12M and 12Ma. For example, a model that generates (predicts) DRR images of the ribs, clavicles, or thoracic vertebrae from a chest X-ray image may be prepared as the learning model 12M, and a model that generates DRR images of the pectoralis major, pectoralis minor, subclavius, serratus anterior, or intercostal muscles from a chest X-ray image may be prepared as the learning model 12Ma.

[0085] The information processing device 10 of this embodiment is capable of executing the processes shown in Figures 4, 6, and 15, generating training data for learning the learning model 12M by the process shown in Figure 4, generating training data for learning the learning model 12Ma by the process shown in Figure 15, and executing the learning process for the learning models 12M and 12Ma by the process shown in Figure 6.

[0086] Next, a process for estimating the bone mineral density of the proximal femur and the muscle mass of the buttocks of a subject from a frontal X-ray image of the hip joint of the subject using learning models 12M and 12Ma will be described. Fig. 20 is a flowchart showing an example of the bone mineral density and muscle mass estimation process procedure of embodiment 5, and Fig. 21 is an explanatory diagram showing an example screen. The process shown in Fig. 20 is the process shown in Fig. 7 with step S101 added between steps S41 and S42 and steps S102 to S106 added after step S46. Descriptions of steps that are the same as those in Fig. 7 will be omitted.

[0087] After acquiring a frontal X-ray image of the hip joint of the subject (S41), the control unit 11 of the information processing device 10 determines whether to estimate the bone density of the proximal femur based on the acquired X-ray image (S101). For example, a doctor or other person may specify, via a menu screen (not shown), whether to estimate the bone density of the bone region or the muscle mass of the muscle region of the subject captured in the X-ray image. If estimation of the bone density of the bone region is specified via the menu screen, the control unit 11 determines to estimate the bone density of the proximal femur (S101: YES) and executes the same processing as steps S42 to S46 in FIG. 7. As a result, the information processing device 10 estimates the bone density of the proximal femur of the subject from the frontal X-ray image of the hip joint of the subject and displays an examination result screen such as that shown in FIG. 8.

[0088] If the control unit 11 determines not to estimate the bone density of the proximal femur (S101: NO), it proceeds to step S102, where it determines whether to estimate the muscle mass of the muscle region of the buttocks, which is the subject of the acquired X-ray image (S102). If the instruction to estimate the muscle mass of the muscle region is not specified via the menu screen, the control unit 11 determines not to estimate the muscle mass of the buttocks (S102: NO) and terminates the process. On the other hand, if the instruction to estimate the muscle mass of the muscle region is specified via the menu screen, the control unit 11 determines to estimate the muscle mass of the buttocks (S102: YES) and executes the same processes as steps S81 to S82 and S45 in FIG. 16 (S103 to S105). As a result, the information processing device 10 estimates the muscle mass of the muscle region of the buttocks of the subject from the frontal X-ray image of the hip joint and outputs the obtained test results to the currently displayed test result screen (S106). Therefore, in addition to the bone density of the proximal femur estimated from the frontal X-ray image of the hip joint, the muscle mass of the buttocks of the subject can be displayed, as shown in Fig. 21. By the above-described processing, the bone density of the bone region and the muscle mass of the muscle region of the target site can be estimated from the X-ray image of the target site, and whether to estimate bone density or muscle mass, or both, can be appropriately switched based on a specification by a user such as a doctor.

[0089] This embodiment also provides the same effects as the above-described embodiments. Furthermore, this embodiment can estimate and present not only the bone density of the bone region of the target region but also the muscle mass of the muscle region of the target region from an X-ray image of the target region. Therefore, the bone and muscle conditions of the imaged subject can be confirmed, enabling early diagnosis of bone loss or osteoporosis, as well as early diagnosis of diseases such as sarcopenia. This embodiment can also apply the modified examples described in the above-described embodiments as appropriate.

[0090] (Embodiment 6) This section describes a process in which an information processing device aligns a bone region (target site or region of interest) in an X-ray image (plain X-ray image) and a bone region (target site or region of interest) in a CT image, which are X-ray images (plain X-ray images) and CT images of the same subject and the same imaging target in the above-described embodiments 1 to 5. The information processing device of this embodiment has the same configuration as the information processing device 10 of embodiments 1 to 5, and therefore a description of the configuration will be omitted.

[0091] FIG. 22 is a flowchart showing an example of the alignment process procedure, and FIGS. 23A to 24B are explanatory diagrams of the alignment process. The alignment process shown in FIG. 22 corresponds to step S16 in FIGS. 4 and 11. Therefore, in this embodiment, the control unit 11 of the information processing device 10 executes the process of FIG. 22 after executing step S15 in FIGS. 4 and 11, and then executes step S17 in FIGS. 4 and 11. The following description uses the pelvis as a region of interest and an example of aligning the pelvis in a frontal hip joint X-ray image with the pelvis in a CT image. However, the bone region used for alignment is not limited to the region of interest. Any bone region captured in the X-ray image and the CT image can be used for alignment. For example, in a frontal hip joint X-ray image, in addition to the pelvis, the femur, the proximal femur, etc. may also be used for alignment.

[0092] 4 and 11 , the control unit 11 identifies a region of interest (here, the pelvis) in the X-ray image (here, the frontal X-ray image of the hip joint) acquired in step S11 (S111). The process of identifying the region of interest in the X-ray image can be performed, for example, by pattern matching using a template that indicates the shape of the region of interest. Alternatively, for example, the process can be performed using a learning model that has been machine-learned to output the region of interest in an X-ray image when the X-ray image is input. As a result, for example, the pelvis region indicated by the solid line in the X-ray image shown in FIG. 23A is identified.

[0093] Next, the control unit 11 generates a pseudo-DRR image of the region of interest based on the region of interest in the CT image extracted in step S15 (S112). Here, the control unit 11 generates a pseudo-DRR image of the pelvis. Specifically, as shown in FIG. 23B , the control unit 11 places the region of interest in the CT image (a 3D CT image of the pelvis) in the 3D virtual space of the X-ray imaging system under predetermined projection conditions (position and angle relative to a virtual X-ray source), and generates a pseudo-DRR image (projection image) by projecting the region of interest from the virtual X-ray source onto a 2D X-ray imaging surface. Here, because the imaging conditions (e.g., the subject's posture, joint flexion angle, etc.) are different between the X-ray image and the CT image, the contour of the region of interest in the pseudo-DRR image generated from the CT image does not match the contour of the region of interest in the X-ray image. 24A , the contour P1 of the region of interest in the X-ray image is indicated by a solid line, and the contour P2 of the region of interest in the pseudo-DRR image is indicated by a dashed line. In this embodiment, the control unit 11 updates the projection conditions of the region of interest in the CT image to identify the projection conditions that maximize the correlation value between the contour P1 of the region of interest in the X-ray image and the contour P2 of the region of interest in the pseudo-DRR image. By using these projection conditions, a pseudo-DRR image can be obtained in which the region of interest in the pseudo-DRR image is accurately aligned with the region of interest in the X-ray image, as shown in FIG. 24B .

[0094] Therefore, the control unit 11 calculates a correlation value between the contour of the region of interest in the X-ray image identified in step S111 and the contour of the region of interest in the pseudo-DRR image generated in step S112 (S113), and determines whether the calculated correlation value is maximum (S114). If the control unit 11 determines that the correlation value is not maximum (S114: NO), it updates the projection conditions used to generate the pseudo-DRR image from the CT image of the region of interest (S115) and repeats the processes of steps S112 to S114 under the updated projection conditions. The control unit 11 repeats the processes of steps S112 to S115 until it determines that the calculated correlation value is maximum. If it determines that the correlation value is maximum (S114: YES), that is, if a pseudo-DRR image with the maximum correlation value has been generated, it identifies the projection conditions at that time (S116).

[0095] In this embodiment, the control unit 11 can perform the processes of steps S112 to S115 using, for example, the method described in a paper titled "3D-2D registration in mobile radiographs: algorithm development and preliminary clinical evaluation" written by the present inventor, Yoshito Otake et al. This paper defines the achievement of registration between the contour of a target region (a bone region, in this case, the pelvis) in a DRR image (a pseudo-DRR image in this case) generated from a three-dimensional region of the target region in a CT image and the contour of the target region in an actual X-ray image as "maximizing the correlation between the gray-scale gradient intensity images of the X-ray image and the DRR image," and discloses a method for determining a configuration (projection conditions, specifically, the three-dimensional position and angle of the target region relative to the imaging system) that maximizes the correlation using a covariance matrix adaptation evolution strategy (CMA-ES). Therefore, the control unit 11 can use the covariance matrix adaptive evolution strategy to identify a pseudo-DRR image that maximizes the correlation value between the contour of the region of interest in the X-ray image and the contour of the pseudo-DRR image, thereby identifying projection conditions for the identified pseudo-DRR image. Under these projection conditions, a pseudo-DRR image of the region of interest can be generated in which the contour P2 of the region of interest in the pseudo-DRR image is aligned with the contour P1 of the region of interest in the X-ray image with high accuracy, as shown in FIG. 24B . Note that in step S114, the control unit 11 may determine whether the calculated correlation value is equal to or greater than a predetermined value, and proceed to processing in step S116 if it is determined that the correlation value is equal to or greater than the predetermined value.

[0096] 4 and 11 , and generates a DRR image by projecting the CT image of the region of interest in a projection direction according to the projection conditions specified in step S116 (S17). Specifically, the control unit 11 calculates an integrated value of pixel values ​​(brightness values, voxel values) of each pixel aligned in the projection direction according to the projection conditions specified in step S116 in the CT image of the region of interest, and sets the calculated integrated value as each pixel value of the DRR image of the region of interest. This generates a DRR image of the target region under projection conditions that are the same as the imaging conditions (position and angle relative to the X-ray source) of the target region (region of interest) in the X-ray image, and obtains a DRR image of the target region that is accurately aligned with the target region in the X-ray image. 4, the control unit 11 can generate training data for learning the learning model 12M by associating the half image of the X-ray image (hip joint X-ray frontal image) with the DRR image of the region of interest projected in the same direction as the X-ray image. Furthermore, the control unit 11 can generate training data for learning the learning model 12M by associating the X-ray image with the DRR image of the region of interest projected in the same direction as the X-ray image, and can generate training data for learning the bone mineral density estimation learning model 12M1 by associating the DRR image of the region of interest with the bone mineral density of the region of interest acquired in step S51 by executing S19 and S52 of FIG.

[0097] Through the above-described processing, training data is generated that associates an X-ray image of a target region (a region of interest that is a bone region) with a DRR image of the target region that is aligned with high accuracy relative to the target region in the X-ray image, and the training data is stored in the training DB 12b. Because the bone region is hard tissue, it is considered that it does not deform when the X-ray image is captured and when the CT image is captured. Therefore, by performing the alignment processing by optimizing the conditions (projection conditions) for generating a DRR image from a CT image as in this embodiment, it is possible to accurately align the contour of the target region (bone region) in the X-ray image with the contour of the target region in the DRR image generated from the CT image. Furthermore, the information processing device 10 of this embodiment is capable of executing the processing shown in FIG. 6 . By executing the learning processing of the learning model 12M by the processing shown in FIG. 6 using the training data generated as described above, it is possible to realize a learning model 12M that can predict a DRR image with high accuracy from an X-ray image. Furthermore, the information processing device 10 of this embodiment is capable of executing the processes shown in Figures 7, 12, 18, and 20, and by using the learning model 12M generated as described above, the bone density of the target area can be estimated with high accuracy using a DRR image predicted with high accuracy from an X-ray image, and highly accurate test results can be obtained.

[0098] In the above-described process shown in FIG. 4 , the process aligns the regions of interest between the X-ray image and the CT image based on the region of interest extracted from the CT image in step S15. However, this configuration is not limiting. For example, the process may align the bone regions between the X-ray image and the CT image based on the bone region extracted from the CT image in step S14. In this case, the control unit 11 of the information processing device 10 performs the alignment process shown in FIG. 22 after processing step S14 in FIG. 4 , and then performs steps S15 and S17 based on the CT image after the alignment process. Specifically, the control unit 11 extracts data of the region of interest (e.g., the proximal left femur) from the CT image after the alignment process (S15), and generates a DRR image projected from the CT image of the extracted region of interest in a projection direction according to the projection conditions identified in step S116 (S17). In the process shown in FIG. 22 , steps S111 to S116 are performed using the bone region extracted in step S14 as the region of interest. The control unit 11 then performs steps S18 to S19. This process also generates training data that associates an X-ray image of the target region (region of interest) with a DRR image of the target region that is highly accurately aligned with the target region in the X-ray image, and stores the training data in the training DB 12b. By using the training data generated in this manner to perform the learning process of the learning model 12M, the learning model 12M can be realized as a learning model that can predict a DRR image from an X-ray image with high accuracy. Furthermore, by using the learning model 12M generated in this manner, a DRR image can be predicted with high accuracy from an X-ray image, and the highly accurately predicted DRR image can be used to estimate the bone density of the target region with high accuracy.

[0099] In this embodiment, the processes other than the above-described alignment process are the same as those in the above-described embodiments, and the same effects as those in the above-described embodiments can be obtained. Furthermore, in this embodiment, spatial alignment between a target region in an X-ray image and a target region in a CT image can be performed with high accuracy. Therefore, by using such X-ray images and DRR images generated from CT images that have been aligned with the X-ray images with high accuracy as training data, a learning model 12M that predicts DRR images from X-ray images with high accuracy can be realized without requiring a large number of cases (training data). Furthermore, the modified examples described in the above-described embodiments can also be applied to this embodiment as appropriate.

[0100] (Embodiment 7) The configuration of the above-mentioned embodiment 6 is applied to the above-mentioned embodiments 1 and 3, and a process will be described in which an information processing device aligns the imaging target (target site or region of interest) in the X-ray image and the CT image based on the bone region in the X-ray image and the CT image, and generates training data to be used for learning the learning models 12M and 12Ma based on the aligned CT image. The information processing device of this embodiment has the same configuration as the information processing device 10 of embodiments 1 and 3, so a description of the configuration will be omitted. That is, the information processing device 10 of this embodiment has learning models 12M and 12Ma.

[0101] Fig. 25 is a flowchart showing an example of a training data generation processing procedure according to embodiment 7. The processing shown in Fig. 25 is the same as the processing shown in Fig. 4 except that steps S121 to S126 are added instead of steps S15 and S16, and steps S127 to S129 are added between steps S19 and S20. Explanations of the same steps as in Fig. 4 will be omitted.

[0102] The control unit 11 of the information processing device 10 of this embodiment executes steps S11 to S14 in FIG. 4 . In step S14, the control unit 11 extracts data of any bone region from the CT image based on the musculoskeletal labeled image generated from the CT image. For example, in the CT image shown in FIG. 5 , the control unit 11 may extract bone regions such as the pelvis, femur, and proximal femur, or multiple or all of these bone regions. After extracting the bone regions from the CT image, the control unit 11 executes the same processes as steps S111 to S116 in FIG. 22 based on the bone regions extracted from the CT image (S121 to S126). Specifically, the control unit 11 identifies the bone region in the X-ray image that is the same as the bone region extracted from the CT image (S121).

[0103] Next, the control unit 11 generates a pseudo-DRR image of the bone region based on the bone region in the CT image extracted in step S14 (S122). The control unit 11 then calculates a correlation value between the contour of the bone region in the X-ray image identified in step S121 and the contour of the bone region in the pseudo-DRR image generated in step S122 (S123), and determines whether the calculated correlation value is maximum (S124). If the control unit 11 determines that the correlation value is not maximum (S124: NO), it updates the projection conditions used to generate the pseudo-DRR image from the CT image of the bone region (S125) and repeats the processes of steps S122 to S124 under the updated projection conditions. The control unit 11 repeats the processes of steps S122 to S125 until it determines that the calculated correlation value is maximum. If it determines that the correlation value is maximum (S124: YES), i.e., if a pseudo-DRR image with the maximum correlation value has been generated, it identifies the projection conditions used at that time (S126). In this embodiment, the control unit 11 also performs the processing of steps S122 to S125 using a covariance matrix adaptive evolution strategy to identify projection conditions in the pseudo-DRR image that maximize the correlation value between the contour of the bone region in the X-ray image and the contour of the pseudo-DRR image.

[0104] The control unit 11 then generates a DRR image based on the CT image of the region of interest (here, a bone region such as the proximal femur) projected in a projection direction according to the projection conditions specified in step S126 (S17). That is, the control unit 11 generates a DRR image of the region of interest viewed from the same direction as the imaging direction of the region of interest in the X-ray image. The control unit 11 then extracts a half image including the left proximal femur from the frontal hip joint X-ray image acquired in step S11 (S18), and stores the extracted X-ray image (half image of the frontal hip joint X-ray image) and the DRR image of the region of interest generated in step S17 in association with each other in the training DB 12b as training data (S19). This allows the accumulation of training data that associates the X-ray image of the target region (the bone region of interest) with the DRR image of the target region generated from the CT image, which is aligned with the target region in the X-ray image with high accuracy.

[0105] Next, the control unit 11 executes the same processes as steps S71 to S73 in FIG. 15 (S127 to S129). Specifically, the control unit 11 erases data on bone regions from the CT image based on the musculoskeletal label image and extracts data on muscle regions (S127). Here, the control unit 11 also defines regions of interest for each muscle type and extracts data (CT images) for each region of interest (muscle region). The control unit 11 then generates DRR images projected in the projection direction according to the projection conditions specified in step S126 based on the data (CT images) of the extracted regions of interest (muscle regions) (S128). The control unit 11 then associates the X-ray image acquired in step S11 with the DRR image of the region of interest (muscle region) generated in step S128 and stores them as training data in the training DB 12b (S129). The control unit 11 then proceeds to the processing of step S20.

[0106] The above-described process aligns the X-ray and CT images with respect to the bone region, and generates training data that can efficiently train the learning models 12M and 12Ma of this embodiment based on the aligned X-ray and CT images. By executing the learning process shown in FIG. 6 using the training data generated in this manner, the information processing device 10 of this embodiment can generate a learning model 12M that can accurately predict, from X-ray images, a DRR image of the bone region that is the subject of the X-ray images, and a learning model 12Ma that can accurately predict, from X-ray images, a DRR image of the muscle region that is the subject of the X-ray images. Furthermore, by using the learning models 12M and 12Ma, the information processing device 10 of this embodiment can generate a DRR image of the bone region that is accurately predicted from X-ray images, and a DRR image of the muscle region that is accurately predicted from X-ray images, thereby estimating the bone density and muscle mass of the target region with high accuracy.

[0107] 26A and 26B are explanatory diagrams showing the effect of bone region-based registration processing. Fig. 26A is a diagram showing the relationship between the muscle mass (specifically, lean muscle mass) of the muscle region calculated (estimated) from the generated predicted DRR image and the muscle mass (specifically, lean muscle mass) of the muscle region measured from the CT image, calculated (estimated) from the predicted DRR image using a learning model 12Ma trained on training data that associates X-ray images with DRR images generated from CT images, without performing bone region-based registration processing on the X-ray images and CT images shown in this embodiment. The vertical axis of the diagram in Fig. 26A shows the lean muscle mass estimated from the predicted DRR image, and the horizontal axis shows the lean muscle mass measured from the CT image. The left side of Fig. 26A shows a diagram of the gluteus medius, and the right side shows a diagram of the iliacus. 26B is a diagram showing the relationship between the muscle mass (specifically, lean muscle mass) of the muscle region calculated (estimated) from the predicted DRR image and the muscle mass (specifically, lean muscle mass) of the muscle region measured from the CT image, calculated (estimated) from the predicted DRR image using the learning model 12Ma trained with training data that correlates the X-ray image and the DRR image generated from the CT image after performing a bone region-based registration process on the X-ray image and the CT image shown in this embodiment. 26B also shows the lean muscle mass estimated from the predicted DRR image on the vertical axis and the lean muscle mass measured from the CT image on the horizontal axis, with the gluteus medius diagram shown on the left side of FIG. 26B and the iliacus diagram shown on the right side. The charts shown in Figures 26A and 26B show the comparison results (verification results) between the lean muscle mass estimated from predicted DRR images predicted from X-ray images collected from 525 patients using a learning model 12Ma trained using 390 pairs of training data generated from X-ray images and CT images collected from patients with osteoarthritis of the hip, and the lean muscle mass measured from the CT images of each patient.

[0108] As can be seen from the diagrams on the left side of Figure 26A and Figure 26B, for the lean muscle mass of the gluteus medius, the correlation coefficient (PCC: Pearson Correlation Coefficient) between the estimated value from the predicted DRR image and the measured value from the CT image was 0.340 when the alignment of this embodiment was not performed, and the correlation coefficient was 0.776 when the alignment of this embodiment was performed, resulting in a higher correlation value. Similarly, as can be seen from the diagrams on the right side of Figure 26A and Figure 26B, for the lean muscle mass of the iliacus, the correlation coefficient was 0.250 when the alignment of this embodiment was not performed, and the correlation coefficient was 0.804 when the alignment of this embodiment was performed, resulting in a higher correlation value. Therefore, by training the learning model 12Ma using training data generated from X-ray images and CT images after the subject has been aligned by the alignment process of this embodiment, it is possible to realize a learning model 12Ma that can predict DRR images that can estimate muscle mass to the same extent as measured values ​​from CT images, even if the learning model 12Ma has been trained with a small amount of training data. Thus, the workload in the training data collection process and learning process can be reduced.

[0109] The diagrams in Figures 26A and 26B show that more accurate lean muscle mass is obtained for the gluteus medius and iliacus muscles when the alignment of this embodiment is performed. However, in a verification process conducted by the inventors, similar results were obtained for the muscle volume of the gluteus medius and iliacus muscles. The lean muscle mass is calculated by calculating the muscle-to-fat ratio for each pixel based on the CT value (pixel value) within each muscle region (individual muscle region) in the CT image, calculating the muscle mass for each pixel based on the calculated ratio, and then summing the muscle masses for each pixel. Muscle volume indicates the volume of each muscle region (individual muscle region) in the CT image.

[0110] In this embodiment, by performing a registration process on the X-ray image and the CT image based on the bone region, the X-ray image and the CT image after the registration process are accurately registered with respect to the muscle region. By training the learning model 12Ma using training data based on the muscle region in the registered X-ray image and the DRR image of the muscle region generated from the CT image, the learning model 12Ma can predict a DRR image of the muscle region that can estimate muscle mass equivalent to the muscle mass (e.g., lean muscle mass, muscle volume) of each muscle region measured from the CT image. Furthermore, in this embodiment, by erasing the data of the bone region from the registered CT image, a CT image (data of the muscle region) that eliminates the influence of the bone region is extracted, and by using such a CT image, training data used for training the learning model 12Ma can be generated with high accuracy. Therefore, in this embodiment, not only can bone density be estimated with high accuracy using a predicted DRR image of the bone region generated from an X-ray image using learning model 12M, but muscle mass can also be estimated with high accuracy using a predicted DRR image of the muscle region generated from an X-ray image using learning model 12Ma.

[0111] In this embodiment, when generating training data used to train the learning models 12M and 12Ma, the present embodiment is the same as the above-described embodiments except for the process of aligning the imaging target in the X-ray image and the CT image based on the bone region, and the same effects as the above-described embodiments can be obtained. Furthermore, in this embodiment, not only the training data used to train the learning model 12M but also the training data used to train the learning model 12Ma are generated based on the X-ray image and the CT image that have been aligned based on the bone region, so that the learning model 12Ma can be realized to predict the DRR image of the muscle region in the X-ray image with high accuracy from the X-ray image. In this embodiment, the modified examples described as appropriate in the above-described embodiments can also be applied.

[0112] (Embodiment 8) A modified example of the learning model 12Ma that generates a predicted DRR image of a muscle region (e.g., gluteus maximus, gluteus medius, iliacus, etc.) included in a target region (e.g., lower abdomen, buttocks, etc.) from an X-ray image of the target region in the above-described embodiments 3 and 7 will be described. The information processing device of this embodiment has the same configuration as the information processing device 10 of embodiments 1, 3, and 7, and therefore a description of the configuration will be omitted.

[0113] FIG. 27 is an explanatory diagram showing an example configuration of the learning model 12Mb according to the eighth embodiment. The learning model 12Mb is a trained model that receives an X-ray image of a target region as input, performs calculations to predict DRR images and muscle mass of muscle regions included in the target region based on the input X-ray image, and outputs the calculation results. The learning model 12Mb shown in FIG. 27 receives a frontal X-ray image of a hip joint as input and predicts a DRR image and muscle mass of the gluteus maximus. Note that the learning model 12Mb is not limited to this configuration. For example, the learning model 12Mb may receive a frontal X-ray image of a hip joint as input and predict DRR images and muscle mass of other muscle regions, such as the gluteus medius and iliacus, or may predict DRR images and muscle mass for each of multiple muscle regions. The learning model 12Mb may also receive a frontal X-ray image of a chest as input and predict DRR images and muscle mass of muscle regions, such as the pectoralis major, pectoralis minor, subclavius, serratus anterior, intercostal muscles, trapezius, latissimus dorsi, teres major, and erector spinae. The muscle mass can be information indicating any quantity related to muscle, such as lean muscle mass, muscle volume, or muscle density.

[0114] The learning model 12Mb of this embodiment has an image conversion layer 12Mb1 and a muscle mass prediction layer 12Mb2. An X-ray image, which is input data to the learning model 12Mb, is input to the image conversion layer 12Mb1. Based on the input X-ray image, the image conversion layer 12Mb1 generates and outputs a DRR image of a muscle region (e.g., the gluteus maximus) included in the X-ray image. The DRR image of the muscle region generated by the image conversion layer 12Mb1 is input to the muscle mass prediction layer 12Mb2. Based on the input DRR image of the muscle region, the muscle mass prediction layer 12Mb2 predicts and outputs the muscle mass (e.g., lean muscle mass, muscle volume, muscle density, etc.) of the muscle region.

[0115] The image conversion layer 12Mb1 may be configured using a model based on, for example, a GAN such as pix2pix, CycleGAN, or StarGAN, a neural network such as VAE or CNN (e.g., U-net), or other learning algorithm, or may be configured using a combination of multiple learning algorithms. When the image conversion layer 12Mb1 is configured using pix2pix, the image conversion layer 12Mb1 may have a configuration similar to the learning model 12Ma shown in FIG. 14. The muscle mass prediction layer 12Mb2 may be configured using an algorithm such as CNN, random forest, SVM (Support Vector Machine), or Transformer, or may be configured using a combination of multiple algorithms. The learning model 12Mb may not have the muscle mass prediction layer 12Mb2, and instead of the muscle mass prediction layer 12Mb2, the control unit 11 of the information processing device 10 may perform processing to calculate the muscle mass of the muscle region based on the sum of the pixel values ​​(brightness values) of each pixel in the DRR image of the muscle region generated by the image conversion layer 12Mb1.

[0116] The 12Mb learning model of this embodiment is generated by learning using training data that associates training X-ray images (frontal hip joint X-ray images) with correct DRR images of the gluteus maximus and the muscle mass of the gluteus maximus. The training X-ray images, correct DRR images, and muscle mass are preferably frontal hip joint X-ray images, DRR images, and muscle mass of subjects diagnosed with normal bone density, osteoporosis, or bone loss using a DXA device. The correct DRR images of the gluteus maximus can be DRR images of the gluteus maximus (muscle region) generated from CT images of the buttocks (lower abdomen) taken with an X-ray CT device. The correct muscle mass of the gluteus maximus can be calculated by summing the muscle mass of each pixel calculated based on the muscle-to-fat ratio in each pixel from the CT value (brightness value) within the muscle region (gluteus maximus) in the CT image, or by measuring the muscle mass of the gluteus maximus using a DXA device.

[0117] In this embodiment, the learning model 12Mb learns to output the correct DRR image and muscle mass included in the training data when an X-ray image included in the training data is input. During the learning process, the learning model 12Mb performs calculations based on the input X-ray image, predicts the DRR image and muscle mass of the gluteus maximus muscle in the input X-ray image, and outputs the prediction result. The learning model 12Mb then compares the predicted DRR image with the correct DRR image and the predicted muscle mass with the correct muscle mass, and optimizes parameters such as the weights (coupling coefficients) between nodes in the image conversion layer 12Mb1 and the muscle mass prediction layer 12Mb2 using steepest descent, backpropagation, or the like, so that the predicted muscle mass and the correct muscle mass are similar. That is, in this embodiment, during the learning process of the learning model 12Mb, the difference (error, loss related to muscle mass) between the muscle mass based on the DRR image of the muscle region generated by the learning model 12Mb and the correct muscle mass can be fed back. Therefore, according to the learning process of this embodiment, when an X-ray image is input, a learning model 12 Mb is generated that predicts the DRR image and muscle mass of the muscle region contained in the X-ray image with higher accuracy.

[0118] Note that the learning model 12Mb of this embodiment may be configured without the muscle mass prediction layer 12Mb2. In this case, during the learning process of the learning model 12Mb, the control unit 11 of the information processing device 10 inputs an X-ray image to the learning model 12Mb, acquires a DRR image of the muscle region from the learning model 12Mb, and calculates the muscle mass of the muscle region from the acquired DRR image. The control unit 11 may then calculate the difference between the calculated muscle mass and the correct muscle mass, and train the learning model 12Mb (image conversion layer 12Mb1) so that the calculated difference is small (so that the calculated muscle mass approximates the correct muscle mass). Even in this learning process, the difference (error, muscle mass loss) between the muscle mass based on the DRR image of the muscle region generated by the learning model 12Mb and the correct muscle mass can be fed back, thereby generating a learning model 12Mb that can accurately predict the DRR image and muscle mass of the muscle region included in the input X-ray image.

[0119] The information processing device 10 of this embodiment is capable of executing the process shown in FIG. 25 , which generates training data for learning the learning model 12Mb through the process shown in FIG. 25 and executes the learning process for the learning model 12Mb through the process shown in FIG. 6 . In the process shown in FIG. 25 , after processing step S128, the control unit 11 of the information processing device 10 calculates the muscle mass of the muscle region based on the DRR image of the muscle region generated in step S128. Then, in step S129, the control unit 11 associates the X-ray image acquired in step S11, the DRR image of the region of interest (muscle region) generated in step S128, and the muscle mass calculated from the DRR image, and stores the associated data in the training DB 12b as training data. The muscle mass of the muscle region may be measured by a DXA device. In this case, the control unit 11 may acquire the muscle mass measured by the DXA device along with the X-ray image and CT image in step S11, and use the acquired muscle mass as training data. 6, the control unit 11 reads out one set of an X-ray image, a DRR image, and muscle mass stored in the storage unit 12 in step S31, and performs a learning process for the learning model 12Mb using the read-out training data in step S32. Furthermore, the information processing device 10 of this embodiment can execute the process shown in Fig. 16, and by using the learning model 12Mb generated as described above, it is possible to predict with high accuracy a DRR image of a muscle region in an X-ray image and to estimate with high accuracy the muscle mass of the muscle region using the predicted DRR image.

[0120] 28A and 28B are explanatory diagrams showing the effect of feedback processing of muscle mass loss. Fig. 28A is a diagram showing the relationship between the muscle mass (specifically, lean muscle mass) of a muscle region calculated (estimated) from the predicted DRR image generated from an X-ray image using a learning model (e.g., learning model 12Ma) trained without feedback processing of muscle mass loss as described above, and the muscle mass (specifically, lean muscle mass) of the muscle region measured from a CT image. The diagram in Fig. 28A shows the lean muscle mass estimated from the predicted DRR image on the vertical axis and the lean muscle mass measured from the CT image on the horizontal axis. The left side of Fig. 28A shows a diagram of the gluteus medius, and the right side shows a diagram of the iliacus. Fig. 28B is a graph showing the relationship between the muscle mass (specifically, lean muscle mass) of a muscle region calculated (estimated) from the predicted DRR image generated from an X-ray image using a learning model 12Mb trained by performing feedback processing on muscle mass loss as described above, and the muscle mass (specifically, lean muscle mass) of the muscle region measured from a CT image. The graph in Fig. 28B shows the lean muscle mass estimated from the predicted DRR image on the vertical axis and the lean muscle mass measured from the CT image on the horizontal axis. The left side of Fig. 28B shows a graph of the gluteus medius, and the right side shows a graph of the iliacus. The charts shown in Figures 28A and 28B show the comparison results (verification results) between the lean muscle mass estimated from predicted DRR images predicted from X-ray images collected from 525 patients and the lean muscle mass measured from the CT images of each patient, using a 12 Mb learning model trained using 390 pairs of training data generated from X-ray images and CT images collected from patients with osteoarthritis of the hip.

[0121] As can be seen from the left side of Figure 28A and the left side of Figure 28B, for the lean muscle mass of the gluteus medius, when the above-mentioned feedback processing of muscle mass loss was not performed, the correlation coefficient between the estimated value from the predicted DRR image and the measured value from the CT image was 0.776, while when the feedback processing was performed, the correlation coefficient was 0.874, indicating that a higher correlation value was obtained when feedback processing of muscle mass loss was performed. Similarly, as can be seen from the right side of Figure 28A and the right side of Figure 28B, for the lean muscle mass of the iliacus, when feedback processing of muscle mass loss was not performed, the correlation coefficient was 0.804, while when feedback processing was performed, the correlation coefficient was 0.861, indicating that a higher correlation value was obtained when feedback processing of muscle mass loss was performed. Therefore, by performing feedback processing of muscle mass loss during the learning process of the learning model 12Mb, even a learning model 12Mb trained with a small amount of training data can be realized that can predict DRR images that can estimate muscle mass to the same extent as measured values ​​from CT images.

[0122] In this embodiment, the graphs in Figures 28A and 28B show that more accurate lean muscle mass can be obtained for the gluteus medius and iliacus muscles by performing feedback processing on muscle mass loss during the learning process of the 12Mb learning model, but the inventor's verification process also yielded similar results for the muscle volume of the gluteus medius and iliacus muscles.

[0123] This embodiment is the same as the above-described embodiments, and provides the same effects as the above-described embodiments, except for the process of feeding back the error (loss) between the correct muscle mass and the muscle mass calculated from the DRR image of the muscle region generated by the learning model 12Mb (image conversion layer 12Mb1) during learning of the learning model 12Mb. Furthermore, in this embodiment, in the learning process of the learning model 12Mb, in addition to the error between the DRR image generated by the image conversion layer 12Mb1 and the correct DRR image, the error (loss) in muscle mass can be fed back. This allows for the realization of a learning model 12Mb that can predict the DRR image of the muscle region in the X-ray image with higher accuracy from the X-ray image. The modified examples described in the above-described embodiments can also be applied to this embodiment.

[0124] In each of the above-described embodiments, simple X-ray images of the target area captured by an X-ray device (X-ray device) are used as input data for the learning models 12M, 12Ma, and 12Mb. However, this configuration is not limited to this. For example, X-ray images obtained by a DXA device may be used as input data for the learning models 12M, 12Ma, and 12Mb. Even with this configuration, the same processing can be performed and the same effects can be obtained as when normal X-ray images (X-ray images captured by an X-ray device) are used as input data. Furthermore, in each of the above-described embodiments, instead of the correct DRR images (information on the tissue volume of the target area obtained from CT images) used as training data for the learning models 12M, 12Ma, and 12Mb, two-dimensional images (information on the tissue volume) obtained from three-dimensional images that can estimate the muscle and bone composition of the target area, such as MRI (Magnetic Resonance Imaging) images, may be used as training data. Even with this configuration, it is possible to execute the same processing as when using a DRR image obtained from a CT image, and similar effects can be obtained.

[0125] The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0126] REFERENCE SIGNS LIST 10 Information processing device 11 Control unit 12 Memory unit 13 Communication unit 14 Input unit 15 Display unit 12a Medical image DB 12b Training DB 12M Learning model 12Ma Learning model 12M1 Learning model for bone density estimation

Claims

1. obtaining training data including an X-ray image of a target area and information regarding tissue volume obtained from a computed tomography (CT) image of the target area; Using the acquired training data, a learning model is generated that outputs information regarding the amount of body tissue in a target area in an X-ray image when the X-ray image is input. A program that causes a computer to carry out processing.

2. When the X-ray image is input, the learning model outputs an image representing the amount of body tissue of the target part. The program according to claim 1 , which causes the computer to execute a process.

3. performing alignment between a position of the target site based on the CT image and a position of the target site based on the X-ray image; The training data includes information regarding the amount of body tissue of the target area obtained from the CT images after registration.

3. The program according to claim 1, which causes the computer to execute a process.

4. Based on the CT image, the target site in the CT image is classified into a plurality of regions including a bone region and a muscle region; obtaining the training data including information regarding bone density of bone regions in the classified CT images; Using the training data, the learning model is generated to output information regarding bone density of the bone region in an X-ray image when the X-ray image is input.

3. The program according to claim 1, which causes the computer to execute a process.

5. Based on the CT image, the target site in the CT image is classified into a plurality of regions including a bone region and a muscle region; obtaining the training data including information about muscle mass of muscle regions in the classified CT images; Using the training data, the learning model is generated to output information regarding the muscle mass of the muscle region in an X-ray image when the X-ray image is input.

3. The program according to claim 1, which causes the computer to execute a process.

6. Identifying a bone region in the CT image and the bone region in the X-ray image; generating a CT image of the bone region viewed from a direction that matches the imaging direction of the bone region in the X-ray image from the CT image; By aligning the bone region in the generated CT image with the bone region in the X-ray image, the position of the target part based on the CT image is aligned with the position of the target part based on the X-ray image. The program according to claim 3, which causes the computer to execute a process.

7. Acquire training data including information on the body tissue volume of the target site obtained from a CT image of the target site and information on the body tissue volume of a site different from the target site; Using the acquired training data, a second learning model is generated that outputs information about the body tissue mass of a site different from the target site when information about the body tissue mass of the target site is input.

3. The program according to claim 1, which causes the computer to execute a process.

8. specifying a projection condition that maximizes a correlation value between an image obtained by projecting a bone region included in the target site in the CT image and a bone region included in the target site in the X-ray image; The training data includes information regarding the amount of body tissue of the target area obtained from a projection image obtained by projecting the target area in the CT image under a specified projection condition.

3. The program according to claim 1, which causes the computer to execute a process.

9. erasing data of a bone region from a projection image obtained by projecting the target portion in the CT image under the specified projection condition; obtaining the training data including information about muscle mass of muscle regions in the projection images from which data about the bone regions has been removed; Using the training data, the learning model is generated to output information regarding muscle mass of a muscle region in an X-ray image when the X-ray image is input. The program according to claim 8, which causes the computer to execute a process.

10. the training data includes an X-ray image of the target area, an image representing a muscle region of the target area obtained from a CT image of the target area, and a muscle mass of the muscle region; the learning model is configured to output, when an X-ray image is input, an image showing a muscle region in the X-ray image; The learning model is trained so that a muscle mass calculated based on an image showing a muscle region in an X-ray image output by the learning model when an X-ray image included in the training data is input is approximated to a muscle mass included in the training data.

3. The program according to claim 1, which causes the computer to execute a process.

11. Obtaining an X-ray image of the target area; The learning model is trained using training data including an X-ray image of a target part and information about the amount of body tissue obtained from a CT image of the target part, and when an X-ray image is input, the learning model outputs information about the amount of body tissue of the target part in the X-ray image, and the acquired X-ray image is input to output information about the amount of body tissue of the target part. A program that causes a computer to carry out processing.

12. The information regarding the body tissue amount to be output is an image representing the body tissue amount of the target part. The program according to claim 11.

13. The information on the body tissue mass to be output is the bone density or muscle mass of the target part.

13. The program according to claim 11 or 12.

14. Further outputting information regarding the amount of body tissue in a region different from the target region in the X-ray image. The program according to claim 11 or 12, which causes the computer to execute a process.

15. The learning model is A CT image is generated in a state in which the bone region in the CT image is viewed from a direction that matches the imaging direction of the bone region specified in the X-ray image, and the bone region in the generated CT image is aligned with the bone region in the X-ray image, thereby aligning the position of the target part based on the CT image with the position of the target part based on the X-ray image, and learning is performed using the training data including information on the body tissue volume of the target part obtained from the CT image after alignment.

13. The program according to claim 11 or 12.

16. The learning model is trained to output information about the muscle mass of the muscle region in the X-ray image when an X-ray image is input, using the training data including information about the muscle mass of the muscle region in the projection image obtained by projecting the target part in the CT image under projection conditions that maximize a correlation value between an image obtained by projecting the bone region included in the target part in the CT image and the bone region included in the target part in the X-ray image, from which data about the bone region has been erased.

13. The program according to claim 11 or 12.

17. The learning model is trained using training data including an X-ray image of the target part, an image showing a muscle region of the target part obtained from a CT image of the target part, and a muscle mass of the muscle region, so that a muscle mass calculated based on an image showing a muscle region in the X-ray image, which is output when an X-ray image included in the training data is input, is approximated to the muscle mass included in the training data.

13. The program according to claim 11 or 12.

18. Obtaining an X-ray image of the target area; The learning model is trained using training data including an X-ray image of a target part and information about the amount of body tissue obtained from a CT image of the target part, and when an X-ray image is input, the learning model outputs information about the amount of body tissue of the target part in the X-ray image, and the acquired X-ray image is input to output information about the amount of body tissue of the target part. An information processing method in which processing is performed by a computer.

19. In an information processing device having a control unit, The control unit: Obtaining an X-ray image of the target area; The learning model is trained using training data including an X-ray image of a target part and information about the amount of body tissue obtained from a CT image of the target part, and when an X-ray image is input, the learning model outputs information about the amount of body tissue of the target part in the X-ray image, and the acquired X-ray image is input to output information about the amount of body tissue of the target part. Information processing device.

20. The learning model is A CT image is generated in a state in which the bone region in the CT image is viewed from a direction that matches the imaging direction of the bone region specified in the X-ray image, and the bone region in the generated CT image is aligned with the bone region in the X-ray image, thereby aligning the position of the target part based on the CT image with the position of the target part based on the X-ray image, and learning is performed using the training data including information on the body tissue volume of the target part obtained from the CT image after alignment.

20. The information processing method according to claim 18.