Program, information processing method, and information processing device

Abstract

Provided is a program or the like with which it is possible to acquire information relating to the amount of body tissue from an X-ray image with high accuracy using a small number of cases. A computer acquires training data including an X-ray image of a subject site, and information related to a body tissue amount obtained from a Compressed Tomography (CT) image of the subject site. Then, the computer uses the acquired training data to generate a learning model that outputs information relating to the body tissue amount of the target site in the X-ray image when the X-ray image is input.

Description

[0001] Technical Area

[0002] The present disclosure relates to a program, an information processing method, and an information processing device. Background Art

[0003] Fractures in the elderly can lead to a decrease in living functions, and there is a possibility that they will need care. Therefore, it is considered important to diagnose osteoporosis by measuring bone density and to provide appropriate treatment at an early stage to prevent fractures. In the measurement of bone density, it is recommended to use a DXA (Dual-energy X-ray Absorptiometry) device, which measures bone density (bone mass) based on the difference in transmittance of X-rays at two energy levels. However, the DXA device is a bed-type device that takes pictures of the subject in a supine position, so it is necessary to ensure the installation space. In addition, although the price is high, the use is limited, so the popularity is low.

[0004] On the other hand, X-ray devices are installed in a large number of medical institutions. Therefore, non-patent documents 1 to 3 propose the following technology: a model for predicting bone density from X-ray images is generated by learning pairs of X-ray images and measurement results (bone density) based on DXA devices using training data.

[0005] Prior art literature

[0006] Non-patent literature

[0007] Non-patent document 1: Chen-I Hsieh et al., "Automated bone mineral density prediction and fracture risk assessment using plain radiographs via deep learning", NATURE COMMUNICATIONS, 12: 5472 (2021)

[0008] Non-patent document 2: Ryoungwoo Jang et al., "Prediction of osteoporosis from simple hip radiography using deep learning algorithm", Scientific Reports, 11:19997(2021)

[0009] Non-patent document 3: Norio Yamamoto et al., "Deep Learning for OsteoporosisClassification Using Hip Radiographs and Patient Clinical Covariates", Biomolecules 2020,10,1534 Summary of the invention

[0010] Problem that the invention aims to solve

[0011] However, the techniques disclosed in Non-Patent Documents 1 to 3 require learning on a large amount of training data, and the processing load of the training data collection process and the learning process is large.

[0012] An object of the present disclosure is to provide a program or the like that can acquire information related to body tissue volume from an X-ray image with high accuracy using a small number of cases.

[0013] Solutions for solving problems

[0014] A program according to one embodiment of the present disclosure causes a computer to perform the following processing: acquiring training data, the training data including an X-ray image of an object part and information related to body tissue volume obtained based on a CT (Computed Tomography) image of the object part, and using the acquired training data, generating a learning model that outputs information related to body tissue volume of the object part in the X-ray image when an X-ray image is input.

[0015] Effects of the Invention

[0016] According to one embodiment of the present disclosure, it is possible to obtain information related to body tissue volume from X-ray images with high accuracy using a small number of cases. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a block diagram showing a configuration example of an information processing device.

[0018] Figure 2 It is an explanatory diagram showing the outline of the learning model.

[0019] Figure 3A It is an explanatory diagram showing the outline of the learning model.

[0020] Figure 3B It is an explanatory diagram showing the outline of the learning model.

[0021] Figure 4 This is a flowchart showing an example of a procedure for generating training data.

[0022] Figure 5 This is a diagram for explaining the process of generating a DRR image for training.

[0023] Figure 6 This is a flowchart showing an example of a process procedure for generating a learning model.

[0024] Figure 7 This is a flowchart showing an example of a bone density estimation process procedure.

[0025] Figure 8 It is an explanatory diagram showing a screen example.

[0026] Fig. 9A This is a graph showing the relationship between the bone density estimated from the predicted DRR image and the bone density measured by the DXA device.

[0027] Fig. 9B This is a graph showing the relationship between the bone density estimated from the predicted DRR image and the bone density measured by the DXA device.

[0028] Fig.10 This is an explanatory diagram showing an overview of the learning model according to the second embodiment.

[0029] Fig.11 This is a flowchart showing an example of a training data generation process procedure according to the second embodiment.

[0030] Fig.12 This is a flowchart showing an example of a bone density estimation process procedure according to the second embodiment.

[0031] Fig.13 It is an explanatory diagram showing a screen example.

[0032] Fig.14 This is an explanatory diagram showing an overview of a learning model according to the third embodiment.

[0033] Fig.15 This is a flowchart showing an example of a training data generation process procedure according to the third embodiment.

[0034] Fig.16 This is a flowchart showing an example of a muscle mass estimation process procedure.

[0035] Fig.17 It is an explanatory diagram showing a screen example.

[0036] Fig.18 This is a flowchart showing an example of a bone density estimation process procedure according to the fourth embodiment.

[0037] Fig.19 It is an explanatory diagram showing a screen example.

[0038] Fig. 20 This is a flowchart showing an example of the bone density and muscle mass estimation processing steps in the fifth embodiment.

[0039] Fig.21 It is an explanatory diagram showing a screen example.

[0040] Fig. 22 This is a flowchart showing an example of the positioning process procedure.

[0041] Fig.23A This is a diagram for explaining the alignment process.

[0042] Fig. 23B This is a diagram for explaining the alignment process.

[0043] Fig.24A This is a diagram for explaining the alignment process.

[0044] Fig. 24B This is a diagram for explaining the alignment process.

[0045] Fig.25 This is a flowchart showing an example of a training data generation process procedure according to the seventh embodiment.

[0046] Fig.26A It is an explanatory diagram showing the effect of the positioning process based on the bone region.

[0047] Fig.26B It is an explanatory diagram showing the effect of the positioning process based on the bone region.

[0048] Fig. 27 This is an explanatory diagram showing an example of the configuration of a learning model according to the eighth embodiment.

[0049] Fig.28A 2 is an explanatory diagram showing the effect of feedback processing related to loss of muscle mass.

[0050] Fig.28B 2 is an explanatory diagram showing the effect of feedback processing related to loss of muscle mass. DETAILED DESCRIPTION

[0051] Hereinafter, the program, the information processing method, and the information processing device of the present disclosure will be described based on the drawings showing embodiments thereof.

[0052] (Implementation Method 1)

[0053] An information processing device for estimating the bone density (information related to body tissue mass) of an object part based on an X-ray image obtained by photographing the object part through an X-ray device is described. The bone density of the lumbar spine or the vicinity of the femur is generally used in the diagnosis of osteopenia and osteoporosis. Therefore, in this embodiment, the following configuration is described: the object part is set to the vicinity of the femur, and the bone density of the vicinity of the femur is estimated based on an X-ray image (hip joint X-ray front image) that includes the vicinity of the femur within the photographing range. However, the object part is not limited to the vicinity of the femur, and may also be other parts such as the lumbar spine and thoracic spine.

[0054] Figure 1 : is a block diagram showing an example of the configuration of an information processing device. The information processing device 10 is a device capable of performing various information processing and sending and receiving information, such as a personal computer, a server computer, a workstation, etc. The information processing device 10 is set up in a medical institution, an inspection institution, a research institution, etc. for use. The information processing device 10 can be a multi-computer composed of a plurality of computers, or it can be implemented using a virtual machine virtually constructed in one device. In the case where the information processing device 10 is composed of a server computer, the information processing device 10 can be a local server set up in a medical institution, etc., or it can be a cloud server that is connected to a network such as the Internet for communication. In the following, the information processing device 10 is set as a computer for explanation.

[0055] The information processing device 10 estimates the bone density of the femur vicinity based on the hip joint X-ray front image, for example. Specifically, as described later, the information processing device 10 prepares a learning model 12M in advance, and the learning model 12M performs machine learning of predetermined training data in advance, takes the hip joint X-ray front image as input, and outputs information related to the bone density of the femur vicinity in the hip joint X-ray front image (information related to body tissue volume). Then, the information processing device 10 inputs the hip joint X-ray front image into the learning model 12M, thereby obtaining information related to the bone density of the femur vicinity from the learning model 12M. In this embodiment, as information related to bone density, a projection image of a three-dimensional region of the femur vicinity in a CT image, namely a DRR image (Digital Reconstructed Radiograph: an X-ray image obtained by projection simulation from a three-dimensional region (region of interest) of a determined part of a CT image) is used. Each pixel value of the CT image is a CT value corresponding to the bone density, and therefore, the DRR image generated based on a determined bone region in the CT image can represent the distribution of bone density, for example, the higher the bone density, the higher the brightness value (pixel value). Therefore, the learning model 12M of the present embodiment is configured to predict and output a DRR image (an image representing the amount of body tissue) of the femur vicinity contained in the X-ray image when a frontal X-ray image of the hip joint is input. In addition, the information processing device 10 calculates the bone density of the femur vicinity based on the DRR image predicted using the learning model 12M, thereby being able to obtain the bone density from the X-ray image. If the bone density decreases, the shooting state in the X-ray image will change. Therefore, in the present embodiment, the learning model 12M can be used to predict a DRR image corresponding to the state of the bone density of the bone region in the X-ray image.

[0056] The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and the above-mentioned units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), and an AI chip (semiconductor for AI). The control unit 11 executes the processing to be performed by the information processing device 10 by appropriately executing the program 12P stored in the storage unit 12.

[0057] The storage unit 12 includes RAM (Random Access Memory), flash memory, hard disk, SSD (Solid State Drive), etc. The storage unit 12 pre-stores the program 12P (program product) executed by the control unit 11 and various data required to execute the program 12P. In addition, the storage unit 12 temporarily stores the data generated when the control unit 11 executes the program 12P. Moreover, the storage unit 12 stores the learning model 12M described later. The learning model 12M is a model obtained by learning in a manner that outputs a DRR image of the vicinity of the femur contained in the X-ray image when a frontal X-ray image of the hip joint is input. It is assumed that the learning model 12M is used as a program module constituting artificial intelligence software. In the storage unit 12, as information defining the learning model 12M, information such as the layers possessed by the learning model 12M, information on the nodes constituting each layer, and weights (coupling coefficients) between nodes are stored.

[0058] In addition, the storage unit 12 stores a medical image DB 12a and a training DB 12b. The medical image DB 12a stores hip joint X-ray front images and CT images prepared for learning the learning model 12M in a corresponding manner. The medical images used for learning include hip joint X-ray front images and CT images of subjects diagnosed as any one of normal bone density cases, osteoporosis cases, and osteopenia cases using a DXA device. The training DB 12b stores training data used for the learning process of the learning model 12M, and the information processing device 10 stores the training data in the training DB 12b by performing the generation process of the training data described later. The learning model 12M, the medical image DB 12a, and the training DB 12b can also be stored in other storage devices connected to the information processing device 10, and can also be stored in other storage devices with which the information processing device 10 can communicate.

[0059] The communication unit 13 is a communication module for connecting to a network N such as the Internet or a LAN (Local Area Network) through wired communication or wireless communication, and performs information transmission and reception with other devices via the network N. The input unit 14 receives operation inputs from the user and sends control signals corresponding to the operation contents to the control unit 11. The display unit 15 is a liquid crystal display or an organic EL display, etc., and displays various information according to instructions from the control unit 11. A part of the input unit 14 and the display unit 15 may be an integral touch panel. It should be noted that the input unit 14 and the display unit 15 are not required, and the information processing device 10 may also be a structure that receives operations through a connected computer and outputs the information to be displayed to an external display device.

[0060] The reading unit 16 reads information stored in a portable storage medium 10a such as a CD (Compact Disc), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, an SD card, a micro SD card, a Compact Flash card (Compact Flash, a registered trademark), etc. The program 12P (program product) and various data stored in the storage unit 12 can be read from the portable storage medium 10a by the control unit 11 via the reading unit 16 and stored in the storage unit 12, or can be downloaded from other devices by the control unit 11 via the communication unit 13 and stored in the storage unit 12.

[0061] Figure 2 to Figure 3B 12M is an explanatory diagram showing the outline of the learning model 12M. Figure 2 The following conceptually shows how a DRR image of a region of interest near the femur is predicted from a hemisection image of a hip joint X-ray frontal image. Figure 3A and Figure 3B The following diagram conceptually shows the situation when the learning model 12M is learning. Figure 3A shows what happens when the recognizer learns, Figure 3B shows what happens when the generator learns. Figure 2 As shown, the learning model 12M is a model that has been learned by taking as input a half-cut image of the vicinity of the femur on one side of the hip joint X-ray front image, which is an object of bone density estimation, and predicting a DRR image of the region of interest. It should be noted that the half-cut X-ray image input to the learning model 12M can be an X-ray image containing the vicinity of the left femur, or an X-ray image containing the vicinity of the right femur. When the X-ray image containing the vicinity of the right femur is input to the learning model 12M, the half-cut X-ray image is reversed left to right and then input to the learning model 12M. In this way, not only the measurement of the vicinity of the left femur can be achieved, but also the measurement of the vicinity of the right femur can be achieved.

[0062] In this embodiment, GAN (Generative Adversarial Network) is used as the learning model 12M. Figure 2 to Figure 3B The learning model 12M shown is composed of pix2pix. GAN has a generator that generates output data based on input data, and a discriminator that identifies the authenticity of the data generated by the generator. The generator and the discriminator compete with each other to perform adversarial learning to build a network. The generator is a module that has an encoder that extracts latent variables from input data and a decoder that generates output data based on the extracted latent variables.

[0063] The learning model 12M is generated by the following processing: preparing training data that associates a training X-ray image (a half-cut image of a hip joint X-ray front image) with a training DRR image, and using the training data to learn the unlearned learning model. It is expected that the training X-ray image and DRR image are hip joint front X-ray images and DRR images of subjects who have been diagnosed as any of normal bone density cases, osteoporosis cases, and osteopenia cases by a DXA device. The information processing device 10 of this embodiment generates the learning model 12M, and the learning model 12M uses the X-ray images and DRR images prepared for training to learn and predict the DRR image based on the X-ray image.

[0064] During the learning process, the information processing device 10 performs Figure 3B Update of the parameters (weights, etc.) of the generator shown, and Figure 3A The parameters of the identifier shown in the figure are updated, and the learning is considered to be completed at the time point when the change of the error function converges. In the update of the parameters of the identifier, the information processing device 10 inputs the training X-ray image into the generator on the basis of fixing the parameters of the generator. The generator receives the input of the training X-ray image and generates a DRR image (information related to the body tissue volume of the object part) as output data. Then, the information processing device 10 uses the pair of X-ray images (training X-ray images) and DRR images (DRR images generated by the generator) equivalent to the input and output of the generator as false data, and provides the pair of X-ray images and DRR images contained in the training data as true data to the identifier, so that the identifier can identify the true and false. The information processing device 10 updates the parameters of the identifier in the following manner: when false data is input, a false value is output, and when true data is input, a true value is output. The updated parameters are the weights (coupling coefficients) between the nodes in the identifier, etc., and the parameter optimization method can use the error back propagation method, the steepest descent method, etc.

[0065] In the update of the parameters of the generator, the parameters of the recognizer are fixed, such as Figure 3B Learning is performed as shown. Here, the information processing device 10 updates the parameters of the generator in the following manner: when a training X-ray image is input into the generator and a DRR image generated by the generator is input into the recognizer, the authenticity is incorrectly judged (judged as true), and an image with features similar to the training X-ray image (similar image gradients, similar statistics of the output distribution of the intermediate layer of the recognizer, etc.) is generated. Here, the updated parameters are the weights (coupling coefficients) between the nodes in the generator, etc., and the parameter optimization method is the error back propagation method, the steepest descent method, etc. Thus, as Figure 2 As shown, a learning model 12M is generated that outputs a DRR image of the femur vicinity in the X-ray image when an X-ray image is input.

[0066] The information processing device 10 prepares such a learning model 12M in advance and uses it when generating (predicting) a DRR image from an X-ray image. In the case where the learning model 12M is used to actually predict a DRR image from an X-ray image, as shown in FIG. Figure 2 As shown, the information processing device 10 uses only the generator. The learning of the learning model 12M can also be performed by other learning devices. The learning model 12M generated by learning by other learning devices is downloaded from the learning device to the information processing device 10 via the network N or via the mobile storage medium 10a and stored in the storage unit 12. It should be noted that the learning model 12M that has been learned can also be downloaded from the learning device to the information processing device 10 by only the generator that generates the DRR image based on the X-ray image.

[0067] In addition to pix2pix, the learning model 12M can also be a GAN such as CycleGAN, StarGAN, etc. In addition, the learning model 12M is not limited to GAN, but can also be a model based on a neural network such as VAE (Variational Autoencoder), CNN (Convolutional Neural Network) (such as U-net), or other learning algorithms, and can also be composed by combining multiple learning algorithms.

[0068] Here, a description is given of a process of generating a training DRR image for learning the learning model 12M. Figure 4 is a flowchart showing an example of a training data generation process step. Figure 5 1 is an explanatory diagram of the generation process of the DRR image for training. The following process is executed by the control unit 11 of the information processing device 10 according to the program 12P stored in the storage unit 12, but can also be performed by other information processing devices or learning devices. In the following process, as the X-ray image and CT image generated for training data, the pair of the hip joint X-ray front image and CT image obtained by photographing the area including the pelvis and the left and right femurs of the subject are respectively corresponded and stored in the medical image DB 12a.

[0069] The control unit 11 of the information processing device 10 reads a pair of hip joint X-ray frontal images and CT images stored in the medical image DB12a (S11). First, the control unit 11 performs a brightness value calibration process on the read CT image (S12) to correct each brightness value (CT value) in the CT image. The brightness value (CT value) of each pixel measured by CT will produce deviations due to individual differences in the X-ray CT device, differences in the setting environment and shooting conditions, etc., and therefore, it is necessary to perform calibration to correct the deviation of the measured value. The calibration process of the CT image is performed, for example, using calibration data obtained when setting up the X-ray CT device, when replacing components such as the tube, when starting shooting, or regularly. The calibration data is generated by shooting a phantom made of a material with known properties by the X-ray CT device, and based on the obtained radiation density (CT value expressed by HU (Hounsfield units)) and the tissue density of the material. Specifically, a conversion formula for converting the radiation density obtained by the phantom into the tissue density of the material is used in the calibration data.

[0070] In addition, the calibration process can use the method described in the paper titled "Automated segmentation of an intensity calibration phantom in clinical CT images using a convolutional neural network" published in the International Journal of Computer Assisted Radiology and Surgery (IJCARS) (published online on March 17, 2021) and authored by the inventor, Keiichi Uemura, et al. The paper discloses a technique of photographing a phantom containing multiple substances of known tissue density (substances with different calcium content) together with a subject using an X-ray CT device, and automatically extracting the photographing area of ​​each substance in the phantom from the obtained CT image using CNN. By utilizing the disclosed technology of the paper, it is possible to generate calibration data for converting the CT value (radiation density) of the photographing area of ​​each substance extracted from the CT image into the tissue density of each substance. By using the calibration data generated in this way to calibrate the CT value of the photographing area of ​​the subject, the accurate tissue density of the subject can be obtained.

[0071] Next, the control unit 11 uses a segmented DNN (Deep Neural Network) to classify each pixel in the CT image into any one of a plurality of regions (musculoskeletal regions) such as bone regions, muscle regions, and other regions for the CT image that has been calibrated in step S12 (S13). The process of classifying each pixel of the CT image into musculoskeletal regions can be, for example, performed using the method described in the paper entitled “Automated Muscle Segmentation from Clinical CT Using Bayesian U-Net for Personalized Musculoskeletal Modeling” by the inventor, Yoshito Ohtake, et al., published in P.1030-1040 of IEEE Transactions on Medical Imaging, VOL.39, No.4, April 2020. The paper discloses the following musculoskeletal segmentation model: taking a CT image as input, classifying each pixel of the input CT image into any one of a bone region, a muscle region, and other regions, and outputting a classified CT image (musculoskeletal label image) in which the label of each region corresponds to each pixel. The musculoskeletal segmentation model disclosed in the paper is composed of a Bayesian U-net. Figure 5 As shown in (1), according to the CT image, each pixel in the CT image can be classified into any one of the three regions, and the musculoskeletal label image corresponding to the label of each region is obtained. Figure 5 In the figure, each pixel in the musculoskeletal label image is schematically represented by a color (lightness or darkness) corresponding to the classified area and muscle type.

[0072] like Figure 5 As shown in (2) in FIG. 1 , the control unit 11 extracts the data of the bone region from the CT image based on the musculoskeletal label image generated from the CT image (S14). Then, as shown in FIG. Figure 5As shown in (3) in the figure, the control unit 11 extracts data of the region of interest (here, the vicinity of the left femur) from the data (CT image) of the extracted bone region (S15). It should be noted that the process of extracting the region of interest from the bone region can be performed, for example, by pattern matching using a template. In this case, a template representing the shape of the vicinity of the left femur is pre-stored in the storage unit 12, and the control unit 11 determines whether there is a region consistent with the template based on the CT image of the bone region, extracts the region consistent with the template from the bone region, and thereby extracts data of the region of interest in the bone region. It should be noted that the process of extracting the region of interest from the bone region can also be performed, for example, using a learning model that has been machine-learned in a manner that outputs the region of interest in the bone region (the region near the left femur) when the CT image of the bone region is input. In this case, the control unit 11 can input the CT image of the bone region into the learning model, and determine and extract the region of interest in the bone region based on the output information from the learning model.

[0073] Next, the control unit 11 aligns the object to be photographed (here, the vicinity of the left femur) in the X-ray image (hip joint X-ray front image) acquired in step S11 and the region of interest in the CT image extracted in step S15 (S16). The control unit 11 detects the brightness gradient (edge) of the X-ray image based on the pixel value of each pixel, and determines the object to be photographed in the X-ray image based on the detected brightness gradient. It should be noted that the control unit 11 can determine the object to be photographed in the X-ray image by pattern matching using a pre-prepared template, or can determine the object to be photographed in the X-ray image by using a pre-learned learning model. Then, the control unit 11 determines a shooting direction consistent with the object to be photographed in the X-ray image for the region of interest in the CT image (the subject to be photographed, i.e., the vicinity of the left femur), and generates a CT image of the state of the region of interest observed from the determined direction. Thus, as Figure 5 As shown in (4), a CT image of the region of interest aligned with the imaging object in the X-ray image can be acquired.

[0074] It should be noted that the alignment of the object in the X-ray image and the object in the CT image can be achieved by, for example, using the method described in the paper entitled “Can Anatomic Measurements of Stem Anteversion Angle Be Considered as the Functional Anteversion Angle?” by the inventor, Kei Ryo Uemura, et al., published in P.595-600 of The Journal of Arthroplasty 33 (2018). The paper discloses a technique for segmenting the CT images of the pelvis and femur using a hierarchical statistical shape model to determine the pelvis and femur in the CT image, and for aligning (corresponding) the pelvis and femur in the CT image with the pelvis and femur in the X-ray image. In addition, the X-ray image and the CT image can be aligned using a method described in a paper entitled "3D-2D registration in mobile radiographs: algorithm development and preliminary clinical evaluation" by the inventor, Yoshito Ohtake, et al., published in Physics in Medicine and Biology 60 (2015) P.2075-2090. The paper discloses a technique for generating a CT image of the state of the object being imaged from the same direction as the X-ray image by parallel translation and rotation of the CT image.

[0075] Then, the control unit 11 generates a DRR image of the region of interest (near the left femur) by projecting each pixel of the CT image in the same direction as the shooting direction of the X-ray image based on the CT image of the region of interest (near the left femur) aligned with the object to be photographed in the X-ray image (near the left femur) (S17). The control unit 11 calculates the cumulative value of each pixel value (brightness value, three-dimensional pixel value) arranged in the CT image in the same direction as the shooting direction of the X-ray image, and uses the calculated cumulative value as each pixel value of the DRR image of the region of interest. Thus, the following is obtained: Figure 5 In the DRR image of the region of interest shown in (5), each pixel value in the DRR image becomes a value corresponding to the bone density at each position.

[0076] The control unit 11 extracts a half-cut image including the vicinity of the left femur from the hip joint X-ray front image acquired in step S11 (S18). Specifically, the control unit 11 extracts the right half region (region including the left femur) obtained by dividing the hip joint X-ray front image into two halves in the center of the left-right direction. The control unit 11 associates the extracted X-ray image (the half-cut image of the hip joint X-ray front image) with the DRR image of the region of interest generated in step S17, and stores them as training data in the training DB 12b (S19). The control unit 11 determines whether there are unprocessed images that have not been processed for the generation of the above-mentioned training data among the X-ray images and CT images stored in the medical image DB 12a (S20). When it is determined that there are unprocessed images (S20: YES), the control unit 11 returns to the processing of step S11, and performs the processing of steps S11 to S19 for the X-ray images and CT images that have not been processed for the generation of training data. When it is determined that there are no unprocessed images (S20: NO), the control unit 11 ends a series of processing. Through the above-mentioned processing, it is possible to generate training data for learning the learning model 12M based on the X-ray images and CT images stored in the medical image DB12a and accumulate them in the training DB12b. In the above-mentioned processing, the example in which the X-ray images and CT images used for generating the training data are stored in the medical image DB12a is described, but the control unit 11 may also be a configuration that obtains the X-ray images and CT images stored in other devices, for example, via the network N or via the mobile storage medium 10a. For example, the control unit 11 may also be a configuration that obtains the X-ray images and CT images from the electronic medical case data stored in the electronic medical case server. In addition, in the above-mentioned processing, the configuration is to align the region of interest of the bone region with the X-ray image after extracting it from the CT image, but it may also be a configuration that extracts the region of interest from the CT image after aligning it with the X-ray image.

[0077] Next, a description will be given of a process of generating a learning model 12M by learning the training data generated by the above-described process. Figure 6 12M is a flowchart showing an example of the steps for generating the learning model 12M. The following processing is performed by the control unit 11 of the information processing device 10 according to the program 12P stored in the storage unit 12, but can also be performed by other learning devices. Figure 4 The training data generation process shown is similar to Figure 6 The generation process of the learning model 12M shown may be performed by a separate device.

[0078] The control unit 11 of the information processing device 10 obtains a training data from the training DB 12b (S31). Specifically, the control unit 11 reads a pair of a half-cut image of the hip joint X-ray frontal image (specifically, an X-ray image of the area near the left femur) and a DRR image stored in the training DB 12b. The control unit 11 uses the read training data to perform a learning process of the learning model 12M (S32). Here, the control unit 11 updates the parameters of the generator and the identifier of the learning model 12M in the above-mentioned order, and generates a learning model 12M that generates and outputs a DRR image of the area near the femur in the X-ray image when the X-ray image included in the training data is input.

[0079] The control unit 11 determines whether there is unprocessed training data that has not been processed for learning among the training data stored in the training DB 12b (S33). When it is determined that there is unprocessed training data (S33: YES), the control unit 11 returns to the process of step S31 and performs the processes of steps S31 to S32 on the training data that has not been processed for learning. When it is determined that there is no unprocessed training data (S33: NO), the control unit 11 ends a series of processes. Through the above-mentioned learning process, a learning model 12M is generated that outputs a DRR image of the femur vicinity by inputting an X-ray image of an area containing the femur vicinity.

[0080] By repeating the learning process using the training data as described above, the learning model 12M can be further optimized. In addition, the learning model 12M that has already been learned can also be re-learned by performing the above-mentioned learning process. In this case, a learning model 12M with higher accuracy can be generated. It should be noted that the learning model 12M of the present embodiment is learned using the training data obtained as follows: using a segmentation technology that classifies musculoskeletal areas with high accuracy based on CT images, and a technology that aligns the object part in the X-ray image with the object part in the CT image with high accuracy, the rich three-dimensional data obtained by CT is spatially associated with the X-ray image. Therefore, it is possible to realize a learning model 12M that can generate high-precision DRR images without requiring a large number of cases (training data).

[0081] Next, a description will be given of a process of estimating the bone density of the femur vicinity from a hip joint X-ray frontal image including the femur vicinity of the subject using the learning model 12M generated as described above. Figure 7 is a flowchart showing an example of a bone density estimation process procedure. Figure 8The following processing is executed by the control unit 11 of the information processing device 10 according to the program 12P stored in the storage unit 12. It should be noted that, hereinafter, the DRR image generated from the X-ray image using the learning model 12M is referred to as a predicted DRR image.

[0082] The control unit 11 of the information processing device 10 obtains a frontal X-ray image of the hip joint obtained by photographing an area including the pelvis and left and right femurs of a subject such as a patient using an X-ray device (S41). The control unit 11 obtains the frontal X-ray image of the hip joint of a patient for whom bone density estimation is desired from electronic medical record data stored in an electronic medical record server, for example. In addition, when the frontal X-ray image of the hip joint of the subject is stored in a mobile storage medium 10a, the control unit 11 may also use the reading unit 16 to read the frontal X-ray image of the hip joint from the mobile storage medium 10a.

[0083] The control unit 11 extracts a half-cut image of the femur vicinity on the side where the bone density is measured from the acquired hip joint X-ray front image (S42). Specifically, the control unit 11 extracts the right half area (the area including the left femur) obtained by dividing the hip joint X-ray front image into two halves in the center of the left-right direction. It should be noted that when the control unit 11 measures the bone density of the right femur vicinity of the subject, after extracting the left half area (the area including the right femur) of the hip joint X-ray front image, it pre-performs a process of reversing it left and right. The control unit 11 generates a predicted DRR image of the femur vicinity in the X-ray image based on the half-cut image of the X-ray image extracted in step S42 (S43). Specifically, the control unit 11 inputs the X-ray image containing the femur vicinity (the half-cut image of the hip joint X-ray front image) into the learning model 12M, and obtains the predicted DRR image of the femur vicinity in the X-ray image as output information from the learning model 12M.

[0084] The control unit 11 calculates the bone density of the femur vicinity based on the generated predicted DRR image of the femur vicinity (S44). Each pixel value of the predicted DRR image becomes a value corresponding to the bone density, and the control unit 11 calculates the bone density of the femur vicinity by, for example, calculating the average value of each pixel value in the predicted DRR image. In addition to the bone density (BMD: Bone Mineral Density), the control unit 11 also calculates the result of the young adult comparison (YAM: Young Adult Mean) calculated based on the bone density, and the result of the same age comparison. The control unit 11 stores the calculated inspection results in, for example, the electronic medical record data of the electronic medical record server (S45).

[0085] The control unit 11 generates a screen showing the inspection result, and outputs it to, for example, the display unit 15 (S46), displays it on the display unit 15, and ends the process. Figure 8 The inspection result screen is shown. Figure 8 The screen shown shows the subject's identification information (such as patient ID, patient name, etc.) and the frontal X-ray image of the hip joint and its shooting time and date. Figure 8 The screen shown shows the predicted DRR image, the name of the object part in the predicted DRR image (in Figure 8 In the example, the control unit 11 reads the bone density estimated based on the predicted DRR image, the young adult comparison, and the same age comparison as the bone density test result based on the hip joint X-ray front image. In addition, for example, when the annotations that should be prompted to the doctor and the like are stored in the storage unit 12 in correspondence with the values ​​of the bone density, the young adult comparison, or the same age comparison, the control unit 11 reads the annotations corresponding to the calculated test results (bone density, young adult comparison, or same age comparison) from the storage unit 12, such as Figure 8 As shown, it can also be displayed on the inspection result screen.

[0086] Through the above-mentioned processing, the bone density of the femur vicinity in the hip joint X-ray frontal image taken by an X-ray device commonly used in medical institutions, etc. can be estimated. In addition, in this embodiment, a predicted DRR image of the femur vicinity generated from the hip joint X-ray frontal image and the bone density estimated from the predicted DRR image can be presented to doctors, etc. Thus, the doctor can judge the state of the femur vicinity of the patient based on the predicted DRR image and the estimated bone density.

[0087] In this embodiment, the learning model 12M automatically extracts the characteristics of the shooting state of the femur vicinity in the X-ray image and predicts the generation of the DRR image. Therefore, it is not necessary to perform an examination using a DXA device or the like, and the bone density can be estimated by only shooting using an X-ray device. Therefore, the bone density of the target part can be estimated based on the X-ray image that can also be taken in a health diagnosis or a small-scale clinic, so the measurement and inspection of the bone density can be easily achieved. As a result, early diagnosis and early treatment intervention of osteopenia or osteoporosis can be performed, and it can be expected to prevent fractures associated with osteopenia or osteoporosis, which can help extend the healthy life span.

[0088] In the present embodiment, as described above, it is possible to achieve spatial alignment of the object part (part near the femur) in the X-ray image and the object part in the CT image with high precision. Thus, by using training data based on X-ray images and CT images (DRR images) that have been aligned with high precision, it is possible to generate a high-precision predicted DRR image without learning a large amount of training data. For example, using a learning model 12M obtained by learning from 200 pairs of training data generated from X-ray images and CT images collected from patients with osteoarthritis of the hip, a predicted DRR image is generated based on the X-ray image, and the comparison result of the bone density near the femur estimated from the predicted DRR image and the bone density near the femur measured using a DXA device is as shown in FIG. Fig. 9A and Fig. 9B shown. Fig. 9A and Fig. 9B This is a graph showing the relationship between the bone density estimated from the predicted DRR image and the bone density measured by the DXA device. Fig. 9A The diagrams and Fig. 9B The graphs shown show verification results based on X-ray images and bone density (BMD) collected by X-ray devices and DXA devices in different medical institutions, respectively.

[0089] Fig. 9A In the graph, the horizontal axis represents the bone density of the femur vicinity estimated from the predicted DRR image, and the vertical axis represents the bone density of the femur vicinity measured by the DXA device. Fig. 9B The horizontal axis of the graph shows the bone density of the femur vicinity measured by the DXA device, and the vertical axis shows the bone density of the femur vicinity estimated from the predicted DRR image. Fig. 9A and Fig. 9B As can be seen from the graph, the bone density of the femur near the estimated part based on the predicted DRR image and the bone density of the femur near the measured part by the DXA device show a high linear correlation. Fig. 9A In the medical institution shown, a correlation coefficient of 0.861 was obtained, and the average error (mean absolute error) between the bone density estimated from the predicted DRR image and the bone density measured by the DXA device was 0.06 g / cm2. Fig. 9B In the medical institution shown, a correlation coefficient of 0.869 was obtained, and the average error between the bone density estimated from the predicted DRR image and the bone density measured by the DXA device was 0.07 g / cm2. In this way, even the learning model 12M after learning with a small amount of training data can predict the bone density to the same degree as the measurement result using the DXA device. As a result, the operational burden in the collection and learning process of training data can be reduced.

[0090] In this embodiment, the structure of using the learning model 12M to generate a predicted DRR image of the vicinity of the femur based on an X-ray image containing the vicinity of the femur, and estimating the bone density of the vicinity of the femur based on the predicted DRR image is described. However, the part of the object of bone density estimation can be the lumbar spine, thoracic spine, cervical spine, clavicle, ribs, hand bones, foot bones, or specific parts thereof, in addition to the vicinity of the femur. For other parts, the same process can be used to generate training data and a learning model, and bone density estimation using the learning model can be performed.

[0091] In the present embodiment, the generation process of training data, the learning process of the learning model 12M using the training data, and the estimation process of bone density using the learning model 12M are not limited to the configuration performed locally by the information processing device 10. For example, an information processing device that performs each of the above-mentioned processes may be separately provided. In addition, a server that performs the generation process of training data and the learning process of the learning model 12M may be provided. In this case, the information processing device 10 is configured as follows: X-ray images and CT images used for training data are sent to the server, the server generates training data based on the X-ray images and CT images, and the learning model 12M is generated by the learning process using the generated training data and sent to the information processing device 10. Thus, the information processing device 10 can use the learning model 12M obtained from the server to realize the estimation process of the bone density of the target part. In addition, a server that performs the estimation process of the bone density using the learning model 12M may be provided. In this case, the information processing device 10 is configured as follows: the X-ray image of the subject is sent to the server, the server performs a process of generating a predicted DRR image and a process of estimating the bone density using the learning model 12M, and sends the generated predicted DRR image and the estimated bone density to the information processing device 10. Even in the case of such a configuration, the same process as the above-mentioned present embodiment can be performed and the same effect can be obtained.

[0092] (Implementation Method 2)

[0093] In the above-mentioned first embodiment, a configuration for generating a predicted DRR image of a target part (e.g., the vicinity of the femur) based on an X-ray image of the target part and estimating the bone density of the target part based on the predicted DRR image is described. In the present embodiment, an information processing device for estimating the bone density of a target part for which the bone density is to be estimated based on an X-ray image of a part different from the target part for which the bone density is to be estimated is described. The information processing device of the present embodiment has the same configuration as the information processing device 10 of the first embodiment, and therefore, the description of the configuration is omitted. It should be noted that the information processing device 10 of the present embodiment has the same configuration as the information processing device 10 of the first embodiment except that Figure 1In addition to the configuration shown in the figure, the storage unit 12 stores a learning model 12M1 for estimating bone density (see Fig.10 ). In addition, in the medical image DB12a of the present embodiment, X-ray images and CT images of a part different from the object part for which the bone density is to be estimated are stored correspondingly to the bone density obtained by measuring the object part for which the bone density is to be estimated by a DXA device. The present embodiment is configured to estimate the bone density of the vicinity of the femur based on the chest X-ray image. Therefore, the medical image DB12a stores the X-ray image and CT image of the chest of the subject, and the bone density obtained by measuring the vicinity of the femur of the subject by a DXA device. It should be noted that the bone density stored in the medical image DB12a can be any bone density of a part used for the diagnosis of bone loss and osteoporosis, and can be the bone density of the lumbar spine, pelvis or femur, or can be the average value or median of the bone density of the whole body.

[0094] Fig.10 1 is an explanatory diagram showing an overview of the learning models 12M and 12M1 of the second embodiment. When estimating the bone density of the femur vicinity, the information processing device 10 of the present embodiment generates a predicted DRR image of the part using the learning model 12M based on the X-ray images of parts different from the femur vicinity, such as the ribs, clavicle, thoracic vertebrae, etc. taken by chest X-ray photography. Then, the information processing device 10 estimates the bone density of the femur vicinity using the learning model 12M1 (second learning model) for bone density estimation based on the generated predicted DRR image. It should be noted that the X-ray image shooting part used for estimating the bone density of the femur vicinity is not limited to the ribs, clavicle, thoracic vertebrae, etc., and each bone in the X-ray image of the femur vicinity that is not shot can be used.

[0095] The learning model 12M1 for bone density estimation is a machine learning model that has learned predetermined training data, and is learned in a manner that takes a DRR image generated based on a CT image of the subject's chest as input and outputs the bone density of the femur vicinity of the subject. It is assumed that the learning model 12M1 for bone density estimation is used as a program module constituting artificial intelligence software. The learning model 12M1 for bone density estimation is composed of, for example, CNN, but it can also be composed of other algorithms such as logistic regression and linear regression, and can also be composed of a combination of multiple algorithms. It is believed that the bone density of human bones is similarly reduced in each bone of the body, so it is believed that the bone density of the bone of the subject of the CT image has a strong correlation with the bone density of bones other than the subject. Therefore, in this embodiment, the learning model 12M1 for bone density estimation can be used to estimate the bone density of a part other than the subject that is expected based on the CT image (DRR image).

[0096] The learning model 12M1 for estimating bone density is generated as follows: training data that associates a DRR image for training with a training (correct) bone density is prepared, and the unlearned learning model 12M1 is subjected to machine learning using the training data. The DRR image for training can be, for example, a DRR image generated from a CT image of the chest of a subject diagnosed as a case of normal bone density, a case of osteoporosis, or a case of osteopenia by a DXA device, and the bone density for training can be the bone density of the femur vicinity of the subject measured by the DXA device. The information processing device 10 of this embodiment generates the learning model 12M1 for estimating bone density as follows: learning is performed using the DRR image and bone density prepared for training, and the bone density of a part other than the subject of the DRR image is output based on the DRR image.

[0097] The learning model 12M1 for bone density estimation is learned in a manner that outputs the correct bone density when a training DRR image is input. In the learning process, the learning model 12M1 performs calculations based on the input DRR image and outputs an output value as a result of the calculation. Then, the learning model 12M1 compares the calculated output value with the correct bone density to optimize the parameters used for the calculation process in a manner that makes the two approximate. The parameter is the weighting (coupling coefficient) between the nodes in the learning model 12M1 for bone density estimation, and the parameter optimization method can utilize the error back propagation method, the steepest descent method, etc. Thus, a learning model 12M1 is generated that outputs the bone density of the femur vicinity when a DRR image generated based on a chest CT image is input. The learning of the learning model 12M1 for bone density estimation can also be performed by other learning devices.

[0098] Fig.11 This is a flowchart showing an example of a training data generation process procedure according to the second embodiment. Fig.11 The treatment shown is in Figure 4 In the process shown, step S51 is added to replace step S11, step S18 is deleted, and step S52 is added between steps S19 and S20. Figure 4 The same steps are omitted in description. It should be noted that, in the present embodiment, in the medical image DB 12a, for learning the learning models 12M and 12M1, chest X-ray images and chest CT images obtained by photographing the chest of each subject, and the bone density near the femur of each subject measured by the DXA device are stored accordingly.

[0099] The control unit 11 of the information processing device 10 of this embodiment reads a set of chest X-ray images, chest CT images, and bone density near the femur from the medical image DB 12a (S51). Then, the control unit 11 executes the Figure 4 The control unit 11 performs the same processing as steps S12 to S17 and S19 in step S51. That is, the control unit 11 performs calibration processing on the chest CT image (S12), classifies each pixel of the chest CT image into a bone region, a muscle region, or any of other regions (S13), and extracts the data of the bone region from the chest CT image (S14). Then, the control unit 11 extracts the data of the region of interest (e.g., ribs) from the CT image of the extracted bone region (S15), and performs alignment of the imaging object (here, ribs) in the region of interest in the chest X-ray image acquired in step S51 and the CT image extracted in step S15 (S16). Moreover, the control unit 11 generates a DRR image of the region of interest based on the CT image of the region of interest (ribs) after the alignment (S17), corresponds the X-ray image acquired in step S51 to the DRR image of the region of interest generated in step S17, and stores them in the training DB 12b as training data for learning the learning model 12M (S19). It should be noted that, in this embodiment, the chest X-ray image read from the medical image DB 12a is directly used as training data, but it can also be used as Figure 4 As in step S18 in , a process of extracting a region for training data from the read X-ray image is performed.

[0100] In addition, the control unit 11 makes the DRR image of the area of ​​interest generated in step S17 correspond to the bone density near the femur acquired in step S51, and stores it in the storage unit 12 (S52) as training data for learning the learning model 12M1 for bone density estimation. It should be noted that the control unit 11 can prepare a DB for accumulating training data for learning the learning model 12M1 for bone density estimation in the storage unit 12 in advance, and store the generated training data in the DB. Then, the control unit 11 transfers to the processing of step S20. Through the above-mentioned processing, training data for learning the learning model 12M can be generated based on the X-ray images, CT images, and bone density stored in the medical image DB12a and accumulated in the training DB12b, and training data for learning the learning model 12M1 for bone density estimation can be generated and accumulated in the DB.

[0101] In this embodiment, the generation of the learning model 12M1 for bone density estimation can be performed by Figure 6 It should be noted that in the generation process of the learning model 12M1 for bone density estimation, Figure 6In step S31 of the method, the control unit 11 reads out a pair of a DRR image of a region of interest (here, the ribs) and a bone density in the vicinity of the femur stored in the storage unit 12. In addition, in step S32, the control unit 11 inputs the DRR image included in the training data into the learning model 12M1 for bone density estimation, and obtains output information from the learning model 12M1 for bone density estimation. The control unit 11 compares the obtained output information with the bone density in the vicinity of the femur included in the training data, and optimizes parameters such as weightings between nodes in the learning model 12M1 for bone density estimation in a manner that makes the two approximate, for example, by using an error back propagation method. Through the above-mentioned processing, a learning model 12M1 for bone density estimation is generated by inputting a DRR image generated based on a CT image of the ribs, thereby outputting the bone density in the vicinity of the femur.

[0102] Next, a description will be given of a process of estimating the bone density of the femur vicinity of the subject from the chest X-ray image of the subject using the learning models 12M and 12M1. Fig.12 is a flowchart showing an example of a bone density estimation process procedure according to Embodiment 2. Fig.13 is an explanatory diagram showing an example of a screen. The control unit 11 of the information processing device 10 acquires a chest X-ray image obtained by photographing the chest of the subject by an X-ray device (S61). Here, the control unit 11 also acquires a chest X-ray image of the subject whose bone density is to be estimated from electronic medical record data, for example.

[0103] The control unit 11 inputs the acquired chest X-ray image into the learning model 12M, and generates a predicted DRR image of the target part (here, the ribs) in the X-ray image (S62). The control unit 11 obtains the predicted DRR image of the ribs as output information of the learning model 12M. It should be noted that in this embodiment, the acquired chest X-ray image is directly input into the learning model 12M, but it can also be Figure 7 The processing of extracting the region of the target part for estimating the bone density of the femur vicinity from the acquired X-ray image is performed as in step S42 in the above, and the X-ray image of the extracted region is input into the structure of the learning model 12M. The target part for estimating the bone density of the femur vicinity can be set to the clavicle, thoracic vertebra, etc. in addition to the ribs.

[0104] The control unit 11 inputs the predicted DRR image of the ribs generated in step S62 into the learning model 12M1 for bone density estimation, and estimates the bone density of the femur vicinity of the subject (S63). It should be noted that the control unit 11 obtains the bone density of the femur vicinity as output information of the learning model 12M1 for bone density estimation. The control unit 11 calculates a numerical value representing the results of the young adult comparison (YAM) and the same age comparison based on the estimated bone density (BMD) (S64). Then, the control unit 11 stores the test results in, for example, electronic medical record data (S65), generates a screen displaying the test results, and outputs it to the display unit 15 (S66). Here, the control unit 11 generates, for example Fig.13 The inspection result screen is shown. Fig.13 The screen shown displays the identification information of the subject and the chest X-ray image and its shooting time and date, and displays the predicted DRR image of the ribs, the name of the target part in the predicted DRR image (in Fig.13 The control unit 11 may read out the bone density of the femur estimated from the predicted DRR image, the young adult comparison, and the same age comparison as the bone density test result based on the chest X-ray image. Here, if the storage unit 12 stores annotations corresponding to the test results (bone density, young adult comparison, or same age comparison), the control unit 11 may also read out the annotations corresponding to the test results from the storage unit 12 and display them on the test result screen.

[0105] The same effects as those of the first embodiment can also be obtained in this embodiment. In addition, in this embodiment, the bone density of the femur vicinity of the subject can be estimated based on the chest X-ray image taken using the X-ray device. Therefore, in this embodiment, when a chest X-ray image is taken in a health diagnosis, for example, the bone density of the femur vicinity of the subject can be estimated based on the chest X-ray image, and the bone density measurement can be more easily performed during other examinations. Therefore, for subjects whose bone density measurement is not the purpose of the examination, the possibility of bone loss or osteoporosis can be diagnosed, and early diagnosis and early treatment intervention can be performed.

[0106] In addition, in this embodiment, a predicted DRR image of a predetermined part (e.g., ribs) generated from a chest X-ray image and the bone density of the femur vicinity estimated from the predicted DRR image can be presented to a doctor or the like. Thus, the doctor can judge the state of the patient's ribs and femur vicinity based on the predicted DRR image of the predetermined part and the estimated bone density of the femur vicinity, and diagnose osteopenia or osteoporosis. It should be noted that the object of bone density estimation is not limited to the femur vicinity, and can be set to any part, and the object of the X-ray image used for bone density estimation can be a part other than the part of the object of bone density estimation.

[0107] In the present embodiment, the learning model 12M and the learning model 12M1 for bone density estimation can be composed of one learning model. Specifically, for example, the learning model can also be constructed in a manner that takes a chest X-ray image as input and outputs the bone density of the femur vicinity of the subject of the chest X-ray image. In addition, in the present embodiment, the following learning model can also be used to replace the learning model 12M1 for bone density estimation: the learning model uses the learning model 12M to learn in a manner that takes a predicted DRR image of a predetermined part (such as ribs) generated according to the chest X-ray image as input, generates a predicted DRR image of the femur vicinity according to the predicted DRR image, and outputs it. In this case, the DRR image of the femur vicinity of the subject can be estimated based on the predicted DRR image of the ribs, and the bone density of the femur vicinity can be estimated based on the estimated DRR image of the femur vicinity. Moreover, for example, the following learning model can also be used: the learning model is learned in a manner that, for example, takes a chest X-ray image as input, generates a predicted DRR image of the femur vicinity of the subject of the chest X-ray image, and outputs it. In this case, the DRR image of the femur of the subject can also be estimated from the chest X-ray image, and thus the bone density of the femur can be estimated based on the DRR image of the femur. In this embodiment, the modified example described in the first embodiment can also be applied.

[0108] (Implementation method 3)

[0109] In the above-mentioned embodiment 1, a configuration is described for generating a predicted DRR image of an object part (bone area such as the femur vicinity) based on an X-ray image of the object part, and estimating the bone density of the object part based on the predicted DRR image. In the present embodiment, an information processing device is described for generating a predicted DRR image of a muscle area of ​​an object part (for example, a muscle area such as the lower abdomen and buttocks) based on an X-ray image of the object part, and estimating the muscle mass of the object part based on the generated predicted DRR image. The information processing device of the present embodiment has the same configuration as the information processing device 10 of embodiment 1, and therefore, the description of the configuration is omitted. It should be noted that in the medical image DB 12a of the present embodiment, the X-ray image and the CT image of the part (muscle area) where the muscle mass is to be estimated are stored correspondingly.

[0110] Fig.14 is an explanatory diagram showing an overview of a learning model 12Ma according to Embodiment 3. The learning model 12Ma of this embodiment has Figure 2The learning model 12M of the embodiment 1 shown in the figure has the same structure and is generated by the same learning process. It should be noted that the learning model 12Ma of the present embodiment uses, for example, a frontal X-ray image of the hip joint (X-ray image of the buttocks) and a DRR image of the muscle area of ​​the buttocks generated based on a CT image obtained by photographing the buttocks (lower abdomen) with an X-ray CT device as training data, and is learned in a manner that a DRR image of the muscle area of ​​the buttocks (predicted DRR image, information related to muscle mass) is output when an X-ray image of the buttocks is input. Fig.14 In the example shown, the learning model 12Ma generates DRR images of multiple muscle regions such as the gluteus maximus, gluteus medius, and hamstrings maximus, but can also be configured to generate a DRR image of any muscle region. For example, if there is a muscle region that can be used for the diagnosis of sarcopenia, the learning model 12Ma can be generated in a manner that generates a DRR image of the muscle region.

[0111] Next, a description will be given of a process for generating training data for learning the learning model 12Ma according to the present embodiment. Fig.15 This is a flowchart showing an example of a training data generation process procedure according to the third embodiment. Fig.15 The treatment shown is in Figure 4 In the process shown, step S71 is added to replace step S14, steps S15 to S16 and S18 are deleted, and steps S72 to S73 are added to replace steps S17 and S19. Figure 4 The same steps are omitted.

[0112] The control unit 11 of the information processing device 10 of this embodiment performs Figure 4 The same processing as steps S11 to S13 in the above is performed. That is, the control unit 11 reads out a pair of X-ray images (hip joint X-ray front images) and CT images from the medical image DB 12a, performs brightness calibration on the read CT images, and then classifies each pixel in the CT image into bone areas, muscle areas, and other areas (musculoskeletal areas). Thus, Figure 5 As shown in (1), a musculoskeletal label image can be obtained after each pixel in the CT image is classified into each region. In this embodiment, the control unit 11 extracts muscle region data from the CT image based on the musculoskeletal label image (S71). The control unit 11 extracts muscle region data (CT image) for each muscle type.

[0113] The control unit 11 generates a DRR image of the muscle region obtained by projecting each pixel of the CT image in a predetermined direction based on the data (CT image) of the extracted muscle region (S72). For example, in the case of receiving an X-ray image obtained by receiving radiation passing from the back to the front of the subject, the control unit 11 generates a DRR image obtained by projecting each pixel of the CT image of the muscle region in the direction from the back to the front of the subject. Here, the control unit 11 also uses the cumulative value of each pixel value (brightness value, three-dimensional pixel value) arranged in the predetermined direction in the CT image as each pixel value of the DRR image. Thus, in this embodiment, each pixel value in the DRR image of the muscle region becomes a value corresponding to the muscle density (muscle volume) at each position. It should be noted that the control unit 11 can perform alignment of the photographed objects (here, each muscle) in each image in the X-ray image acquired in step S11 and the CT image of the muscle region extracted in step S71, and generate a DRR image obtained by projecting each pixel of the aligned CT image in the same direction as the shooting direction of the X-ray image.

[0114] The control unit 11 matches the X-ray image acquired in step S11 with the DRR image of the muscle area generated in step S72, and stores them as training data in the training DB 12b (S73). Then, the control unit 11 transfers to the processing of step S20. Through the above processing, it is possible to generate training data for learning the learning model 12Ma of this embodiment based on the X-ray images and CT images stored in the medical image DB 12a and accumulate them in the training DB 12b. By using the training data generated in this way, in this embodiment, it is possible to Figure 6 The same learning process as shown generates the learning model 12Ma.

[0115] Next, a description will be given of a process of estimating the muscle mass of the subject's buttocks from the subject's hip joint X-ray frontal image using the learning model 12Ma. Fig.16 is a flowchart showing an example of muscle mass estimation processing steps. Fig.17 It is an explanatory diagram showing a screen example. Fig.16 The treatment shown is in Figure 7 In the process shown, step S42 is deleted, and steps S81 to S82 are added instead of steps S43 to S44. Figure 7 The same steps are omitted.

[0116] The control unit 11 of the information processing device 10 obtains a hip joint X-ray front image of the subject (S41). The control unit 11 generates a predicted DRR image of the muscle area in the X-ray image based on the hip joint X-ray front image (S81). Specifically, the control unit 11 inputs the hip joint X-ray front image into the learning model 12Ma, and obtains the predicted DRR image of the muscle area of ​​the buttocks in the X-ray image as the output information from the learning model 12Ma. Then, the control unit 11 calculates the muscle mass of each muscle in the buttocks based on the generated predicted DRR image of the muscle area (S82). In this embodiment, each pixel value of the predicted DRR image becomes a value corresponding to the muscle density (muscle density), and the control unit 11 calculates the muscle density in the muscle area by calculating the average value of each pixel value in the predicted DRR image. In addition, the control unit 11 calculates the muscle mass in the muscle area based on the calculated muscle density and the volume of the muscle area. It should be noted that the control unit 11 can calculate the muscle mass in the muscle area by calculating the muscle mass of each pixel based on each pixel in the predicted DRR image, and accumulating the muscle mass corresponding to each pixel. Moreover, the control unit 11 can predict the muscle mass of the entire body of the subject based on the muscle mass in each muscle area. For example, the muscle mass of each muscle of the subject, such as the gluteus maximus, gluteus medius, and hamstrings, can be matched and registered with the muscle mass of the whole body of the subject in advance, thereby predicting the muscle mass of the whole body of the subject by the muscle mass of each muscle estimated based on the predicted DRR image. In addition, the control unit 11 can predict the muscle mass of the upper limbs or the muscle mass of the lower limbs of the subject based on the muscle mass in each muscle area.

[0117] The control unit 11 stores the examination results including the calculated muscle density and muscle mass of each muscle in, for example, electronic medical record data (S45), and generates a Fig.17 The inspection result screen shown is output to the display unit 15 (S46). Fig.17 The screen shown shows the subject's identification information and the hip joint X-ray frontal image and its shooting time and date. Fig.17 The screen shown shows the predicted DRR image of each muscle, the name of the target part in the predicted DRR image (in Fig.17 In addition, when the annotations to be prompted to the doctor and the like are stored in the storage unit 12 in correspondence with the muscle density or muscle mass, the control unit 11 can read the annotations corresponding to the calculated inspection results (muscle density or muscle mass) from the storage unit 12, and Fig.17 The results are displayed as shown in the inspection result screen.

[0118] Through the above-mentioned processing, in this embodiment, the muscle mass of the muscle area in the hip joint X-ray frontal image can be estimated based on the hip joint X-ray frontal image taken using an X-ray device. In addition, in this embodiment, the predicted DRR image of the muscle area generated based on the hip joint X-ray frontal image and the muscle mass estimated based on the predicted DRR image can be prompted to the doctor or the like. Thus, the doctor can judge the state of the patient's muscle based on the predicted DRR image and the estimated muscle mass. In this embodiment, the learning model 12Ma automatically extracts the characteristics of the shooting state of the muscle area in the X-ray image and generates the predicted DRR image of the muscle area. Therefore, there is no need to perform an inspection using a DXA device, etc., and only shooting using an X-ray device can estimate the muscle mass. Therefore, the muscle mass of the target part can be estimated based on the X-ray image that can also be taken in a health diagnosis or a small-scale clinic, so the measurement and inspection of the muscle mass can be easily implemented. Thus, early diagnosis of sarcopenia, in which muscle mass and muscle strength decrease due to aging, can be performed, and it can be expected to prevent and inhibit the onset and progression of sarcopenia, which helps to extend the healthy life span.

[0119] In this embodiment, the following learning model 12Ma can be realized: it is possible to generate a predicted DRR image of a muscle area in an X-ray image with high accuracy based on an X-ray image by learning based on a small amount of training data. Therefore, in this embodiment, it is possible to predict muscle mass and muscle density based on an X-ray image to the same extent as the measurement result using a DXA device, and it is possible to reduce the operational burden in the collection and learning of training data.

[0120] In this embodiment, the use of the learning model 12Ma to generate a predicted DRR image of the muscle area of ​​the buttocks (lower abdomen) based on an X-ray image of the buttocks (lower abdomen), and to estimate the muscle density and muscle mass of the buttocks based on the predicted DRR image, is described. In addition to the buttocks, the parts of the object of muscle density and muscle mass estimation may also be the upper limbs, lower limbs, chest, abdomen, whole body, etc. For other parts, the same process can be used to generate training data and a learning model, and the muscle density and muscle mass can be estimated using the learning model.

[0121] In this embodiment, the same effects as those of the above-mentioned embodiments 1 to 2 can also be obtained. In addition, in this embodiment, the muscle mass of the target part can be estimated based on the X-ray image of the target part taken using an X-ray device. As a result, it is easier to perform muscle mass measurement inspections, so that for subjects who are not concerned about their muscle mass, the possibility of diseases such as sarcopenia can be diagnosed, and early diagnosis and early treatment intervention of sarcopenia can be performed. In addition, in this embodiment, the modified examples appropriately described in the above-mentioned embodiments 1 to 2 can also be applied.

[0122] (Implementation 4)

[0123] An embodiment obtained by combining the configuration of the first embodiment with the configuration of the second embodiment is described. That is, in this embodiment, an information processing device is described: the information processing device generates a predicted DRR image of the target part based on an X-ray image obtained by photographing the target part, estimates the bone density of the target part based on the generated predicted DRR image, and estimates the bone density of a part different from the target part. The information processing device of this embodiment has the same configuration as the information processing device 10 of the first and second embodiments, so the description of the configuration is omitted. That is, the information processing device 10 of this embodiment has a learning model 12M that generates a predicted DRR image of the target part based on the X-ray image of the target part, and a bone density estimation learning model 12M1 that estimates the bone density of a part different from the target part based on the DRR image of the target part. In the following example, the learning model 12M is set as a model that generates a predicted DRR image of the femur vicinity based on an X-ray image of the femur vicinity, and the bone density estimation learning model 12M1 is set as a model that estimates the bone density of the ribs based on the DRR image of the femur vicinity. However, the learning models 12M and 12M1 are not limited to such a configuration, and a plurality of models may be prepared for the learning models 12M and 12M1, respectively. For example, as the learning model 12M, a model for generating (predicting) a DRR image of the pelvis based on an X-ray image of the pelvis, a model for generating a DRR image of the ribs, clavicle, or thoracic vertebrae based on an X-ray image of the chest, etc. may be further prepared. In addition, as the learning model 12M1 for bone density estimation, a model for estimating the bone density of the ribs, clavicle, thoracic vertebrae, lumbar vertebrae, cervical vertebrae, hand bones, or foot bones based on a DRR image of the vicinity of the femur, pelvis, or femur, a model for estimating the bone density of the vicinity of the femur, lumbar vertebrae, cervical vertebrae, hand bones, or foot bones based on a DRR image of the ribs, clavicle, or thoracic vertebrae, etc. may be prepared.

[0124] The information processing device 10 of this embodiment can execute Fig.11 and Figure 6 The processing shown, through Fig.11 The processing shown generates training data for learning the learning models 12M and 12M1. Figure 6 The illustrated process executes the learning process of the learning models 12M and 12M1.

[0125] Next, a description will be given of a process of estimating the bone density of the femur vicinity and the bone density of the ribs of the subject from the front X-ray image of the subject's hip joint using the learning models 12M and 12M1. Fig.18 is a flowchart showing an example of a bone density estimation process procedure according to the fourth embodiment. Fig.19 It is an explanatory diagram showing a screen example. Fig.18 The treatment shown is in Figure 7 In the process shown, steps S91 to S96 are added after step S46. Figure 7 The same steps are omitted. The control unit 11 of the information processing device 10 executes Figure 7 The information processing device 10 thus estimates the bone density of the femur vicinity of the subject based on the front X-ray image of the subject's hip joint, and displays the bone density of the femur vicinity of the subject as shown in FIG. Figure 8 The inspection result screen is shown.

[0126] The control unit 11 determines whether the bone density of a part other than the object of the X-ray image should be estimated (S91). Fig.19 As shown in FIG. 1 , an input field for selecting a part other than the object of the X-ray image is provided in the inspection result screen. When other parts are input through the input field, the control unit 11 determines that the bone density of the input part should be estimated. Fig.19 In the example shown, a pull-down menu for selecting an arbitrary part is provided in the input field, and when the doctor or the like wants to confirm the bone density of other parts, the desired part is selected through the pull-down menu. When the control unit 11 determines that the bone density of other parts is not to be estimated (S91: NO), that is, when no other parts are input in the input field, the processing ends.

[0127] When the control unit 11 determines that the bone density of other parts is to be estimated (S91: YES), the control unit 11 selects a learning model 12M1 for estimating bone density for the target part based on the part input into the input field (S92). Here, a predicted DRR image of the femur vicinity is generated in step S43, and the ribs are selected as other parts of the object for bone density estimation, so the control unit 11 selects a learning model 12M1 for estimating bone density for estimating the bone density of the ribs based on the DRR image of the femur vicinity. Then, the control unit 11 inputs the predicted DRR image of the femur vicinity generated in step S43 into the selected learning model 12M1 for estimating bone density, and estimates the bone density of the ribs (other parts) of the subject based on the output information from the learning model 12M1 for estimating bone density (S93). Then, the control unit 11 executes the same as Fig.12 The same processing (S94-S95) as steps S64-S65 in the above embodiment is performed, and the obtained inspection results are output to the inspection result screen being displayed (S96). Fig.19 As shown, in addition to the bone density of the femur vicinity estimated from the hip joint X-ray frontal image, the bone density of the subject's ribs can also be displayed.

[0128] In this embodiment, the same effects as those of the above-mentioned embodiments can also be obtained. In addition, in this embodiment, based on the X-ray image obtained by photographing the object part, not only the bone density of the object part can be estimated, but also the bone density of other parts can be estimated and prompted. Thus, in this embodiment, based on the X-ray image taken during health diagnosis or medical treatment, not only the bone density of the part of the object of the X-ray image can be estimated, but also the bone density of other parts can be estimated. Therefore, in the case of taking an X-ray image of a part other than the part where the bone density is to be confirmed, the bone density of the part where the bone density is to be confirmed can also be predicted, thereby bringing about early diagnosis and early treatment intervention of bone loss or osteoporosis. In addition, in this embodiment, the modified examples appropriately described in the above-mentioned embodiments can also be applied.

[0129] (Implementation method 5)

[0130] An embodiment obtained by combining the configuration of the first embodiment with the configuration of the third embodiment is described. That is, in this embodiment, an information processing device that estimates the bone density in the bone region of the target part based on the X-ray image obtained by photographing the target part and estimates the muscle mass in the muscle region of the target part is described. The information processing device of this embodiment has the same configuration as the information processing device 10 of the first and third embodiments, so the description of the configuration is omitted. That is, the information processing device 10 of this embodiment has a learning model 12M that generates a predicted DRR image of the bone region of the target part based on the X-ray image of the target part, and a learning model 12Ma that generates a predicted DRR image of the muscle region of the target part based on the X-ray image of the target part. In the following example, the learning model 12M is set as a model that generates a predicted DRR image of the femur vicinity based on the hip joint X-ray front image, and the learning model 12Ma is set as a model that generates a predicted DRR image of the muscle region of the buttocks based on the hip joint X-ray front image. However, the learning models 12M and 12Ma are not limited to such a configuration, and multiple models are prepared for the learning models 12M and 12Ma, respectively. For example, as a learning model 12M, a model for generating (predicting) DRR images of the ribs, clavicle, or thoracic vertebrae based on an X-ray image of the chest can be prepared; and as a learning model 12Ma, a model for generating DRR images of the pectoralis major, pectoralis minor, subclavian, serratus anterior, or intercostal muscles based on an X-ray image of the chest can be prepared.

[0131] The information processing device 10 of this embodiment can execute Figure 4 , Figure 6 and Fig.15 The processing shown, through Figure 4 The processing shown generates training data for learning model 12M, by Fig.15 The processing shown generates training data for learning the learning model 12Ma, by Figure 6 The illustrated process executes the learning process of the learning models 12M and 12Ma.

[0132] Next, a description will be given of a process of estimating the bone density of the femur vicinity and the muscle mass of the buttocks of the subject based on the front X-ray image of the subject's hip joint using the learning models 12M and 12Ma. Fig. 20 is a flowchart showing an example of a bone density and muscle mass estimation process step according to Embodiment 5. Fig.21 It is an explanatory diagram showing a screen example. Fig. 20 The treatment shown is in Figure 7 In the process shown, step S101 is added between steps S41 and S42, and steps S102 to S106 are added after step S46. Figure 7 The same steps are omitted.

[0133] After acquiring the front X-ray image of the hip joint of the subject (S41), the control unit 11 of the information processing device 10 determines whether the bone density of the femur vicinity should be estimated based on the acquired X-ray image (S101). For example, a doctor or the like specifies through a menu screen (not shown) whether to estimate the bone density of the bone region or the muscle mass of the muscle region for the X-ray image subject. When the estimation of the bone density of the bone region is specified through the menu screen, the control unit 11 determines to estimate the bone density of the femur vicinity (S101: YES), and executes the same as above. Figure 7 The information processing device 10 thus estimates the bone density of the femur vicinity of the subject based on the front X-ray image of the subject's hip joint, and displays the bone density of the femur vicinity of the subject as shown in FIG. Figure 8 The inspection result screen is shown.

[0134] When the control unit 11 determines that the bone density of the femur vicinity is not to be estimated (S101: NO), the control unit 11 proceeds to the processing of step S102 to determine whether the muscle mass of the buttocks muscle region, which is the subject of the acquired X-ray image, should be estimated (S102). When the estimation of the muscle mass of the muscle region is not specified via the menu screen, the control unit 11 determines that the muscle mass of the buttocks is not to be estimated (S102: NO), and ends the processing. On the other hand, when the estimation of the muscle mass of the muscle region is specified via the menu screen, the control unit 11 determines that the muscle mass of the buttocks is to be estimated (S102: YES), and executes the same as Fig.16The information processing device 10 estimates the muscle mass of the subject's hip muscle area based on the subject's hip joint X-ray front image, and outputs the obtained test result to the test result screen being displayed (S106). Fig.21 As shown, in addition to the bone density of the femur vicinity estimated from the frontal X-ray image of the hip joint, the muscle mass of the buttocks of the subject can also be displayed. Through the above-mentioned processing, the bone density of the bone region and the muscle mass of the muscle region of the target part can be estimated based on the X-ray image of the target part, and it is possible to appropriately switch whether to use either bone density or muscle mass as the estimation object or both of them according to the designation of the user such as a doctor.

[0135] In this embodiment, the same effects as those of the above-mentioned embodiments can also be obtained. In addition, in this embodiment, based on the X-ray image obtained by photographing the object part, not only the bone density of the bone area of ​​the object part can be estimated, but also the muscle mass of the muscle area of ​​the object part can be estimated and prompted. Thus, the state of the bones and muscles in the photographed object can be confirmed, and therefore, early diagnosis of bone loss or osteoporosis, and early diagnosis of diseases such as sarcopenia can be performed. In this embodiment, the modified examples appropriately described in the above-mentioned embodiments can also be applied.

[0136] (Implementation 6)

[0137] In the above-mentioned embodiments 1 to 5, the information processing device aligns the bone region (target part or region of interest) in the X-ray image and the bone region (target part or region of interest) in the CT image in the X-ray image (simple X-ray image) and the CT image obtained by photographing the same object of the same subject. The information processing device of this embodiment has the same configuration as the information processing device 10 of embodiments 1 to 5, and therefore the description of the configuration is omitted.

[0138] Fig. 22 is a flowchart showing an example of the alignment processing steps. Figures 23A to 24B This is a diagram for explaining the alignment process. Fig. 22 The alignment shown is Figure 4 and Fig.11 Thus, in this embodiment, the control unit 11 of the information processing device 10 Figure 4 and Fig.11 After the processing of step S15 is performed Fig. 22 Then, execute Figure 4 and Fig.11The following describes the alignment of the pelvis in the hip joint X-ray front image and the pelvis in the CT image, taking the pelvis as the region of interest. However, the bone region used for alignment is not limited to the region of interest. Any bone region captured in the X-ray image and the CT image can be used for alignment. For example, in the hip joint X-ray front image, in addition to the pelvis, the femur, the vicinity of the femur, etc. can also be used for alignment.

[0139] exist Figure 4 and Fig.11 After processing step S15, the control unit 11 determines the region of interest (here, the pelvis) in the X-ray image (here, the front X-ray image of the hip joint) acquired in step S11 (S111). The process of determining the region of interest in the X-ray image can be performed, for example, by pattern matching using a template representing the shape of the region of interest, and can also be performed, for example, using a learning model that has been machine-learned in a manner that outputs the region of interest in the X-ray image when an X-ray image is input. Thus, for example, the region of interest in the X-ray image can be determined. Fig.23A The X-ray image shown shows the region of the pelvis indicated by a solid line.

[0140] Next, the control unit 11 generates a pseudo DRR image of the region of interest based on the region of interest in the CT image extracted in step S15 (S112). Here, the control unit 11 generates a pseudo DRR image of the pelvis. Specifically, Fig. 23B As shown, the control unit 11 configures the region of interest in the CT image (three-dimensional CT image of the pelvis) in the three-dimensional virtual space of the X-ray imaging system under predetermined projection conditions (position and angle relative to the virtual X-ray source), and generates a pseudo DRR image (projection image) obtained by projecting the region of interest from the virtual X-ray source onto the two-dimensional X-ray imaging plane. Here, the shooting conditions (such as the posture of the subject, the bending angle of the joints, etc.) when shooting the X-ray image and the shooting conditions when shooting the CT image are different, so the outline of the region of interest in the pseudo DRR image generated from the CT image is inconsistent with the outline of the region of interest in the X-ray image. Fig.24A In the figure, the solid line represents the contour P1 of the region of interest in the X-ray image, and the dotted line represents the contour P2 of the region of interest in the pseudo DRR image. In this embodiment, the control unit 11 updates the projection conditions of the region of interest in the CT image, thereby determining the projection conditions under which the correlation value between the contour P1 of the region of interest in the X-ray image and the contour P2 of the region of interest in the pseudo DRR image is maximized. With such projection conditions, Fig. 24B As shown, a pseudo DRR image can be obtained in which the region of interest in the pseudo DRR image is accurately aligned with the region of interest in the X-ray image.

[0141] Thus, the control unit 11 calculates the correlation value between the contour of the region of interest in the X-ray image determined in step S111 and the contour of the region of interest in the pseudo DRR image generated in step S112 (S113), and determines whether the calculated correlation value is the maximum (S114). When the control unit 11 determines that the correlation value is not the maximum (S114: NO), the control unit 11 updates the projection conditions when the pseudo DRR image is generated from the CT image of the region of interest (S115), and repeats the processing of steps S112 to S114 under the updated projection conditions. The control unit 11 repeats the processing of steps S112 to S115 until it is determined that the calculated correlation value is the maximum. When it is determined that the correlation value is the maximum (S114: YES), that is, when a pseudo DRR image with the maximum correlation value can be generated, the projection conditions at this time are determined (S116).

[0142] In the present embodiment, the control unit 11 can perform the processing of steps S112 to S115 using, for example, the method described in the paper entitled "3D-2D registration in mobile radiographs: algorithm development and preliminary clinical evaluation" written by the inventor, Yoshito Ohtake, et al. The paper discloses that the realization of alignment of the contour in the DRR image (here, pseudo DRR image) of the object part (bone area, here, pelvis) generated from the three-dimensional area of ​​the object part in the CT image and the contour of the object part in the real X-ray image is defined as "maximization of the correlation between the depth gradient intensity image of the X-ray image and the DRR image respectively", and a method of finding a configuration (projection conditions, specifically the three-dimensional position and angle of the object part relative to the imaging system) that maximizes the above correlation is used using a covariance matrix adaptation evolution strategy (CMA-ES). Thus, the control unit 11 determines the pseudo DRR image in which the correlation value between the contour of the region of interest in the X-ray image and the contour of the pseudo DRR image is the largest by using the covariance matrix adaptive evolution strategy, thereby being able to determine the projection conditions in the determined pseudo DRR image. Under such projection conditions, Fig. 24B As shown, a pseudo DRR image of the region of interest can be generated after the contour P2 of the region of interest in the pseudo DRR image is aligned with the contour P1 of the region of interest in the X-ray image with high precision. It should be noted that in step S114, the control unit 11 can be configured to determine whether the calculated correlation value is greater than a predetermined value, and when it is determined to be greater than the predetermined value, proceed to the processing of step S116.

[0143] Then, the control unit 11 proceeds to Figure 4 and Fig.11 The processing of step S17 generates a DRR image (S17) obtained by projecting the CT image of the region of interest in the projection direction according to the projection conditions determined in step S116. Specifically, the control unit 11 calculates the cumulative value of the pixel values ​​(brightness values, three-dimensional pixel values) of each pixel arranged in the projection direction according to the projection conditions determined in step S116 in the CT image of the region of interest, and uses the calculated cumulative value as each pixel value of the DRR image of the region of interest. In this way, a DRR image of the object part under the same projection conditions (position and angle relative to the X-ray source) as the shooting conditions of the object part (region of interest) in the X-ray image is generated, and the DRR image of the object part that is accurately aligned with the object part in the X-ray image is obtained. Then, the control unit 11 executes Figure 4 By performing the processing of steps S18 to S19, the half-cut image of the X-ray image (hip joint X-ray front image) can be matched with the DRR image of the region of interest projected in the same direction as the shooting direction of the X-ray image, thereby generating training data for learning the learning model 12M. Fig.11 S19 and S52 can generate training data for learning the learning model 12M by corresponding the X-ray image with the DRR image of the area of ​​interest projected in the same direction as the shooting direction of the X-ray image, and can generate training data for learning the learning model 12M1 for bone density estimation by corresponding the DRR image of the area of ​​interest with the bone density of the area of ​​interest acquired in step S51.

[0144] Through the above-mentioned processing, training data is generated by associating an X-ray image of the object part (bone area, i.e., area of ​​interest) with a DRR image of the object part after being aligned with respect to the object part in the X-ray image with high precision, and the training data is accumulated in the training DB 12b. It should be noted that the bone area is hard tissue, and therefore it is believed that it will not be deformed when the X-ray image is taken and when the CT image is taken. Therefore, by performing an alignment process by optimizing the conditions (projection conditions) when generating the DRR image based on the CT image as in the present embodiment, the contour of the object part (bone area) in the X-ray image can be accurately aligned with the contour of the object part in the DRR image generated based on the CT image. In addition, the information processing device 10 of the present embodiment can execute Figure 6 The process shown is performed by using the training data generated as described above. Figure 6 The processing shown in the figure performs the learning processing of the learning model 12M, and can realize the learning model 12M that can predict the DRR image with high accuracy from the X-ray image. Figure 7 , Fig.12 , Fig.18 and Fig. 20 The processing shown uses the learning model 12M generated as described above, thereby making it possible to estimate the bone density of the target site with high accuracy using the DRR image predicted with high accuracy from the X-ray image, thereby obtaining a high-accuracy inspection result.

[0145] The above processing is Figure 4 In the processing shown, the X-ray image and the region of interest in the CT image are aligned based on the region of interest extracted from the CT image in step S15, but the process is not limited to this. For example, the X-ray image and the bone region in the CT image may be aligned based on the bone region extracted from the CT image in step S14. In this case, the control unit 11 of the information processing device 10 Figure 4 After the processing of step S14 in Fig. 22 The control unit 11 performs the alignment processing shown in the figure, and executes the processing of step S15 and step S17 based on the CT image after the alignment processing. Specifically, the control unit 11 extracts the data of the region of interest (e.g., the vicinity of the left femur) from the CT image after the alignment processing (S15), and generates a DRR image (S17) projected in the projection direction according to the projection conditions determined in step S116 based on the CT image of the extracted region of interest. It should be noted that the Fig. 22 In the process shown, the bone region extracted in step S14 is used as the region of interest, and the processes of steps S111 to S116 are performed. Then, the control unit 11 performs the processes of steps S18 to S19. In such a process, training data is also generated by associating the X-ray image of the object part (region of interest) with the DRR image of the object part after the object part in the X-ray image is aligned with high precision, and accumulated in the training DB 12b. By using the training data generated in this way to perform the learning process of the learning model 12M, it is possible to realize a learning model 12M that can predict a high-precision DRR image based on the X-ray image. In addition, by using the learning model 12M generated in this way, the DRR image can be predicted with high precision based on the X-ray image, and the bone density of the object part can be estimated with high precision using the DRR image predicted with high precision.

[0146] In the present embodiment, the processing other than the above-mentioned alignment processing is the same as that of the above-mentioned embodiments, and the same effect as that of the above-mentioned embodiments can be obtained. In addition, in the present embodiment, the spatial alignment of the object part in the X-ray image and the object part in the CT image can be performed with high precision. Thus, by using such an X-ray image and a DRR image generated based on the CT image after being aligned with the X-ray image with high precision as training data, a learning model 12M that predicts the DRR image with high precision based on the X-ray image can be realized without a large number of cases (training data). In addition, in the present embodiment, the modified examples appropriately described in the above-mentioned embodiments can also be applied.

[0147] (Implementation 7)

[0148] The following processing is described: In the above-mentioned embodiments 1 and 3, the configuration of the above-mentioned embodiment 6 is applied, and the information processing device performs alignment of the imaging object (object part or region of interest) in the X-ray image and the CT image based on the bone region in the X-ray image and the CT image, and generates training data for learning the learning models 12M and 12Ma based on the aligned CT image. The information processing device of this embodiment has the same configuration as the information processing device 10 of embodiments 1 and 3, and therefore, the description of the configuration is omitted. That is, the information processing device 10 of this embodiment has the learning models 12M and 12Ma.

[0149] Fig.25 This is a flowchart showing an example of a training data generation process procedure according to the seventh embodiment. Fig.25 The treatment shown is in Figure 4 In the process shown, steps S121 to S126 are added to replace steps S15 and S16, and steps S127 to S129 are added between steps S19 and S20. Figure 4 The same steps are omitted.

[0150] The control unit 11 of the information processing device 10 of this embodiment executes Figure 4 It should be noted that in step S14, the control unit 11 extracts data of any bone region from the CT image based on the musculoskeletal label image generated from the CT image. Figure 5 In the CT image shown in FIG. 1 , the control unit 11 can extract bone regions such as the pelvis, the thigh bone, and the vicinity of the thigh bone, or multiple or all of these bone regions. After extracting the bone regions in the CT image, the control unit 11 performs the same operation as described above based on the bone regions extracted from the CT image. Fig. 22Specifically, the control unit 11 specifies the same bone region in the X-ray image as the bone region extracted from the CT image (S121).

[0151] Next, the control unit 11 generates a pseudo DRR image of the bone region based on the bone region in the CT image extracted in step S14 (S122). Then, the control unit 11 calculates the correlation value between the contour of the bone region in the X-ray image determined in step S121 and the contour of the bone region in the pseudo DRR image generated in step S122 (S123), and determines whether the calculated correlation value is the maximum (S124). When the control unit 11 determines that the correlation value is not the maximum (S124: NO), the control unit 11 updates the projection conditions when the pseudo DRR image is generated from the CT image of the bone region (S125), and repeats the processing of steps S122 to S124 under the updated projection conditions. The control unit 11 repeats the processing of steps S122 to S125 until it determines that the calculated correlation value is the maximum. When it is determined that the correlation value is the maximum (S124: YES), that is, when a pseudo DRR image with the maximum correlation value can be generated, the projection conditions at this time are determined (S126). In this embodiment, the control unit 11 also uses the covariance matrix adaptive evolution strategy for the processing of steps S122 to S125 to determine the projection conditions in the pseudo DRR image that maximize the correlation value between the contour of the bone region in the X-ray image and the contour of the pseudo DRR image.

[0152] Then, the control unit 11 generates a DRR image (S17) projected in the projection direction according to the projection conditions determined in step S126 based on the CT image of the region of interest (here, the bone region such as the femur vicinity). That is, the control unit 11 generates a DRR image of the region of interest in a state observed from the same direction as the shooting direction of the region of interest in the X-ray image. Then, the control unit 11 extracts a half-cut image containing the vicinity of the left femur from the hip joint X-ray front image acquired in step S11 (S18), and corresponds the extracted X-ray image (the half-cut image of the hip joint X-ray front image) to the DRR image of the region of interest generated in step S17, and stores it in the training DB 12b as training data (S19). Thus, the training data obtained by associating the X-ray image of the object part (the region of interest, i.e., the bone region) with the DRR image of the object part generated based on the CT image after being aligned with the object part in the X-ray image with high precision is accumulated.

[0153] Next, the control unit 11 executes Fig.15The same processing as steps S71 to S73 in step S127 to S129 is performed. Specifically, the control unit 11 deletes the data of the bone area from the CT image based on the musculoskeletal label image, and extracts the data of the muscle area (S127). Here, the control unit 11 also sets each type of muscle as a region of interest, and extracts the data (CT image) of each region of interest (muscle area). In addition, based on the data (CT image) of the extracted region of interest (muscle area), the control unit 11 generates a DRR image (S128) projected in the projection direction according to the projection conditions determined in step S126. Then, the control unit 11 corresponds the X-ray image acquired in step S11 to the DRR image of the region of interest (muscle area) generated in step S128, and stores them in the training DB12b as training data (S129). Then, the control unit 11 proceeds to the processing of step S20.

[0154] Through the above-mentioned processing, the object (object part or region of interest) in the X-ray image and the CT image can be aligned based on the bone region, and based on the aligned X-ray image and the CT image, training data that enables the learning model 12M, 12Ma of this embodiment to learn with good efficiency is generated. By using the training data generated in this way, the training model 12M, 12Ma of this embodiment can be trained efficiently. Figure 6 According to the learning process shown, the information processing device 10 of this embodiment can generate a learning model 12M that predicts the DRR image of the bone region, which is the object of the X-ray image, from the X-ray image with high accuracy, and a learning model 12Ma that predicts the DRR image of the muscle region, which is the object of the X-ray image, from the X-ray image with high accuracy. In addition, the information processing device 10 of this embodiment can generate a DRR image of the bone region predicted from the X-ray image with high accuracy, and a DRR image of the muscle region predicted from the X-ray image with high accuracy by using the learning models 12M and 12Ma, and can estimate the bone density and muscle mass of the target part with high accuracy.

[0155] Fig.26A and Fig.26B It is an explanatory diagram showing the effect of the positioning process based on the bone region. Fig.26A It is a graph showing the relationship between the muscle mass of the muscle region (specifically, the fat-free muscle mass) obtained by generating a predicted DRR image of the muscle region based on an X-ray image and calculating (estimating) the predicted DRR image generated based on the learned model 12Ma, and the muscle mass of the muscle region measured based on a CT image (specifically, the fat-free muscle mass), wherein the learned model 12Ma is obtained by learning the training data corresponding to the X-ray image and the DRR image generated based on the CT image without performing bone region-based alignment processing on the X-ray image and the CT image shown in this embodiment. Fig.26AThe vertical axis of the graph represents the fat-free muscle mass estimated from the predicted DRR image, and the horizontal axis represents the fat-free muscle mass measured from the CT image. Fig.26A A diagram of the gluteus medius is shown on the left, and a diagram of the iliacus is shown on the right. Fig.26B It is a graph showing the relationship between the muscle mass of the muscle region (specifically, the fat-free muscle mass) obtained by generating a predicted DRR image of the muscle region based on the X-ray image using the learning model 12Ma, and calculating (estimating) the muscle mass of the muscle region (specifically, the fat-free muscle mass) based on the generated predicted DRR image, and the muscle mass of the muscle region measured based on the CT image (specifically, the fat-free muscle mass). The learning model 12Ma is obtained by learning the training data corresponding to the X-ray image and the DRR image generated based on the CT image after the X-ray image and the CT image shown in this embodiment are aligned based on the bone region. Fig.26B The graph of FIG. 1 also shows the fat-free muscle mass estimated from the predicted DRR image on the vertical axis and the fat-free muscle mass measured from the CT image on the horizontal axis. Fig.26B A diagram of the gluteus medius is shown on the left, and a diagram of the iliacus is shown on the right. Fig.26A and Fig.26B The graph shown shows the comparison results (verification results) between the fat-free muscle mass estimated from the predicted DRR images predicted from X-ray images collected from 525 patients using the learning model 12Ma and the fat-free muscle mass measured based on the CT images of each patient. The learning model 12Ma is learned, for example, using 390 pairs of training data generated from X-ray images and CT images collected from patients with osteoarthritis of the hip.

[0156] according to Fig.26A Left and Fig.26B As can be seen from the graph on the left, for the fat-free muscle mass of the gluteus medius, the correlation coefficient (PCC: Pearson Correlation Coefficient) between the estimated value based on the predicted DRR image and the measured value based on the CT image when the alignment of the present embodiment is not performed is 0.340, and the correlation coefficient is 0.776 when the alignment of the present embodiment is performed, and a higher correlation value is obtained when the alignment of the present embodiment is performed. Fig.26A Right side and Fig.26BAs can be seen from the graph on the right, for the fat-free muscle mass of the iliacus muscle, the correlation coefficient is 0.250 when the alignment of the present embodiment is not performed, and the correlation coefficient is 0.804 when the alignment of the present embodiment is performed, and a higher correlation value is obtained when the alignment of the present embodiment is performed. Thus, the learning model 12Ma is learned using training data generated by X-ray images and CT images after the alignment of the photographed object by the alignment processing of the present embodiment, so that even the learning model 12Ma that has been learned by a small amount of training data can realize a learning model 12Ma that can estimate the muscle mass to the same extent as the measured value based on the CT image and can predict the DRR image. Thus, the operational burden in the collection and learning processing of the training data can be reduced.

[0157] Fig.26A and Fig.26B The graph shows that when the fat-free muscle mass of the gluteus medius and iliacus is aligned according to the present embodiment, a more accurate fat-free muscle mass can be obtained, and in the verification process of the inventor, the same result was obtained for the muscle volume of the gluteus medius and iliacus. It should be noted that the fat-free muscle mass is calculated based on the CT value (pixel value) in each muscle area (individual muscle area) in the CT image. The muscle mass of each pixel is calculated based on the calculated ratio and the muscle mass is calculated by the sum of the muscle mass of each pixel. The muscle volume represents the volume of each muscle area (individual muscle area) in the CT image.

[0158] In the present embodiment, by performing alignment processing on the X-ray image and the CT image based on the bone region, the X-ray image and the CT image after the alignment processing become a state in which the muscle region is also accurately aligned. By using the training data of the muscle region in the X-ray image after the alignment and the DRR image of the muscle region generated according to the CT image to learn the learning model 12Ma, it is possible to realize a learning model 12Ma that can estimate the muscle amount (for example, fat-free muscle amount, muscle volume) of each muscle region measured according to the CT image to the same extent as the muscle amount (for example, fat-free muscle amount, muscle volume) of each muscle region. In addition, in the present embodiment, by deleting the data of the bone region from the CT image after the alignment, extracting the CT image (data of the muscle region) after excluding the influence of the bone region, and by using such a CT image, it is possible to generate training data for learning the learning model 12Ma with high precision. Therefore, in the present embodiment, not only can the bone density be estimated with high precision by using the predicted DRR image of the bone region generated by the learning model 12M according to the X-ray image, but also the muscle amount can be estimated with high precision by using the predicted DRR image of the muscle region generated by the learning model 12Ma according to the X-ray image.

[0159] In the present embodiment, when generating training data for learning the learning models 12M and 12Ma, in addition to performing the alignment processing of the photographed object in the X-ray image and the CT image based on the bone region, the same effect as the above-mentioned embodiments can be obtained. In addition, in the present embodiment, based on the X-ray image and the CT image after the alignment based on the bone region, not only the training data for learning the learning model 12M but also the training data for learning the learning model 12Ma is generated, so that the learning model 12Ma that predicts the DRR image of the muscle region in the X-ray image with high accuracy can be realized. In the present embodiment, the modified examples appropriately described in the above-mentioned embodiments can also be applied.

[0160] (Implementation 8)

[0161] A modification of the learning model 12Ma for generating a predicted DRR image of a muscle region (e.g., gluteus maximus, gluteus medius, iliacus, etc.) included in an object part (e.g., lower abdomen, buttocks, etc.) based on an X-ray image of the object part in the above-mentioned embodiments 3 and 7 is described. The information processing device of this embodiment has the same structure as the information processing device 10 of embodiments 1, 3, and 7, and therefore, the description of the structure is omitted.

[0162] Fig. 27 This is an explanatory diagram showing an example of the structure of the learning model 12Mb of Embodiment 8. The learning model 12Mb is a model that is learned by taking an X-ray image of a target part as input, performing calculations to predict a DRR image and a muscle volume of a muscle area included in the target part based on the input X-ray image, and outputting the calculation results. Fig. 27 The learning model 12Mb shown is a structure that takes the front X-ray image of the hip joint as input and predicts the DRR image and muscle mass of the gluteus maximus. It should be noted that the learning model 12Mb is not limited to this structure. For example, it can also take the front X-ray image of the hip joint as input and predict the DRR image and muscle mass of other muscle areas such as the gluteus medius and iliacus. It can also predict the DRR image and muscle mass of each muscle area for multiple muscle areas. In addition, the learning model 12Mb can be a structure that takes the front X-ray image of the chest as input and predicts the DRR image and muscle mass of muscle areas such as the pectoralis major, pectoralis minor, subclavius, serratus anterior, intercostal muscles, trapezius, latissimus dorsi, teres major, erector spinae, etc. The muscle mass can be represented by information such as fat-free muscle mass, muscle volume, muscle density, etc., which is related to the muscle.

[0163] The learning model 12Mb of this embodiment has an image conversion layer 12Mb1 and a muscle mass prediction layer 12Mb2. The input data of the learning model 12Mb, i.e., the X-ray image, is input to the image conversion layer 12Mb1, and the image conversion layer 12Mb1 generates and outputs a DRR image of the muscle area (e.g., gluteus maximus) contained in the X-ray image based on the input X-ray image. The DRR image of the muscle area generated by the image conversion layer 12Mb1 is input to the muscle mass prediction layer 12Mb2, and the muscle mass prediction layer 12Mb2 predicts the muscle mass (e.g., fat-free muscle mass, muscle volume, muscle density, etc.) of the muscle area based on the input DRR image of the muscle area and outputs it.

[0164] The image conversion layer 12Mb1 may be composed of a model based on a GAN such as pix2pix, CycleGAN, StarGAN, a neural network such as VAE, CNN (such as U-net), or other learning algorithms, or may be composed of a combination of multiple learning algorithms. It should be noted that when the image conversion layer 12Mb1 is composed of pix2pix, the image conversion layer 12Mb1 may have the same Fig.14 The muscle mass prediction layer 12Mb2 may be constructed by using algorithms such as CNN, random forest, SVM (Support Vector Machine), Transformer, etc., or may be constructed by combining multiple algorithms. It should be noted that the learning model 12Mb may not have the muscle mass prediction layer 12Mb2, or may be constructed as follows to replace the muscle mass prediction layer 12Mb2: the control unit 11 of the information processing device 10 calculates the muscle mass of the muscle area based on the sum of the pixel values ​​(brightness values) of each pixel in the DRR image of the muscle area generated by the image conversion layer 12Mb1.

[0165] The learning model 12Mb of the present embodiment is generated by learning using training data obtained by associating a training X-ray image (hip joint X-ray front image) with a DRR image of the correct gluteus maximus and the muscle mass of the gluteus maximus. It is expected that the training X-ray image, the correct DRR image and the muscle mass are, for example, the hip joint X-ray front image, the DRR image, and the muscle mass of the subject diagnosed as any one of a normal bone density case, an osteoporosis case, and an osteopenia case by a DXA device. It should be noted that the correct DRR image of the gluteus maximus can use the DRR image of the gluteus maximus (muscle area) generated from the CT image obtained by photographing the buttocks (lower abdomen) by an X-ray CT device. In addition, the correct muscle mass of the gluteus maximus can also be used by the muscle mass of the gluteus maximus calculated by the sum of the muscle mass of each pixel, the sum of the muscle mass of each pixel is calculated based on the CT value (brightness value) in the muscle area (gluteus maximus) in the CT image based on the ratio of muscle to fat in each pixel, and the muscle mass of the gluteus maximus measured by the DXA device can also be used.

[0166] The learning model 12Mb of the present embodiment learns by outputting the correct DRR image and muscle mass contained in the training data when the X-ray image contained in the training data is input. In the learning process, the learning model 12Mb performs calculations based on the input X-ray image, predicts the DRR image and muscle mass of the gluteus maximus in the input X-ray image, and outputs the prediction result. Then, the learning model 12Mb compares the predicted DRR image with the correct DRR image, and compares the predicted muscle mass with the correct muscle mass, so that they are respectively approximate, and optimizes the parameters such as the weighting (coupling coefficient) between the nodes in the image conversion layer 12Mb1 and the muscle mass prediction layer 12Mb2 using the steepest descent method, the error back propagation method, etc. That is, in the present embodiment, in the learning process of the learning model 12Mb, the difference (error, loss related to muscle mass) between the muscle mass of the DRR image of the muscle area generated by the learning model 12Mb and the correct muscle mass can be fed back. Therefore, according to the learning process of this embodiment, a learning model 12Mb is generated that predicts the DRR image and muscle volume of the muscle area included in the X-ray image with higher accuracy when the X-ray image is input.

[0167] It should be noted that the learning model 12Mb of the present embodiment may be a structure without the muscle mass prediction layer 12Mb2. In this case, in the learning process of the learning model 12Mb, the control unit 11 of the information processing device 10 inputs the X-ray image into the learning model 12Mb, obtains the DRR image of the muscle area from the learning model 12Mb, and calculates the muscle mass of the muscle area based on the obtained DRR image. Then, the control unit 11 calculates the difference between the calculated muscle mass and the correct muscle mass, and makes the learning model 12Mb (image conversion layer 12Mb1) learn in a way that makes the calculated difference smaller (so that the calculated muscle mass is close to the correct muscle mass). In such a learning process, it is also possible to feedback the difference (error, loss related to muscle mass) between the muscle mass of the DRR image of the muscle area generated by the learning model 12Mb and the correct muscle mass, and generate a learning model 12Mb that can predict the DRR image and muscle mass of the muscle area contained in the input X-ray image with high precision.

[0168] The information processing device 10 of this embodiment can execute Fig.25 The processing shown, through Fig.25 The processing shown generates 12Mb of training data for learning the learning model, by Figure 6 The processing shown executes the learning processing of the learning model 12Mb. Fig.25 In the process shown, the control unit 11 of the information processing device 10 calculates the muscle mass of the muscle area based on the DRR image of the muscle area generated in step S128 after processing in step S128. Then, in step S129, the control unit 11 corresponds the X-ray image acquired in step S11, the DRR image of the region of interest (muscle area) generated in step S128, and the muscle mass calculated based on the DRR image, and stores them as training data in the training DB 12b. It should be noted that the muscle mass of the muscle area can use the muscle mass measured by a DXA device. In this case, the control unit 11 can acquire, for example, the muscle mass measured by a DXA device together with the X-ray image and the CT image in step S11, and use the acquired muscle mass for training data. In addition, in Figure 6 In the process shown, the control unit 11 reads a set of X-ray images, DRR images, and muscle mass stored in the storage unit 12 in step S31, and performs a learning process of the learning model 12Mb using the read training data in step S32. Fig.16 The processing shown, by using the learning model 12Mb generated as described above, can predict the DRR image of the muscle region in the X-ray image with high accuracy, and estimate the muscle mass of the muscle region with high accuracy using the predicted DRR image.

[0169] Fig.28A and Fig.28B 2 is an explanatory diagram showing the effect of feedback processing related to loss of muscle mass. Fig.28A It is a graph showing the relationship between the muscle mass (specifically, the fat-free muscle mass) of the muscle region obtained by generating a predicted DRR image of the muscle region based on an X-ray image using a learning model (for example, learning model 12Ma) and calculating (estimating) the predicted DRR image based on the generated predicted DRR image, and the muscle mass (specifically, the fat-free muscle mass) of the muscle region measured based on a CT image, wherein the learning model is learned without performing feedback processing related to muscle mass loss as described above. Fig.28A The vertical axis of the graph represents the fat-free muscle mass estimated from the predicted DRR image, and the horizontal axis represents the fat-free muscle mass measured from the CT image. Fig.28A A diagram of the gluteus medius is shown on the left, and a diagram of the iliacus is shown on the right. Fig.28B It is a graph showing the relationship between the muscle mass (specifically, the fat-free muscle mass) of the muscle region obtained by generating a predicted DRR image of the muscle region based on an X-ray image and calculating (estimating) the predicted DRR image generated, and the muscle mass (specifically, the fat-free muscle mass) of the muscle region measured based on a CT image, wherein the learning model 12Mb is learned by performing feedback processing on the loss related to the muscle mass as described above. Fig.28B The vertical axis of the graph represents the fat-free muscle mass estimated from the predicted DRR image, and the horizontal axis represents the fat-free muscle mass measured from the CT image. Fig.28B A diagram of the gluteus medius is shown on the left, and a diagram of the iliacus is shown on the right. Fig.28A and Fig.28B The graph shown shows the comparison results (verification results) of the fat-free muscle mass estimated by the predicted DRR images predicted based on X-ray images collected from 525 patients and the fat-free muscle mass measured based on the CT images of each patient using the learning model 12Mb, wherein the learning model 12Mb is learned, for example, using 390 pairs of training data generated from X-ray images and CT images collected from patients with osteoarthritis of the hip.

[0170] according to Fig.28A Left and Fig.28B As can be seen from the graph on the left, for the fat-free muscle mass of the gluteus medius, the correlation coefficient between the estimated value based on the predicted DRR image and the measured value based on the CT image is 0.776 when the feedback processing related to the loss of muscle mass is not performed, and the correlation coefficient is 0.874 when the feedback processing related to the loss of muscle mass is performed. Similarly, according to Fig.28A Right side and Fig.28BAs can be seen from the graph on the right, for the fat-free muscle mass of the iliac muscle, the correlation coefficient is 0.804 when no feedback processing related to the loss of muscle mass is performed, and the correlation coefficient is 0.861 when the feedback processing is performed, and a higher correlation value is obtained when the feedback processing related to the loss of muscle mass is performed. Therefore, by performing feedback processing related to the loss of muscle mass during the learning process of the learning model 12Mb, even if the learning model 12Mb is learned with a small amount of training data, it is possible to realize a learning model 12Mb that can estimate the muscle mass to the same extent as the measured value based on the CT image and can predict the DRR image.

[0171] In this embodiment, Fig.28A and Fig.28B The chart shows that for the fat-free muscle mass of the gluteus medius and iliacus, a more accurate fat-free muscle mass can be obtained when feedback processing related to the loss of muscle mass is performed during the learning process of the learning model 12Mb. The same result was obtained for the muscle volume of the gluteus medius and iliacus in the verification process of the inventor.

[0172] In the present embodiment, when learning the learning model 12Mb, for the muscle amount of the muscle area calculated based on the DRR image of the muscle area generated by the learning model 12Mb (image conversion layer 12Mb1), the processing of feeding back the error (loss) of the correct muscle amount is the same as in the above-mentioned embodiments, and the same effect as in the above-mentioned embodiments can be obtained. In addition, in the present embodiment, in the learning process of the learning model 12Mb, in addition to the error between the DRR image generated by the image conversion layer 12Mb1 and the correct DRR image, the error (loss) of the muscle amount can also be fed back, so that the learning model 12Mb that can predict the DRR image of the muscle area in the X-ray image with higher accuracy can be realized. In the present embodiment, the modified examples appropriately described in the above-mentioned embodiments can also be applied.

[0173] The above-mentioned embodiments are structures that use simple X-ray images obtained by photographing the target part by an X-ray device as input data for the learning models 12M, 12Ma, and 12Mb, but are not limited to this structure. For example, it is also possible to use an X-ray image obtained by a DXA device as input data for the learning models 12M, 12Ma, and 12Mb. In the case of such a structure, the same processing as the case of using a normal X-ray image (X-ray image photographed by an X-ray device) as input data can be performed, and the same effect can be obtained. In addition, it is possible to use a two-dimensional image (information related to body tissue volume) obtained from a three-dimensional image such as an MRI (Magnetic Resonance Imaging: Nuclear Magnetic Resonance Imaging) image that can estimate the composition of muscles and bones of the target part, instead of the correct DRR image (information related to body tissue volume of the target part obtained from a CT image) used for the training data of the learning models 12M, 12Ma, and 12Mb in the above-mentioned embodiments. In the case of such a structure, the same processing as the case of using a DRR image obtained from a CT image can be performed, and the same effect can be obtained.

[0174] The embodiments disclosed this time should be considered as examples in all points and not restrictive. The scope of the present invention is indicated by the claims rather than the above meaning, and is intended to include all changes within the meaning and scope equivalent to the claims.

[0175] Description of reference numerals:

[0176] 10 Information processing device

[0177] 11. Control Unit

[0178] 12 Storage

[0179] 13 Ministry of Communications

[0180] 14 Input section

[0181] 15 Display

[0182] 12a Medical Image DB

[0183] 12b Training DB

[0184] 12M Learning Model

[0185] 12Ma Learning Model

[0186] 12M1 Learning model for bone density estimation

Claims

1. A program, characterized in that Make the computer execute the following processing: Acquiring training data, wherein the training data includes an X-ray image of a target part and information related to body tissue volume obtained from a CT (Computed Tomography) image of the target part; Using the acquired training data, a learning model is generated that outputs information related to the body tissue volume of the target site in the X-ray image when an X-ray image is input.

2. The program according to claim 1, characterized in that The computer is caused to execute the following processing: The learning model is generated so as to output an image representing a body tissue volume of the target site when the X-ray image is input.

3. The program according to claim 1 or 2, characterized in that The computer is caused to execute the following processing: aligning a position of the target site based on the CT image with a position of the target site based on the X-ray image; The training data including information related to the body tissue volume of the target site obtained from the aligned CT image is acquired.

4. The program according to any one of claims 1 to 3, characterized in that The computer is caused to execute the following processing: Based on the CT image, classifying the object part in the CT image into a plurality of regions, the plurality of regions including a bone region and a muscle region; acquiring the training data including information related to the bone density of the bone region in the classified CT image; The learning model that outputs information related to the bone density of the bone region in the X-ray image when an X-ray image is input is generated using the training data.

5. The program according to any one of claims 1 to 4, characterized in that The computer is caused to execute the following processing: Based on the CT image, classifying the object part in the CT image into a plurality of regions, the plurality of regions including a bone region and a muscle region; acquiring the training data including information related to the muscle mass of the muscle region in the classified CT image; The learning model that outputs information related to the muscle volume of the muscle region in the X-ray image when an X-ray image is input is generated using the training data.

6. The program according to any one of claims 1 to 5, characterized in that The learning model comprises: a generator that generates information related to the body tissue volume of the target part based on an input X-ray image; and a recognizer that recognizes the authenticity of the information related to the body tissue volume generated by the generator. The program causes the computer to execute the following processing: The learning model is generated by allowing the generator and the discriminator to learn adversarially.

7. The program according to any one of claims 1 to 6, characterized in that: The computer is caused to execute the following processing: acquiring training data, the training data including information related to the body tissue volume of the target part obtained based on the CT image of the target part, and information related to the body tissue volume of a part different from the target part; Using the acquired training data, a second learning model is generated that outputs information related to the body tissue mass of a site other than the target site when information related to the body tissue mass of the target site is input.

8. The program according to any one of claims 1 to 7, characterized in that The computer is caused to execute the following processing: determining a projection condition in which a correlation value between an image obtained by projecting a bone region included in the target part in the CT image and the bone region included in the target part in the X-ray image becomes maximum; The training data is acquired, wherein the training data includes information related to a body tissue volume of the target part obtained from a projection image obtained by projecting the target part in the CT image under a determined projection condition.

9. The program according to claim 8, characterized in that The computer is caused to execute the following processing: deleting data of a bone region from a projection image obtained by projecting the object part in the CT image under the determined projection condition; acquiring the training data including information related to the muscle mass of the muscle region in the projection image from which the data of the bone region is deleted; The learning model that outputs information related to the muscle volume of a muscle region in the X-ray image when an X-ray image is input is generated using the training data.

10. The program according to any one of claims 1 to 9, characterized in that The training data includes an X-ray image of the target part, an image representing a muscle region of the target part obtained from a CT image of the target part, and a muscle mass of the muscle region. The learning model is configured to output an image representing a muscle region in the X-ray image when an X-ray image is input. The program causes the computer to execute the following processing: The learning model is learned in the following manner: the muscle mass calculated based on the image representing the muscle area in the X-ray image output by the learning model when the X-ray image included in the training data is input is close to the muscle mass included in the training data.

11. A program, characterized in that The computer is caused to execute the following processing: Acquiring an X-ray image of the object part; The acquired X-ray image is input into a learning model, and information related to the body tissue volume of the object part is output. The learning model is learned using training data including the X-ray image of the object part and the information related to the body tissue volume obtained based on the CT image of the object part, and outputs the information related to the body tissue volume of the object part in the X-ray image when the X-ray image is input.

12. The program according to claim 11, characterized in that The output information related to body tissue volume is an image showing the body tissue volume of the target site.

13. The program according to claim 11 or 12, characterized in that The output information related to body tissue mass is the bone density of the target site.

14. The program according to any one of claims 11 to 13, characterized in that: The computer is caused to execute the following processing: Information related to the volume of body tissue of a site other than the target site in the X-ray image is also output.

15. The program according to any one of claims 11 to 14, characterized in that: The output information related to body tissue mass is the muscle mass of the target area.

16. The program according to any one of claims 11 to 15, characterized in that: The learning model uses the training data to learn in the following manner: when an X-ray image is input, information related to the muscle mass of the muscle area in the X-ray image is output, the training data includes information related to the muscle mass of the muscle area in the projection image, and the projection image is obtained by deleting data of the bone area from the projection image obtained by projecting the object part in the CT image under a projection condition, and the projection condition is a projection condition in which the correlation value between the image obtained by projecting the bone area included in the object part in the CT image and the bone area included in the object part in the X-ray image becomes the maximum.

17. The program according to any one of claims 11 to 16, characterized in that: The learning model uses the training data to learn in the following manner: the muscle mass calculated based on the image representing the muscle area in the X-ray image output when the X-ray image included in the training data is input is approximate to the muscle mass included in the training data, and the training data includes the X-ray image of the object part, the image representing the muscle area of ​​the object part obtained based on the CT image of the object part, and the muscle mass of the muscle area.

18. An information processing method, characterized in that: The following processing is performed by the computer: Acquiring an X-ray image of the object part; The acquired X-ray image is input into a learning model, and information related to the body tissue volume of the object part is output. The learning model is learned using training data including the X-ray image of the object part and the information related to the body tissue volume obtained based on the CT image of the object part, and when the X-ray image is input, the learning model outputs the information related to the body tissue volume of the object part in the X-ray image.

19. An information processing device comprising a control unit, characterized in that: The control unit is used to: Acquire an X-ray image of the target area, The acquired X-ray image is input into a learning model, and information related to the body tissue volume of the object part is output. The learning model is learned using training data including the X-ray image of the object part and the information related to the body tissue volume obtained based on the CT image of the object part, and when the X-ray image is input, the learning model outputs the information related to the body tissue volume of the object part in the X-ray image.

Images