Program, model generation method, neural network system, information processing method, and information processing device.

Abstract

This provides a program that can accurately predict information about bone condition from simple X-ray images. [Solution] The computer acquires a simple X-ray image of a subject and attributes of the subject, including the subject's age, according to the program. The computer then reads a learning model having a feature extraction layer that extracts feature quantities from the subject's simple X-ray image and a second layer located downstream of the feature extraction layer that outputs information about the subject's bone condition when the feature quantities extracted by the feature extraction layer and the subject's attributes are input. The acquired simple X-ray image is input to the feature extraction layer of the read learning model, and the acquired attributes and the feature quantities extracted by the feature extraction layer are input to the second layer, thereby outputting information about the subject's bone condition.

Description

Technical Field

[0006] ,

[0001] The present disclosure relates to a program, a model generation method, a neural network system, an information processing method, and an information processing apparatus.

Background Art

[0002] With the progress of aging, the number of patients with osteoporosis is increasing. Osteoporosis causes fractures, and fractures in the elderly not only interfere with daily life but may also lead to a bedridden or nursing care-required state. Therefore, appropriate therapeutic intervention needs to be carried out early. Bone density is used for the diagnosis of osteoporosis. In Patent Document 1, a technique for estimating bone density based on learned parameters from input information having a frontal image of a simple X-ray image of a human skeleton has been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the technique disclosed in Patent Document 1, it is necessary to learn a large amount of teacher data, and the burden of collecting and learning teacher data is large.

[0005] In one aspect, an object is to provide a program or the like capable of accurately predicting information regarding the state of bones from a simple X-ray image.

Means for Solving the Problems

[0006] The program, relating to one aspect, reads a learning model having a feature extraction layer that acquires a simple X-ray image of a subject and attributes including the subject's age, and extracts feature quantities from the subject's simple X-ray image, and a second layer located downstream of the feature extraction layer that outputs information regarding the subject's bone condition when the feature quantities extracted by the feature extraction layer and the subject's attributes are input. The program then inputs the acquired simple X-ray image into the feature extraction layer of the read learning model, and inputs the acquired attributes and the feature quantities extracted by the feature extraction layer into the second layer, thereby causing the computer to execute a process that outputs information regarding the subject's bone condition. [Effects of the Invention]

[0007] In one respect, it is possible to predict information about the condition of bones with high accuracy from simple X-ray images. [Brief explanation of the drawing]

[0008] [Figure 1] This is an explanatory diagram showing an example of the configuration of an information processing system. [Figure 2] This is a block diagram showing an example configuration of an information processing device and an image server. [Figure 3] This is an explanatory diagram showing an example of a learning model configuration. [Figure 4] This is a flowchart illustrating an example of the process for generating a learning model. [Figure 5] This flowchart shows an example of a bone density prediction process. [Figure 6] This is an explanatory diagram showing an example screen. [Figure 7] This is an explanatory diagram showing an example screen. [Figure 8] This is an explanatory diagram showing a modified example of the learning process of a learning model. [Figure 9] This is an explanatory diagram showing variations of the learning model. [Figure 10] This is an explanatory diagram showing an example configuration of the learning model of Embodiment 2. [Figure 11]This flowchart shows an example of the bone density prediction processing procedure in Embodiment 2. [Figure 12] This is an explanatory diagram showing an example of a learning model configuration for males. [Figure 13] This flowchart shows an example of the bone density prediction processing procedure in Embodiment 3. [Figure 14] This is an explanatory diagram showing an example of a future bone density prediction model. [Figure 15] This flowchart shows an example of the bone density prediction processing procedure in Embodiment 4. [Modes for carrying out the invention]

[0009] The program, model generation method, neural network system, information processing method, and information processing device of this disclosure will be described below with reference to drawings illustrating their embodiments.

[0010] (Embodiment 1) This invention describes an information processing device that predicts the bone density (information about the condition of bones) of a subject based on a simple X-ray image taken of the subject using an X-ray imaging device. The information processing device of this embodiment predicts the bone density (BMD: Bone Mineral Density, unit: g / cm2) of a subject based on a two-dimensional simple X-ray image (chest X-ray image) taken of the subject's chest using an X-ray imaging device, and outputs information according to the prediction result. The subject includes patients of medical institutions and those undergoing health checkups. The imaging site is the chest, which is also taken in general health checkups, but it may also be the neck, abdomen, arms, hands, legs, or feet, as long as it is a site where bones can be imaged.

[0011] FIG. 1 is an explanatory diagram showing a configuration example of an information processing system. The information processing system of the present embodiment includes a medical imaging device 30, an image server 20 that stores medical images taken by the medical imaging device 30, and an information processing device 10. The information processing device 10 and the image server 20 are communicatively connected via a network N. The network N may be a dedicated network, the Internet, or a public communication line, or may be a LAN (Local Area Network) constructed in a medical institution or the like where the information processing system is provided. The image server 20 and the medical imaging device 30 may be configured to communicate directly by wired communication or wireless communication via a cable, or may be configured to communicate via the network N.

[0012] The medical imaging device 30 includes an X-ray imaging device that irradiates a subject with X-rays and visualizes the X-rays that have passed through the subject's body to capture an image of a desired site. The image server 20 is a server that stores simple X-ray images (simple X-ray image data) taken by the X-ray imaging device of the medical imaging device 30, for example, in association with the information of the subject. In addition to the X-ray imaging device, the medical imaging device 30 may include a computed tomography (CT) device and a magnetic resonance imaging (MRI) device. In this case, the image server 20 stores CT images taken by the CT device and MRI images taken by the MRI device in addition to the simple X-ray images. Further, the medical imaging device 30 may include a bone density measurement device such as a DXA (Dual-energy X-ray Absorptiometry) device that measures bone density using X-rays of two energy levels, an ultrasonic device that measures the bone density of bones such as heels using ultrasonic waves, and an MD (Micro Densitometry) device that measures the bone density of the bones of the hand using X-rays.

[0013] The image server 20 is a device capable of various information processing and information transmission and reception, and is composed of, for example, a server computer, a personal computer, etc. The image server 20 may be a local server installed in, for example, a medical institution, or may be a cloud server communicatively connected via a network N such as a dedicated network or the Internet. The information processing device 10 is a device capable of various information processing and information transmission and reception, and is composed of, for example, a personal computer, a server computer, a workstation, etc. The information processing device 10 is installed in, for example, a medical institution and used by a doctor or the like. In the present embodiment, a doctor or the like accesses the image server 20 using the information processing device 10, receives a medical image stored in the image server 20, displays it on the information processing device 10, and diagnoses the condition of the subject based on the displayed medical image.

[0014] FIG. 2 is a block diagram showing a configuration example of the information processing device 10 and the image server 20. The information processing device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are connected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), a MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), and an AI chip (semiconductor for AI). The control unit 11 executes the processes to be performed by the information processing device 10 by appropriately executing the program 12P stored in the storage unit 12.

[0015] The memory unit 12 includes RAM (Random Access Memory), flash memory, hard disk, SSD (Solid State Drive), etc. The memory unit 12 pre-stores the program 12P (program product) executed by the control unit 11 and various data necessary for the execution of program 12P. The memory unit 12 also temporarily stores data generated when the control unit 11 executes program 12P. The memory unit 12 also stores a learning model M that has been trained on training data by machine learning. The learning model M is intended to be used as a program module that constitutes artificial intelligence software. The learning model M performs predetermined calculations on input values ​​and outputs the calculation results. The memory unit 12 stores data that defines the learning model M, such as information on the layers of the learning model M, information on the nodes constituting each layer, and weights (connection coefficients) between nodes. The learning model M may be stored in other storage devices connected to the information processing device 10, or in other storage devices that the information processing device 10 can communicate with.

[0016] The communication unit 13 is a communication module for connecting to the network N via wired or wireless communication, and transmits and receives information with other devices via the network N. The input unit 14 receives operation input from the user of the information processing device 10 and sends control signals corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display or an organic EL display, etc., and displays various information according to instructions from the control unit 11. Part of the input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit. Note that the input unit 14 and the display unit 15 are not mandatory, and the information processing device 10 may be configured to receive operations through a connected computer, or to output the information to be displayed to an external display device.

[0017] The reading unit 16 reads information stored on a portable storage medium 10a such as a CD (Compact Disc), DVD (Digital Versatile Disc), USB (Universal Serial Bus) memory, SD card, microSD card, or CompactFlash (registered trademark). The program 12P (program product) and various data to be stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 10a via the reading unit 16 and stored in the storage unit 12. Alternatively, the program 12P and various data may be written to the storage unit 12 during the manufacturing stage of the information processing device 10, or the control unit 11 may download them from another device via the communication unit 13 and store them in the storage unit 12.

[0018] In this embodiment, the information processing device 10 is not limited to a single computer, but may be a computer system consisting of multiple computers and peripheral devices, or a multicomputer system comprising multiple computers. Furthermore, the information processing device 10 may be a virtual machine virtually constructed by software within a single device. When the information processing device 10 is configured as a server computer, it may be a local server installed in a medical institution, etc., or a cloud server connected via a network N such as the Internet. In the following description, the information processing device 10 will be described as a single computer. Also, the program 12P may be executed on a single computer, or on multiple computers interconnected via a network N.

[0019] The image server 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, a reading unit 26, etc., and each of these units is connected via a bus. The control unit 21, storage unit 22, communication unit 23, input unit 24, display unit 25, and reading unit 26 each have the same configuration as the control unit 11, storage unit 12, communication unit 13, input unit 14, display unit 15, and reading unit 16 of the information processing device 10, so a detailed explanation is omitted. In addition to the program 22P (program product) executed by the control unit 21, the storage unit 22 of the image server 20 also stores the image DB 22a. The image DB 22a is a database that stores medical images taken by the medical image acquisition device 30. Medical images are, for example, images in an image format called DICOM (Digital Imaging and Communications in Medicine). In DICOM images, subject information such as the subject's ID, name, age (date of birth), gender, height, and weight can be attached to the image file as tag information. Therefore, in image DB22a, the information of the subject being photographed is associated with the medical image data (image file) and stored.

[0020] The information processing device 10 of this embodiment predicts the bone density of a subject based on a frontal chest X-ray image taken from the front of the subject using an X-ray imaging device, for example, including the subject's chest, including the clavicle and ribs, within the imaging range.Hereafter, the frontal chest X-ray image may be simply referred to as a chest X-ray image.Specifically, as will be described later, the information processing device 10 prepares a learning model M that has previously performed machine learning to learn predetermined training data, and takes a chest X-ray image as input and outputs the bone density (information about the bone condition) of the bones (ribs, clavicle, etc.) captured in the input chest X-ray image.The information processing device 10 then inputs the chest X-ray image of the subject to be predicted into the learning model M and obtains the bone density of the subject from the learning model M.

[0021] Figure 3 is an explanatory diagram showing an example of the configuration of the learning model M. The learning model M is a pre-trained model that takes a subject's chest X-ray image (simple X-ray image), age, and sex as input data, performs calculations to predict the subject's bone density based on the input data, and outputs the calculation results. The learning model M (neural network system) has a feature extraction layer M1, a bone density prediction layer M2 (second layer) located after the feature extraction layer M1, a first input layer M3, and a second input layer M4. The chest X-ray image, which is the input data for the learning model M, is input to the first input layer M3, and then input to the feature extraction layer M1 via the first input layer M3. The feature extraction layer M1 extracts and outputs the image features of the input chest X-ray image. The subject's age and sex, which are the input data for the learning model M, are input to the second input layer M4, and then input to the bone density prediction layer M2 via the second input layer M4. Therefore, the features of the chest X-ray image output from the feature extraction layer M1, along with the age and sex input via the second input layer M4, are input to the bone density prediction layer M2. The bone density prediction layer M2 performs calculations to predict the bone density of the subject based on the input data, and outputs the predicted bone density as the output data of the learning model M.

[0022] The feature extraction layer M1 is constructed using an existing model such as ResNet (Residual Network). Specifically, the output layer (fully connected layer) of ResNet is removed, and the feature extraction layer M1 is constructed using the input layer and hidden layers (convolutional layers that perform convolutional operations) of ResNet. In addition to ResNet, the feature extraction layer M1 may also be constructed using the input layer and hidden layers of image recognition models such as VGG and EfficientNet. The bone density prediction layer M2 can be constructed using algorithms such as CNN (Convolutional Neural Network), decision trees, random forests, SVM (Support Vector Machine), and Transformers, and may be constructed using a combination of multiple algorithms. Furthermore, the bone density prediction layer M2 may be constructed using only the output layer (fully connected layer) of a model such as CNN, or it may be constructed using multiple fully connected layers.

[0023] The learning model M is generated by preparing training data that associates the chest X-ray image (plain X-ray image), age, and sex (attributes) of the training subject with the correct bone density (information about the subject's bone condition) of the subject, and then performing machine learning using this training data. The training chest X-ray image can be an image taken by an X-ray imaging device, and the correct bone density can be the bone density measured in the subject's ribs, clavicle, etc., using a DXA device, for example. Alternatively, the correct bone density may be the bone density measured in the femoral neck or proximal femur of the subject. The training data is generated by associating chest X-ray images, age, sex, and bone density for subjects diagnosed with normal bone density, subjects diagnosed with osteoporosis, and subjects diagnosed with osteopenia. The training data thus generated is stored in a training DB (not shown) prepared in the memory unit 12, for example. The subject's age and sex can be obtained, for example, from the tag information of the chest X-ray image.

[0024] In the learning process, the information processing device 10 fixes the parameters (connection coefficients such as weights between nodes) in the feature extraction layer M1, and then inputs the chest X-ray image, age, and gender included in the training data into the learning model M, training the learning model M to output the correct bone density. Specifically, the information processing device 10 inputs the chest X-ray image into the feature extraction layer M1 via the first input layer M3, the feature extraction layer M1 extracts the features from the input chest X-ray image, and outputs the extraction results. The information processing device 10 also inputs the features output from the feature extraction layer M1 and the subject's age and gender input via the second input layer M4 into the bone density prediction layer M2. The bone density prediction layer M2 performs calculations to predict the subject's bone density based on the input features, age, and gender, and outputs the prediction results. The learning model M then compares the predicted bone density with the correct bone density and optimizes the parameters such as the weights (connection coefficients) between nodes in the bone density prediction layer M2 so that the two approximate each other. Methods for optimizing the parameters can include the steepest descent method and backpropagation. This results in a learning model M that outputs the bone density of a subject when their chest X-ray image, age, and sex are input.

[0025] The learning model M shown in Figure 3 is configured to output bone density, for example, as a continuous value, and outputs one of the continuous values ​​as a so-called regression problem. However, it may also be configured to treat it as a classification problem, and to determine which of the selectable values ​​(bone density values ​​that a person can have) is correct for bone density. In this case, the learning model M determines the optimal value from a set of pre-set values ​​for bone density and outputs the determination result. When treated as a classification problem, the learning model M may be configured to have multiple output nodes corresponding to each selectable value for bone density, and each output node outputs the probability (confidence level) that it should be determined to be the corresponding value. In this case, the output value of each output node is, for example, a value between 0 and 1, and the sum of the probabilities output from each output node is 1.0 (100%). In such a configuration, when the subject's chest X-ray image, age, and sex are input, the learning model M outputs the probability that it should be determined to be the bone density assigned to each output node. In a learning model M with this configuration, the information processing device 10 identifies the numerical value associated with the output node that outputs the highest output value (confidence level) among the output values ​​from each output node as the bone density of the subject to be predicted. Alternatively, instead of having multiple output nodes that output confidence levels for each numerical value, the learning model M with this configuration may have a single output node that outputs the numerical value (bone density) with the highest confidence level.

[0026] Furthermore, in the learning process, in addition to optimizing the parameters in the bone density prediction layer M2, the parameters in the feature extraction layer M1 may also be optimized. In this case, for example, the information processing device 10 does not fix the parameters in the feature extraction layer M1, but instead inputs the chest X-ray image, age, and gender included in the training data into the learning model M and trains it to output the correct bone density. Specifically, the information processing device 10 inputs the chest X-ray image into the feature extraction layer M1, and inputs the features output from the feature extraction layer M1, along with the subject's age and gender, into the bone density prediction layer M2. The learning model M then compares the predicted bone density output from the bone density prediction layer M2 with the correct bone density and optimizes the parameters in the feature extraction layer M1, in addition to the parameters in the bone density prediction layer M2, so that the two approximate each other. In this case, a learning model M is obtained in which the parameters in both the bone density prediction layer M2 and the feature extraction layer M1 are optimized.

[0027] The learning model M may be trained using another learning device. The trained learning model M generated by training using another learning device is downloaded from the learning device to the information processing device 10 via the network N or the portable storage medium 10a and stored in the storage unit 12. The learning model M is not limited to the configuration shown in Figure 3. In addition to features extracted from chest X-ray images, age, and sex, the learning model M may also be configured to input information such as the subject's height, weight, features extracted from CT images, and features extracted from MRI images into the bone density prediction layer M2.

[0028] The process of generating a learning model M by learning from training data is described below. Figure 4 is a flowchart showing an example of the procedure for generating the learning model M. The following process is performed by the control unit 11 of the information processing device 10 according to the program 12P stored in the memory unit 12, but it may also be performed by other learning devices.

[0029] The control unit 11 of the information processing device 10 first acquires the information to be used as training data from the image server 20 to generate training data, and then uses the generated training data to train the learning model M. In the following process, it is assumed that the simple X-ray images and bone density used to generate the training data are stored in the image DB 22a of the image server 20, consisting of chest X-ray images taken of the subject's chest using an X-ray imaging device and bone density measured using a DXA device.

[0030] The control unit 11 of the information processing device 10 acquires a chest X-ray image (plain X-ray image), attributes (age and sex), and bone density of one subject from the image DB 22a of the image server 20 (S11). The subject's attributes can be acquired, for example, from the tag information of the DICOM data if the chest X-ray image or bone density measurement information is DICOM data. The control unit 11 is not limited to acquiring the chest X-ray image, attributes, and bone density information from the image server 20, and may acquire the information directly or via the network N from the X-ray imaging device and DXA device of the medical image acquisition device 30. The control unit 11 associates the acquired bone density as the correct label with the acquired chest X-ray image and attributes and stores it in the storage unit 12 as training data (S12). The control unit 11 stores the training data in a training DB (not shown) provided in the storage unit 12, for example.

[0031] The control unit 11 determines whether there is any unprocessed data among the data stored in the image DB 22a of the image server 20 for which the above-mentioned training data has not been generated (S13). If it determines that there is unprocessed data (S13: YES), the control unit 11 returns to the process of step S11 and executes the processes of steps S11 to S12 for the data for which the generation of training data has not been processed, and stores the training data associated with the chest X-ray images, attributes, and bone density of other subjects in the storage unit 12. The control unit 11 repeats the processes of steps S11 to S13 until it determines that there is no unprocessed data. As a result, training data used for training the learning model M is generated and stored in the training DB based on the chest X-ray images, attributes, and bone density stored in the image DB 22a of the image server 20.

[0032] If the control unit 11 determines that there is no unprocessed data (S13: NO), it trains the learning model M using the training data stored in the training DB as described above. The control unit 11 reads one of the training data stored in the training DB (S14) and performs the learning process of the learning model M based on the read training data (S15). Here, the control unit 11 inputs the chest X-ray image and attributes (age and gender) included in the training data into the learning model M and obtains output data from the learning model M. The learning model M extracts features from the input chest X-ray image using the feature extraction layer M1, and the bone density prediction layer M2 performs calculations based on the features of the chest X-ray image extracted by the feature extraction layer M1 and the input attributes (age and gender), and outputs the calculated output value. The control unit 11 compares the output value output from the learning model M with the correct bone density included in the training data and trains the learning model M so that the two approximate each other. During the learning process, the learning model M optimizes parameters such as the weights between nodes in the bone density prediction layer M2 by using, for example, a backpropagation method that sequentially updates from the output layer to the input layer. If the learning process also optimizes the parameters in the feature extraction layer M1 in addition to the bone density prediction layer M2, the learning model M optimizes the parameters in the feature extraction layer M1 along with the parameters in the bone density prediction layer M2.

[0033] The control unit 11 determines whether there is any unprocessed training data stored in the training DB that has not yet undergone learning processing (S16). If it determines that there is unprocessed training data (S16: YES), the control unit 11 returns to the process in step S14 and performs the processing in steps S14 to S15 on the unprocessed training data. The control unit 11 repeats the processing in steps S14 to S16 until it determines that there is no unprocessed training data. As a result, the learning process of the learning model M is executed using the training data accumulated in the training DB. If it determines that there is no unprocessed training data (S16: NO), the control unit 11 terminates the series of processes.

[0034] The learning process described above generates a learning model M that outputs the bone density of a subject when the subject's chest X-ray image and attributes are input. The information processing device 10 can use this learning model M to obtain the bone density of the subject predicted from the subject's chest X-ray image and attributes. In the process described above, the training data generation process in steps S11 to S13 and the learning model M generation process in steps S14 to S16 may be performed by separate devices.

[0035] The learning model M can be further optimized by repeatedly performing the learning process using the training data described above. Alternatively, an already trained learning model M can be fine-tuned using the learning process described above with training data specific to each medical institution, in which case a learning model M tailored to each medical institution can be generated.

[0036] Next, we will explain the process of predicting bone density from chest X-ray images of a subject using the learning model M generated as described above. Figure 5 is a flowchart showing an example of the bone density prediction process, and Figures 6 and 7 are explanatory diagrams showing example screens. The following process is executed by the control unit 11 of the information processing device 10 according to the program 12P stored in the storage unit 12.

[0037] The information processing device 10 has a viewer function for users such as doctors to view medical images stored in the image server 20. The user instructs the system to view a chest X-ray image (plain X-ray image) of a desired subject (predicted subject) via the input unit 14. The control unit 11 of the information processing device 10 acquires the chest X-ray image (plain X-ray image) that the user has instructed to view from the image DB 22a of the image server 20 (S21). Note that the control unit 11 is not limited to acquiring chest X-ray images from the image server 20, but may also acquire them directly from the X-ray imaging device of the medical image acquisition device 30 or via the network N. The control unit 11 displays the acquired chest X-ray image on the display unit 15 (S22). The control unit 11 displays a screen such as the one shown in Figure 6. The screen shown in Figure 6 displays information about the subject (e.g., subject ID, subject name, etc.), the chest X-ray image, and the date and time it was taken. The screen shown in Figure 6 displays a bone density analysis button to instruct the user to predict the bone density of the subject based on the displayed chest X-ray image. The user operates the bone density analysis button when they want to predict the bone density from the chest X-ray image.

[0038] The control unit 11 determines whether the bone density analysis button has been operated (S23). If it determines that the button has not been operated (S23: NO), it waits until it is operated. If the control unit 11 receives a command to close the screen without the bone density analysis button being operated, it terminates the screen display and ends the series of processes. If it determines that the bone density analysis button has been operated (S23: YES), the control unit 11 obtains the age and gender of the subject whose chest X-ray image is being displayed (S24). Attributes such as age and gender are included in the tag information of the DICOM image, and the control unit 11 obtains the age and gender of the subject from the tag information of the chest X-ray image. If the age and gender of the subject are not included in the tag information of the chest X-ray image, the control unit 11 may obtain the age and gender of the subject from an electronic medical record server (not shown), etc.

[0039] The control unit 11 predicts the bone density of the subject based on the displayed chest X-ray image and the subject's age and sex acquired in step S24 (S25). Here, the control unit 11 reads the learning model M from the storage unit 12, inputs the chest X-ray image, age, and sex into the learning model M, and obtains the subject's bone density as output data from the learning model M. From the predicted bone density, the control unit 11 calculates a T-score (young adult comparison) showing the comparison result with the average bone density of young adults (YAM: Young Adult Mean), and a Z-score (same age comparison) showing the comparison result with the average bone density of the subject's age group (S26). The control unit 11 stores the predicted bone density and the calculated T-score and Z-score in, for example, an electronic medical record server.

[0040] The control unit 11 generates a measurement results screen that displays the bone density measurement results based on the predicted bone density and the T-score and Z-score calculated from the bone density (S27), and displays the generated measurement results screen on the display unit 15 (S28). The control unit 11 generates and displays a screen such as the one shown in Figure 7. The screen shown in Figure 7 displays the predicted bone density and the calculated T-score and Z-score as the predicted bone density result (measurement result) based on the chest X-ray image of the subject, in the area to the right of the screen shown in Figure 6. For the T-score and Z-score, in addition to the calculated values, an evaluation chart is displayed that shows whether or not they are equivalent to those of young adults or people of the same age based on the calculated values. In the example in Figure 7, an evaluation chart is displayed that if the T-score and Z-score are 80% or higher, they are equivalent to those of young adults or people of the same age; if they are 70% or higher but less than 80%, they are lower than those of young adults or people of the same age; and if they are less than 70%, they are considerably lower than those of young adults or people of the same age. Furthermore, the screen shown in Figure 7 displays a graph with age on the horizontal axis and bone density on the vertical axis, plotting the average bone density of people at each age. The age and bone density of the person being predicted are plotted on the graph with a "+" sign. The bone age of the person being predicted can be predicted by comparing the predicted bone density of the person being predicted with the average bone density of people at each age. Specifically, the control unit 11 compares the predicted bone density of the person being predicted with the average bone density of people at each age and identifies the age that matches the bone density of the person being predicted. The age thus identified becomes the bone age of the person being predicted, and the control unit 11 may display the identified bone age of the person being predicted on the measurement results screen, as shown in Figure 7. The control unit 11 may also calculate the fracture risk from the bone density of the person being predicted using a pre-set calculation formula and display the calculated fracture risk on the measurement results screen. Furthermore, if, for example, comments to be presented to a doctor or other medical professional are stored in the memory unit 12 in association with each numerical value such as bone density, T-score, or Z-score, the control unit 11 may read the comments corresponding to the predicted results (bone density, T-score, or Z-score, etc.) from the memory unit 12 and display them on the measurement results screen.

[0041] Through the process described above, in this embodiment, the bone density of the bones in a plain chest X-ray image taken using an X-ray imaging device commonly used in medical institutions can be predicted and presented to a physician or other medical professional. Therefore, for example, when a subject visits a medical institution and a chest X-ray image is taken, or when a subject takes a chest X-ray image during a health checkup, a physician or other medical professional can determine the bone condition of the subject based on the bone density, T-score, and Z-score predicted from the plain chest X-ray image taken.

[0042] In this embodiment, the learning model M automatically extracts the characteristics of the bone imaging state in the frontal chest X-ray image and predicts bone density. Therefore, bone density can be predicted simply by taking an X-ray image, without the need to perform examinations using a DXA device or the like. Consequently, bone density measurements can be easily performed even for subjects for whom bone density measurement is not the purpose of the examination, enabling early diagnosis and treatment intervention for osteopenia or osteoporosis. Furthermore, for subjects who have used a medical institution for reasons other than osteopenia or osteoporosis, and for subjects who have undergone a health checkup, bone density can be predicted each time a frontal chest X-ray image is taken, allowing for monitoring of bone density even before it declines.

[0043] In this embodiment, an existing model is used in the feature extraction layer M1 of the learning model M. Based on the features extracted from the chest X-ray image by the feature extraction layer M1, and the subject's age and sex, the bone density prediction layer M2 predicts the subject's bone density. Therefore, by optimizing the parameters of the bone density prediction layer M2 during the learning process of the learning model M, it is possible to realize a learning model M that can predict bone density with high accuracy from chest X-ray images without performing a massive amount of learning.

[0044] In this embodiment, a configuration was described in which bone density (e.g., ribs, clavicle, thoracic vertebrae, etc.) in a chest X-ray image is predicted from a frontal chest X-ray image using a learning model M. However, the plain X-ray images to be processed are not limited to frontal chest X-ray images; for example, lateral chest X-ray images taken from the side of the subject's chest may also be used, or plain X-ray images taken of other parts of the body such as the neck, abdomen, arms, hands, legs, or feet. Even when plain X-ray images of other parts of the body are used as the target of processing, training data and a learning model are generated by the same process, and bone density prediction using the learning model is possible. Alternatively, the system may be configured to perform training using plain X-ray images of parts of the subject other than the chest as training data, and then predict bone density using chest X-ray images taken of the subject's chest.

[0045] In this embodiment, the process of generating training data, training the learning model M using the training data, and predicting bone density using the learning model M are not limited to being performed locally by the information processing device 10. For example, separate information processing devices may be provided to perform each of the above-described processes. Alternatively, a server may be provided to perform the training data generation process and the learning model M training process. In this case, the information processing device 10 is configured to send a set of simple X-ray images, attributes, and bone density of the subject to be used as training data to the server, and the server generates training data from the received simple X-ray images, attributes, and bone density, generates a learning model M through a learning process using the generated training data, and sends it to the information processing device 10. In this case, the information processing device 10 can perform bone density prediction processing for the subject using the learning model M obtained from the server. Alternatively, a server may be provided to perform bone density prediction processing using the learning model M. In this case, the information processing device 10 is configured to send simple X-ray images of the subject to be predicted to the server, and the server performs bone density prediction processing using the learning model M and sends the bone density predicted from the received simple X-ray images to the information processing device 10. Even with this configuration, the same processing as in the embodiment described above is possible, and the same effects can be obtained.

[0046] (Variation 1) In the embodiment described above, the learning model M was trained using the subject's chest X-ray frontal image, attributes, and bone density as training data, but the model is not limited to this configuration. For example, the learning model M may be trained using training data that includes a chest X-ray lateral image taken from the side of the subject's chest using an X-ray imaging device, a CT image taken of the subject's chest using a CT scanner, and an MRI image taken of the subject's chest using an MRI scanner. Alternatively, the training data may include plain X-ray images (plain X-ray frontal image, plain X-ray lateral image) taken of parts of the subject other than the chest using an X-ray imaging device, a CT image taken using a CT scanner, and an MRI image taken using an MRI scanner.

[0047] Figure 8 is an explanatory diagram showing a modified example of the learning process of the learning model M. Figure 8 conceptually shows the learning process of the learning model M shown in Figure 3. The learning model M shown in Figure 8 has a feature extraction layer M1, a bone density prediction layer M2, a first input layer M3, and a second input layer M4, as well as feature extraction layers M1a and M1b having the same configuration as the feature extraction layer M1, and first input layers M3a and M3b having the same configuration as the first input layer M3. In the learning process shown in Figure 8, the learning model M learns using training data that associates the subject's frontal chest X-ray image (hereinafter referred to as the frontal chest X-ray image), age and sex, CT image, and lateral chest X-ray image (hereinafter referred to as the lateral chest X-ray image) with the correct bone density of the subject. In the learning process, the information processing device 10 fixes the parameters (connection coefficients such as weights between nodes) in the feature extraction layers M1, M1a, and M1b, and then inputs the frontal chest X-ray image included in the training data to the feature extraction layer M1 via the first input layer M3, the CT image to the feature extraction layer M1a via the first input layer M3a, and the lateral chest X-ray image to the feature extraction layer M1b via the first input layer M3b. The information processing device 10 then inputs the features output from the feature extraction layers M1, M1a, and M1b, along with the age and sex input via the second input layer M4, to the bone density prediction layer M2. The bone density prediction layer M2 performs a calculation to predict bone density based on the three input features, age, and sex, and outputs the prediction result. The learning model M then compares the predicted bone density with the ground truth bone density and optimizes the parameters in the bone density prediction layer M2 so that the two approximate each other.

[0048] Even when performing the learning process described above, a learning model M is obtained that outputs the bone density of a subject when the subject's frontal chest X-ray image, age, and sex are input, as shown in Figure 3. When predicting bone density from a frontal chest X-ray image, the learning model M uses only the feature extraction layer M1 and the bone density prediction layer M2. Furthermore, if the subject's CT image and lateral chest X-ray image are taken and stored in the image DB 22a of the image server 20, the bone density of the subject may be predicted using feature extraction layers M1a and M1b in addition to the feature extraction layer M1 and the bone density prediction layer M2. In this case, in addition to the subject's frontal chest X-ray image, age, and sex, the CT image and lateral chest X-ray image are input to the learning model M, and the bone density is predicted by the learning model M.

[0049] In this modified example, the training data used to train the learning model M may further include the subject's height, weight, medical history, medication history, and other medical images. Even when learning is performed using such training data, the learning model M shown in Figure 3 can be obtained. In the learning model M shown in Figure 8, in addition to the feature extraction layer M1, there is a feature extraction layer M1a that extracts features from CT images and a feature extraction layer M1b that extracts features from lateral chest X-ray images, but the configuration is not limited to this. For example, in the learning process, the CT image and lateral chest X-ray image may be input to the feature extraction layer M1 along with the anterior chest X-ray image, and the features extracted from each image by the feature extraction layer M1 may be input to the bone density prediction layer M2.

[0050] (Modification 2) A modified version of the learning model M will be described. Figure 9 is an explanatory diagram showing a modified version of the learning model M. The learning model Ma shown in Figure 9A takes a chest X-ray image (frontal chest X-ray image), age, and gender of the person to be predicted as input data, performs a calculation to predict the T-score of the person to be predicted based on the input data, and is trained to output the result of the calculation. The learning model Ma has a feature extraction layer Ma1, a T-score prediction layer Ma2, a first input layer Ma3, and a second input layer Ma4. The feature extraction layer Ma1, the first input layer Ma3, and the second input layer Ma4 are the same as the feature extraction layer M1, the first input layer M3, and the second input layer M4 of the learning model M shown in Figure 3. The chest X-ray image, which is the input data for the learning model Ma, is input to the feature extraction layer Ma1 via the first input layer Ma3, and the feature extraction layer Ma1 extracts and outputs the image features of the input chest X-ray image. The features of the chest X-ray image output from the feature extraction layer Ma1, along with the subject's age and gender input via the second input layer Ma4, are input to the T-score prediction layer Ma2. The T-score prediction layer Ma2 performs calculations to predict the subject's T-score based on the input data, and outputs the predicted T-score as the output data of the learning model Ma.

[0051] The learning model Ma shown in Figure 9A is generated by machine learning using training data that associates the chest X-ray image, age, and sex (attributes) of a training subject with the correct T-score of that subject (information about the subject's bone condition). The correct T-score can be, for example, a T-score (comparison between young adults) calculated from bone density measured in the subject's ribs, clavicle, etc. using a DXA device. The learning model Ma can be generated by the same process as shown in Figure 4. In step S11 in Figure 4, the control unit 11 obtains the chest X-ray image, age, sex, and T-score of one subject from the image DB 22a of the image server 20, associates the obtained T-score as the correct label with the obtained chest X-ray image, age, and sex, and stores it in the storage unit 12 as training data. The T-score is assumed to be stored in the image DB 22a of the image server 20, similar to the bone density. As a result, in step S14, the control unit 11 can acquire training data that associates T-scores with chest X-ray images and attributes, and use such training data to train the learning model Ma.

[0052] In the learning model Ma shown in Figure 9A, instead of the T-score, information about bone condition, such as the Z-score or fracture risk calculated from bone density, may be used as output data. In this case as well, it can be generated by the same learning process as learning models M and Ma. When the Z-score is used as the output data for the learning model, for example, the learning model can be generated by training data in which the correct Z-score is calculated from bone density measured in the subject's ribs, clavicle, etc. using a DXA device (comparison with the same age). When fracture risk is used as the output data for the learning model, for example, the learning model can be generated by training data in which the correct fracture risk is calculated from the subject's bone density measured using a DXA device, the subject's age, and gender.

[0053] The learning model Mb shown in Figure 9B is trained to take the target person's chest X-ray image (frontal chest X-ray), age, and sex, as well as the target person's height and weight, as input data, perform calculations to predict the target person's bone density based on the input data, and output the calculation results. The learning model Mb has a feature extraction layer Mb1, a bone density prediction layer Mb2, a first input layer Mb3, and a second input layer Mb4. The feature extraction layer Mb1 and the first input layer Mb3 are the same as the feature extraction layer M1 and the first input layer M3 of the learning model M shown in Figure 3. The height and weight, which are input data for the learning model Mb, are input to the bone density prediction layer Mb2 via the second input layer Mb4. The bone density prediction layer Mb2 calculates the bone density of the subject based on the features of the chest X-ray image extracted by the feature extraction layer Mb1 and the age, sex, height, and weight input via the second input layer Mb4, and outputs the predicted bone density as output data for the learning model Mb.

[0054] The learning model Mb shown in Figure 9B is generated by machine learning using training data that associates the chest X-ray image, age, sex, height, and weight of a training subject with the correct bone density of that subject. The subject's height and weight can be obtained, for example, from the tag information of the chest X-ray image (DICOM image). The learning model Mb can be generated by the same process as shown in Figure 4. In step S11 in Figure 4, the control unit 11 obtains the chest X-ray image, age, sex, height, weight, and bone density of one subject from the image DB 22a of the image server 20, and stores the obtained bone density as the correct label for the obtained chest X-ray image, age, sex, height, and weight in the storage unit 12 as training data. As a result, in step S14, the control unit 11 obtains training data that associates bone density with the chest X-ray image, age, sex, height, and weight, and can train the learning model Mb using this training data.

[0055] The input data for the learning model Mb shown in Figure 9B may consist of the subject's chest X-ray image, age, sex, height, and weight, or it may consist of the subject's chest X-ray image and age, plus at least one of the subject's sex, height, and weight. Furthermore, in the learning model Mb shown in Figure 9B, instead of bone density, information about bone condition such as T-score, Z-score, or fracture risk calculated from bone density may be used as output data. This can also be generated by the same learning process.

[0056] (Embodiment 2) This section describes an information processing device that, when predicting bone density from a subject's simple X-ray image using a learning model M, divides the simple X-ray image into multiple regions and performs feature extraction processing on each of the divided simple X-ray images (hereinafter referred to as divided X-ray images). Since the information processing system of this embodiment can be implemented using the same device as the information processing system of Embodiment 1, a description of the configuration of each device will be omitted.

[0057] Figure 10 is an explanatory diagram showing an example configuration of the learning model M of Embodiment 2. The learning model M of this embodiment is trained to take as input data a plurality of segmented X-ray images obtained by dividing a plain chest X-ray frontal image of a subject into multiple regions (four in Figure 10), age, and sex, perform calculations to predict the bone density of the subject based on the input data, and output the result of the calculation. The learning model M shown in Figure 10 has four feature extraction layers M1, four first input layers M3, a bone density prediction layer M2, a second input layer M4, and a coupling section M5. In the learning model M shown in Figure 10, the four segmented X-ray images divided from the plain chest X-ray frontal image are input to the four feature extraction layers M1 via the four first input layers M3, and features are extracted by each feature extraction layer M1. The feature quantities extracted from each segmented X-ray image by the feature extraction layer M1 are input to the merging layer M5. The merging layer M5 combines the four feature quantities to generate and output a combined feature quantity that represents the feature quantities of the frontal chest X-ray image before segmentation. The combined feature quantity output from the merging layer M5, along with the subject's age and gender input via the second input layer M4, are input to the bone density prediction layer M2. The bone density prediction layer M2 performs a calculation to predict the subject's bone density, and the predicted bone density is output as the output data of the learning model M. Note that the number of segmented X-ray images separated from the frontal chest X-ray image is not limited to four. Also, in the learning model M shown in Figure 10, there are four feature extraction layers M1, and the segmented X-ray images obtained by dividing the frontal chest X-ray image into four are input to each of the four feature extraction layers M1. However, it is also possible to have a configuration with one feature extraction layer M1, and the four segmented X-ray images are sequentially input to this one feature extraction layer M1.

[0058] The training data for the learning model M in this embodiment is the same as the training data for the learning model M in Embodiment 1. Specifically, the learning model M in this embodiment is generated by machine learning using training data that associates the chest X-ray image, age, and sex of a training subject with the correct bone density of that subject. The learning model M in this embodiment can also be generated by the same process as shown in Figure 4. In step S15 in Figure 4, the control unit 11 divides the training chest X-ray image into multiple regions (for example, four), inputs each divided X-ray image into the feature extraction layer M1, and extracts the features of each divided X-ray image. The multiple features extracted by the feature extraction layer M1 are input into the merging unit M5, merged by the merging unit M5, and the bone density prediction layer M2 performs calculations based on the merged features and the attributes (age and sex) input for training, and outputs the calculated output value. The control unit 11 compares the output value from the learning model M with the correct bone density included in the training data, and trains the learning model M so that the two approximate each other. In addition, if the parameters in the feature extraction layer M1 are optimized in addition to the bone density prediction layer M2 during the learning process, the learning model M optimizes the parameters in the feature extraction layer M1 along with the parameters in the bone density prediction layer M2.

[0059] Figure 11 is a flowchart showing an example of the bone density prediction processing procedure in Embodiment 2. The process shown in Figure 11 is the same as the process shown in Figure 5, but with step S31 added between steps S24 and S25. The steps that are the same as in Figure 5 will not be explained. In this embodiment, after processing in step S24, the control unit 11 of the information processing device 10 divides the chest X-ray image (plain X-ray image) displayed on the screen shown in Figure 6 into multiple regions (S31). Then, the control unit 11 predicts the bone density of the subject (prediction subject) based on the divided X-ray images and the age and sex of the subject obtained in step S24 (S25). Here, the control unit 11 inputs the divided X-ray images, age and sex to the learning model M and obtains the bone density of the subject as output information from the learning model M. The learning model M extracts features from the input segmented X-ray images using the feature extraction layer M1, combines the extracted features using the merging unit M5, and predicts bone density using the bone density prediction layer M2 based on the combined features, age, and sex. Subsequently, the control unit 11 performs the processing from step S26 onward.

[0060] In this embodiment as well, the same effects as in Embodiment 1 described above can be obtained. Furthermore, in this embodiment, in the learning model M, multiple feature extraction layers M1 extract features for each segmented X-ray image obtained by dividing the chest plain X-ray frontal image, so that the feature extraction processing of the segmented X-ray images by the feature extraction layers M1 can be performed in parallel. Therefore, it is possible to shorten the feature extraction processing time and speed up the processing. In this embodiment as well, the modifications described as appropriate in Embodiment 1 described above can be applied.

[0061] (Embodiment 3) This document describes an information processing device that predicts bone density from simple X-ray images of a subject using male and female learning models M. Since the information processing system of this embodiment can be implemented using the same equipment as the information processing system of Embodiment 1, a description of the configuration of each device will be omitted. In this embodiment, the storage unit 12 of the information processing device 10 stores a male learning model Mc and a female learning model (not shown) instead of the learning model M.

[0062] Figure 12 is an explanatory diagram showing an example configuration of the male learning model Mc. The male learning model Mc is trained to take a frontal chest X-ray image (plain X-ray image) and age of a male subject as input data, perform a calculation to predict the bone density of the subject based on the input data, and output the result of the calculation. The male learning model Mc has the same configuration as the learning model M of Embodiment 1 shown in Figure 3, and includes a feature extraction layer Mc1, a bone density prediction layer Mc2, a first input layer Mc3, and a second input layer Mc4. In the male learning model Mc, the features of the chest X-ray image output from the feature extraction layer Mc1 and the age of the subject input via the second input layer Mc4 are input to the bone density prediction layer Mc2, and the bone density prediction layer Mc2 performs a calculation to predict the bone density of the subject, and the predicted bone density is output as the output data of the male learning model Mc.

[0063] The male learning model Mc is generated by machine learning using training data that associates chest X-ray images and age of male subjects with the correct bone density of those subjects. The male learning model Mc can also be generated by the same process as shown in Figure 4. In step S11 in Figure 4, the control unit 11 obtains chest X-ray images, age, and bone density of male subjects from the image DB 22a of the image server 20, associates the obtained bone density as the correct label with the obtained chest X-ray images and age, and stores it in the storage unit 12 as training data. As a result, in step S14, the control unit 11 obtains training data that associates bone density with chest X-ray images and age, and can use this training data to train the male learning model Mc. The female learning model has the same configuration as the male learning model Mc and can be generated by the same learning process. The training data for the female learning model can use data that associates chest X-ray images and age of female subjects with the correct bone density of those subjects.

[0064] Figure 13 is a flowchart showing an example of the bone density prediction processing procedure in Embodiment 3. The process shown in Figure 13 is the same as the process shown in Figure 5, but with the addition of step S41 between steps S24 and S25. The steps that are the same as in Figure 5 will not be explained. In this embodiment, after processing in step S24, the control unit 11 of the information processing device 10 reads out the learning model (male learning model Mc or female learning model) corresponding to the gender of the person to be predicted, which was acquired in step S24, from the storage unit 12 (S41). Then, the control unit 11 uses the read-out learning model to predict the bone density of the person to be predicted based on the displayed chest X-ray frontal image and the age of the person (S25). After that, the control unit 11 performs the processing from step S26 onwards.

[0065] The configuration of this embodiment is applicable to embodiments 1 and 2 described above, and similar processing can be performed and similar effects can be obtained even when applied to embodiments 1 and 2. In this embodiment, by using a male learning model Mc and a female learning model, a learning model specific to the gender of the person to be predicted can be realized, making it possible to predict bone density with greater accuracy. In this embodiment as well, the modifications described as appropriate in embodiments 1 and 2 described above can be applied.

[0066] (Embodiment 4) This document describes an information processing device that predicts a subject's future bone density based on periodically acquired bone density data. Since the information processing system in this embodiment can be implemented using the same equipment as the information processing system in Embodiment 1, a detailed explanation of the configuration of each device will be omitted. In this embodiment, the storage unit 12 of the information processing device 10 further stores a future bone density prediction model M6 (second learning model).

[0067] Figure 14 is an explanatory diagram showing an example of the configuration of the future bone density prediction model M6. The future bone density prediction model M6 shown in Figure 14A is trained to take time-series bone density data of a subject acquired periodically as input data, perform calculations to predict the subject's future bone density based on the input data, and output the calculation results. The model M6 shown in Figure 14A predicts bone density at 1 year, 3 years, 5 years, 7 years, and 10 years in the future. The bone density predicted by the future bone density prediction model M6 is not limited to these, and may be bone density every year (e.g., 1 year, 2 years, 3 years, etc.), bone density more than 10 years in the future, or bone density at any future time (e.g., 5 years, 10 years). The future bone density prediction model M6 can be constructed using algorithms such as LSTM (Long Short Term Memory), RNN (Recurrent Neural Network), and Transformer (BERT, GPT (Generative Pre-trained Transformer)), and may be constructed by combining multiple algorithms.

[0068] The future bone density prediction model M6 is generated by machine learning using training data that associates the time-series bone density of a training subject with the ground truth bone density of the same subject in the future (e.g., 1 year, 3 years, 5 years, 7 years, 10 years later) (information about the future bone state). The training time-series bone density and the ground truth future bone density can be, for example, bone density measured in the subject's ribs, clavicle, femoral neck, or proximal femur using a DXA device. Alternatively, bone density predicted from the subject's plain X-ray image (e.g., frontal chest X-ray image) may be used as the training time-series bone density and the ground truth future bone density. The training subject's bone density can be obtained from the image DB22a of the image server 20, and statistical data of bone density aggregated from people of each age may also be used. The correct bone density (future bone density) will be the bone density measured on a date that is a predetermined period (e.g., 1 year, 3 years, 5 years, 7 years, 10 years) after the last bone density measurement in the time series.

[0069] In the learning process, the information processing device 10 inputs the time-series bone density data included in the training data into the future bone density prediction model M6 and trains it to output the correct future bone density. Specifically, the information processing device 10 inputs the time-series bone density data into the future bone density prediction model M6 and obtains the future bone density (1 year, 3 years, 5 years, 7 years, 10 years) output from the future bone density prediction model M6. The future bone density prediction model M6 compares the predicted future bone density with the correct future bone density and optimizes parameters such as node weights (connection coefficients) so that the two approximate each other. As a result, a future bone density prediction model M6 is obtained that outputs the future bone density of a subject when the subject's time-series bone density data is input.

[0070] Figure 14B shows a modified version of the future bone density prediction model M6. The future bone density prediction model M6 shown in Figure 14B is trained to take the subject's time-series bone density, age, and sex as input data, perform calculations to predict the subject's future bone density based on the input data, and output the calculation results. The future bone density prediction model M6 shown in Figure 14B is generated by machine learning using training data that associates the training subject's time-series bone density, age, and sex with the correct future bone density of the subject. The training age is the age on the last bone density measurement date in the training time-series bone density data. The future bone density prediction model M6 shown in Figure 14B can also be generated by the same learning process as the future bone density prediction model M6 shown in Figure 14A.

[0071] Figure 15 is a flowchart showing an example of the bone density prediction processing procedure in Embodiment 4. The process shown in Figure 15 is the same as the process shown in Figure 5, but with steps S51 to S52 added between steps S26 and S27. The steps that are the same as in Figure 5 will not be explained. In this embodiment, after processing in step S26, the control unit 11 of the information processing device 10 acquires the past bone density of the person to be predicted (S51). In this embodiment, by periodically performing bone density prediction processing using simple X-ray images (e.g., frontal chest X-ray images), the periodically predicted bone density is registered, for example, in an electronic medical record server. For example, if bone density prediction is performed based on a frontal chest X-ray image taken during a health checkup once a year, bone density data will be accumulated in the electronic medical record server once a year. Therefore, the control unit 11 can acquire time-series bone density data by acquiring the past bone density of the person to be predicted from, for example, the electronic medical record server. Note that the past bone density may include not only the bone density predicted from simple X-ray images, but also the bone density of the person to be predicted measured using a DXA device or the like.

[0072] The control unit 11 adds the bone density predicted in step S25 to the past bone density obtained in step S51 to obtain time-series bone density data, and predicts the future bone density of the subject based on the time-series bone density (S52). Here, the control unit 11 inputs the time-series bone density into the future bone density prediction model M6, and obtains the subject's future bone density as output data from the future bone density prediction model M6. Subsequently, the control unit 11 performs the processing from step S27 onward. In step S27, the control unit 11 generates a measurement result screen that displays the subject's future bone density in addition to the configuration shown in Figure 7. Through the above processing, it is possible to present not only the bone density predicted from the subject's chest X-ray frontal image, but also the future bone density predicted from periodically predicted or measured bone densities.

[0073] The configuration of this embodiment is applicable to embodiments 1 to 3 described above, and similar processing can be performed and similar effects can be obtained even when applied to embodiments 1 to 3. In this embodiment, not only the bone density predicted based on the simple X-ray image at the time the simple X-ray image is taken can be presented, but also the bone density predicted periodically and the future bone density predicted from the bone density measured periodically can be presented. Therefore, doctors and others can diagnose the bone condition of the subject by also considering the predicted future bone density for the subject. In this embodiment as well, the modifications described as appropriate in embodiments 1 to 3 described above can be applied.

[0074] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used.

[0075] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended. [Explanation of symbols]

[0076] 10 Information Processing Devices 11 Control Unit 12 Storage section 13 Communications Department 20 Image Servers 21 Control Unit 22 Memory section 23 Communications Department M Learning Model M1 Feature Extraction Layer M2 bone density prediction layer M3 First Input Layer M4 2nd input layer

Claims

1. Obtain a simple X-ray image of the subject and attributes including the subject's age. A learning model is read, which includes a feature extraction layer that extracts feature quantities from a subject's simple X-ray image, and a second layer located after the feature extraction layer that outputs information about the subject's bone condition when the feature quantities extracted by the feature extraction layer and the subject's attributes are input. The acquired simple X-ray image is input to the feature extraction layer of the read learning model, and the acquired attributes and the features extracted by the feature extraction layer are input to the second layer to output information about the bone condition of the subject. A program that instructs a computer to perform a process.

2. The aforementioned information regarding bone condition includes either a T-score, bone density, or fracture risk. The program according to claim 1.

3. The aforementioned second layer is a plurality of fully connected layers. The program according to claim 1 or 2.

4. The aforementioned attributes further include at least one of the subject's gender, height, and weight. The program according to claim 1 or 2.

5. The acquired simple X-ray image is divided into multiple regions, Each of the divided simple X-ray images is input to the feature extraction layer of the learning model, and features are extracted from each of the divided simple X-ray images. A combined feature is generated by combining multiple extracted features. By inputting the acquired attributes and the generated combined features into the second layer, information regarding the bone condition of the subject is output. The program according to claim 1 or 2, which causes the computer to perform the processing.

6. The aforementioned learning model includes a male learning model and a female learning model. The gender of the aforementioned subject is obtained, If the subject is male, the acquired simple X-ray image is input to the feature extraction layer of the male-oriented learning model, and the acquired attributes and the features extracted by the feature extraction layer are input to the second layer of the male-oriented learning model to output information regarding the subject's bone condition. If the subject is female, the acquired simple X-ray image is input to the feature extraction layer of the female-specific learning model, and the acquired attributes and the features extracted by the feature extraction layer are input to the second layer of the female-specific learning model to output information regarding the subject's bone condition. The program according to claim 1 or 2, which causes the computer to perform the processing.

7. Based on the outputted information regarding the bone condition, the bone age of the subject is determined. The program according to claim 1 or 2, which causes the computer to perform the processing.

8. Obtain information on the state of multiple bones obtained in a time series, When time-series information on the bone state is input to a second learning model that outputs information on the subject's future bone state, the acquired time-series information on the bone state is input to the second learning model, which outputs information on the subject's future bone state. The program according to claim 1 or 2, which causes the computer to perform the processing.

9. Training data is obtained by associating the subject's simple X-ray image and attributes including the subject's age with information regarding the subject's bone condition. Based on the acquired training data, a learning model is generated having a feature extraction layer that extracts feature quantities from the subject's simple X-ray image, and a second layer located after the feature extraction layer that outputs information about the subject's bone condition when the feature quantities extracted by the feature extraction layer and the subject's attributes are input. A model generation method in which a computer performs the processing.

10. A first input layer that receives input of a simple X-ray image of the subject, A feature extraction layer that extracts feature quantities from the simple X-ray image input via the first input layer, A second input layer that accepts input of attributes including the age of the subject, When the features extracted by the feature extraction layer and the attributes input via the second input layer are input, the second layer outputs information about the bone condition of the subject. A neural network system having [a certain characteristic].

11. Obtain a simple X-ray image of the subject and attributes including the subject's age. A learning model is read, which includes a feature extraction layer that extracts feature quantities from a subject's simple X-ray image, and a second layer located after the feature extraction layer that outputs information about the subject's bone condition when the feature quantities extracted by the feature extraction layer and the subject's attributes are input. The acquired simple X-ray image is input to the feature extraction layer of the read learning model, and the acquired attributes and the features extracted by the feature extraction layer are input to the second layer to output information about the bone condition of the subject. An information processing method in which a computer performs the processing.

12. An information processing device having a control unit, The control unit, Obtain a simple X-ray image of the subject and attributes including the subject's age. A learning model is read, which includes a feature extraction layer that extracts feature quantities from a subject's simple X-ray image, and a second layer located after the feature extraction layer that outputs information about the subject's bone condition when the feature quantities extracted by the feature extraction layer and the subject's attributes are input. The acquired simple X-ray image is input to the feature extraction layer of the read learning model, and the acquired attributes and the features extracted by the feature extraction layer are input to the second layer to output information about the bone condition of the subject. Information processing device.

Images