Information processing systems, information processing methods, programs, and recording media

Abstract

This invention provides an information processing system, information processing method, and program for estimating bone density in order to diagnose osteoporosis. [Solution] The information processing system comprises a processing unit and an estimation unit. The processing unit outputs classification information regarding the division of parts included in the first skeleton based on first input information including a first simple X-ray image of the first skeleton of a first person and first learned parameters. The estimation unit estimates the bone density of at least a part of the first skeleton based on second input information based on the first input information and classification information and second learned parameters. The second learned parameters are set based on first training data including a training simple X-ray image of the second skeleton of a second person and first training data including the bone density of the second person.

Description

Technical Field

[0001] This disclosure relates to estimation techniques.

Background Art

[0002] Patent Document 1 describes a technique for diagnosing osteoporosis. Patent Document 2 describes a technique for estimating bone strength.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Means for Solving the Problems

[0004] In one embodiment, an information processing system includes a processing unit and an estimation unit. The processing unit outputs classification information regarding classification of a part included in the first skeleton based on first input information including a first simple X-ray image of a first skeleton possessed by a first person and first learned parameters. The estimation unit estimates bone density of at least a part of the first skeleton based on second input information based on the first input information and the classification information and second learned parameters. The second learned parameters are set based on first learning data including a learning simple X-ray image of a second skeleton possessed by a second person and first teacher data including bone density of the second person.

Brief Description of the Drawings

[0005] [Figure 1] It is a diagram showing an example of the configuration of a computer device (estimation device). [Figure 2] It is a diagram for explaining the operation of the estimation device. [Figure 3] It is a diagram showing an example of the configuration of a neural network. [Figure 4] This figure shows an example of how training image data and reference bone density are correlated with each other. [Figure 5] This is a diagram illustrating the operation of the estimation device. [Figure 6] This figure shows an example of an estimation system. [Figure 7] This figure shows an example of the configuration of an estimation device. [Figure 8] This figure shows an example of the configuration of an estimation device. [Figure 9] This figure shows an example of the configuration of an estimation device. [Figure 10] This figure shows an example of the configuration of an estimation device. [Figure 11] This figure shows an example of the configuration of an estimation device. [Figure 12] This figure shows an example of the configuration of an estimation device. [Figure 13] This is a conceptual diagram illustrating the configuration of the estimation system. [Figure 14] This is a conceptual diagram that schematically shows a part of the estimation system's configuration. [Figure 15] This is a conceptual diagram that schematically shows a part of the estimation system's configuration. [Figure 16] This is a conceptual diagram that schematically shows a part of the estimation system's configuration. [Figure 17] This is a conceptual diagram that schematically shows a part of the estimation system's configuration. [Figure 18] This is a conceptual diagram schematically showing some of the configurations of other embodiments of the estimation system. [Modes for carrying out the invention]

[0006] Embodiment 1. Figure 1 is a block diagram showing an example of the configuration of the computer device 1 according to this embodiment 1. The computer device 1 functions as an estimation device for estimating bone density. Hereafter, the computer device 1 may be referred to as "estimation device 1".

[0007] As shown in FIG. 1, the estimation device 1 includes, for example, a control unit 10, a storage unit 20, a communication unit 30, a display unit 40, and an input unit 50. The control unit 10, the storage unit 20, the communication unit 30, the display unit 40, and the input unit 50 are electrically connected to each other by, for example, a bus 60.

[0008] The control unit 10 can comprehensively manage the operation of the estimation device 1 by controlling other components of the estimation device 1. The control unit 10 can also be referred to as a control device or a control circuit. The control unit 10 includes at least one processor to provide control and processing capabilities for executing various functions, as will be described in more detail below.

[0009] According to various embodiments, at least one processor may be implemented as a single integrated circuit (IC), or as a plurality of communicatively connected integrated circuits (ICs) and / or discrete circuits. At least one processor can be executed according to various known technologies.

[0010] In one embodiment, the processor includes, for example, one or more circuits or units configured to execute one or more data calculation procedures or processes by executing instructions stored in a related memory. In other embodiments, the processor may be firmware (e.g., discrete logic components) configured to execute one or more data calculation procedures or processes.

[0011] According to various embodiments, the processor may include one or more processors, controllers, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), digital signal processing devices, programmable logic devices, field programmable gate arrays, or any combination of these devices or configurations, or any combination of other known devices and configurations, and may execute the functions described below. In this example, the control unit 10 includes, for example, a CPU (Central Processing Unit).

[0012] The storage unit 20 includes a non-temporary recording medium readable by the CPU of the control unit 10, such as a ROM (Read Only Memory) and a RAM (Random Access Memory). In the storage unit 20, a control program 100 for controlling the estimation device 1 is stored. Various functions of the control unit 10 are realized by the CPU of the control unit 10 executing the control program 100 in the storage unit 20. It can also be said that the control program 100 is a bone density estimation program for causing the computer device 1 to function as an estimation device. In this example, by the control unit 10 executing the control program 100 in the storage unit 20, an approximator 280 capable of outputting an estimated value 300 of bone density is formed in the control unit 10 as shown in FIG. 2. The approximator 280 includes, for example, a neural network 200. It can also be said that the control program 100 is a program for causing the computer device 1 to function as the neural network 200. Hereinafter, the estimated value of bone density may be referred to as the "bone density estimated value". The configuration example of the neural network 200 will be described in detail later.

[0013] In addition to the control program 100, the storage unit 20 stores learned parameters 110 related to the neural network 200, estimation data 120 (hereinafter also referred to as "input information"), learning data 130, and teacher data 140. The learning data 130 and the teacher data 140 are data used when training the neural network 200. The learned parameters 110 and the estimation data 120 are data used when the learned neural network 200 estimates bone density.

[0014] Training data 130 is the data that is input to the input layer 210 of the neural network 200 during training. Training data 130 is also called training data. Supervisor data 140 is data that shows the correct bone density values. Supervisor data 140 is compared with the output data output from the output layer 230 of the neural network 200 during training. Training data 130 and supervisor data 140 are sometimes collectively called supervised training data.

[0015] The estimation data 120 is the data input to the input layer 210 of the trained neural network 200 when it estimates bone density. The trained parameters 110 are the parameters learned within the neural network 200. The trained parameters 110 can be said to be the parameters that were tuned during the training of the neural network 200. The trained parameters 110 include weighting coefficients that represent the weight of the connections between artificial neurons. As shown in Figure 2, the trained neural network 200 performs calculations based on the trained parameters 110 on the estimation data 120 input to the input layer 210 and outputs the bone density estimate 300 from the output layer 230.

[0016] The data input to the input layer 210 may be input to the input layer 210 via the input unit 50, or it may be input directly to the input layer 210. When the data is input directly to the input layer 210, the input layer 210 may be part of or all of the input unit 50. Hereafter, the bone density estimate 300 may be referred to as the estimation result 300.

[0017] The communication unit 30 is connected to a communication network, including the Internet, via wired or wireless connection. The communication unit 30 can communicate with other devices such as cloud servers and web servers through the communication network. The communication unit 30 can input information received from the communication network to the control unit 10. The communication unit 30 can also output information received from the control unit 10 to the communication network.

[0018] The display unit 40 is, for example, a liquid crystal display or an organic EL display. The display unit 40 is controlled by the control unit 10 and is capable of displaying various types of information such as characters, symbols, and graphics.

[0019] The input unit 50 is capable of receiving input from the user to the estimation device 1. The input unit 50 includes, for example, a keyboard and a mouse. The input unit 50 may also include a touch panel capable of detecting user operations on the display surface of the display unit 40.

[0020] The configuration of the estimation device 1 is not limited to the above example. For example, the control unit 10 may have multiple CPUs. The control unit 10 may also have at least one DSP. Furthermore, all or some of the functions of the control unit 10 may be implemented by hardware circuits that do not require software to implement those functions. The storage unit 20 may also have a computer-readable non-temporary recording medium other than ROM and RAM. The storage unit 20 may, for example, have a small hard disk drive and an SSD (Solid State Drive). The storage unit 20 may also have a memory such as a USB (Universal Serial Bus) memory that is detachable from the estimation device 1. Hereafter, a memory that is detachable from the estimation device 1 may be referred to as "detachable memory".

[0021] <Example of a neural network configuration> Figure 3 shows an example of the configuration of neural network 200. In this example, neural network 200 is, for example, a convolutional neural network (CNN). As shown in Figure 3, neural network 200 comprises, for example, an input layer 210, a hidden layer 220, and an output layer 230. The hidden layer 220 is also called an intermediate layer. The hidden layer 220 comprises, for example, multiple convolutional layers 240, multiple pooling layers 250, and a fully connected layer 260. In neural network 200, the fully connected layer 260 is located before the output layer 230. In neural network 200, convolutional layers 240 and pooling layers 250 are alternately arranged between the input layer 210 and the fully connected layer 260.

[0022] Note that the configuration of the neural network 200 is not limited to the example in Figure 3. For example, the neural network 200 may include one convolutional layer 240 and one pooling layer 250 between the input layer 210 and the fully connected layer 260. Also, the neural network 200 may be a neural network other than a convolutional neural network.

[0023] <Examples of estimation data, training data, and training data> Estimation data 120 includes image data of a simple X-ray image showing the bone to be estimated for bone density. The subject of bone density estimation is, for example, a human. Therefore, it can be said that estimation data 120 includes image data of a simple X-ray image showing human bone. Training data 130 includes image data of multiple simple X-ray images showing human bone. A simple X-ray image is a two-dimensional image and is also called a general X-ray image or radiograph. Note that the subject of bone density estimation may not be a human. For example, the subject of bone density estimation may be an animal such as a dog, cat, or horse. In addition, the bones to be estimated are mainly cortical bone and cancellous bone of biological origin, but the bones to be estimated may also include artificial bone mainly composed of calcium phosphate, or regenerated bone artificially produced by regenerative medicine, etc.

[0024] Hereafter, the image data included in estimation data 120 may be referred to as "estimation image data." The simple X-ray image shown by the image data included in estimation data 120 may also be referred to as "estimation simple X-ray image." Similarly, the image data included in training data 130 may be referred to as "training image data." The simple X-ray image shown by the image data included in training data 130 may also be referred to as "training simple X-ray image." Training data 130 contains multiple training X-ray image data, each representing a different training simple X-ray image.

[0025] Examples of areas used for imaging simple X-ray images for estimation include the head, neck, chest, waist, hip joint, knee joint, ankle joint, foot, toes, shoulder joint, elbow joint, wrist joint, hand, fingers, or temporomandibular joint. In other words, the estimation data 120 includes image data of a simple X-ray image obtained when X-rays are irradiated onto the head, image data of a simple X-ray image obtained when X-rays are irradiated onto the neck, image data of a simple X-ray image obtained when X-rays are irradiated onto the chest, image data of a simple X-ray image obtained when X-rays are irradiated onto the waist, image data of a simple X-ray image obtained when X-rays are irradiated onto the hip joint, image data of a simple X-ray image obtained when X-rays are irradiated onto the knee joint, image data of a simple X-ray image obtained when X-rays are irradiated onto the ankle joint, and X-rays obtained when X-rays are irradiated onto the foot. Image data of plain X-ray images obtained by irradiation, image data of plain X-ray images obtained by irradiating the toes with X-rays, image data of plain X-ray images obtained by irradiating the shoulder joint with X-rays, image data of plain X-ray images obtained by irradiating the elbow joint with X-rays, image data of plain X-ray images obtained by irradiating the wrist joint with X-rays, image data of plain X-ray images obtained by irradiating the hand with X-rays, image data of plain X-ray images obtained by irradiating the fingers with X-rays, or image data of plain X-ray images obtained by irradiating the temporomandibular joint with X-rays are used. Plain X-ray images obtained by irradiating the chest with X-rays include plain X-ray images showing the lungs and plain X-ray images showing the thoracic vertebrae. However, the types of areas to be photographed for estimation are not limited to these. Furthermore, the estimation plain X-ray image may be a frontal image showing the target area from the front, or a lateral image showing the target area from the side.

[0026] The multiple training image data contained in the training data 130 each represent multiple training plain X-ray images, and the imaging sites of these images include, for example, at least one of the following: head, neck, chest, waist, hip joint, knee joint, ankle joint, foot, toes, shoulder joint, elbow joint, wrist joint, hand, fingers, and temporomandibular joint. In other words, the training data 130 includes image data of a plain X-ray image obtained when X-rays are irradiated onto the head, image data of a plain X-ray image obtained when X-rays are irradiated onto the neck, image data of a plain X-ray image obtained when X-rays are irradiated onto the chest, image data of a plain X-ray image obtained when X-rays are irradiated onto the waist, image data of a plain X-ray image obtained when X-rays are irradiated onto the hip joint, image data of a plain X-ray image obtained when X-rays are irradiated onto the knee joint, image data of a plain X-ray image obtained when X-rays are irradiated onto the ankle joint, and plain X-ray image obtained when X-rays are irradiated onto the foot. The learning data 130 includes at least one of 15 types of image data, including X-ray image data, plain X-ray image data obtained by irradiating the toes with X-rays, plain X-ray image data obtained by irradiating the shoulder joint with X-rays, plain X-ray image data obtained by irradiating the elbow joint with X-rays, plain X-ray image data obtained by irradiating the wrist joint with X-rays, plain X-ray image data obtained by irradiating the hand with X-rays, plain X-ray image data obtained by irradiating the fingers with X-rays, and plain X-ray image data obtained by irradiating the temporomandibular joint with X-rays. The learning data 130 may include some of the 15 types of image data, or it may include all of the types of image data. However, this does not apply to the types of areas to which the learning plain X-ray images are taken. In addition, multiple learning plain X-ray images may include frontal views or lateral views. Furthermore, multiple learning plain X-ray images may include both frontal and lateral views of the same area.

[0027] The training data 140 includes, for each of the multiple training image data included in the training data 130, a measurement of bone density of a person with bones visible in the training plain X-ray image shown by the training image data. The multiple bone density measurements included in the training data 140 include, for example, at least one of the following: bone density measured by irradiating the lumbar spine with X-rays, bone density measured by irradiating the proximal femur with X-rays, bone density measured by irradiating the radius with X-rays, bone density measured by irradiating the metacarpal bones with X-rays, bone density measured by applying ultrasound to the arm, and bone density measured by applying ultrasound to the heel. Hereafter, the bone density measurements included in the training data 140 may be referred to as "reference bone density."

[0028] Here, the DEXA (dual-energy X-ray absorptiometry) method is known as a method for measuring bone density. In a DEXA device that measures bone density using the DEXA method, when measuring bone density of the lumbar spine, X-rays (specifically two types of X-rays) are irradiated onto the lumbar spine from the front. Also, in a DEXA device, when measuring bone density of the proximal femur, X-rays are irradiated onto the proximal femur from the front.

[0029] The training data 140 may include bone density of the lumbar spine measured by a DEXA device, or bone density of the proximal femur measured by a DEXA device. Furthermore, the training data 140 may include bone density measured by irradiating the target area with X-rays from the side. For example, the training data 140 may include bone density measured by irradiating the lumbar spine with X-rays from the side.

[0030] Another known method for measuring bone density is ultrasound. In devices that measure bone density using ultrasound, for example, ultrasound may be applied to the arm to measure the bone density of that arm, or ultrasound may be applied to the heel to measure the bone density of that heel. The training data 140 may include bone density measured by ultrasound.

[0031] Multiple training image data contained in the training data 130 show multiple training plain X-ray images, each depicting the bones of different individuals. As shown in Figure 4, the memory unit 20 associates each of the multiple training image data contained in the training data 130 with the reference bone density of the person whose bones are visible in the training plain X-ray image shown by that training image data. It can also be said that each of the multiple training plain X-ray images used for training the neural network 200 is associated with the reference bone density of the person whose bones are visible in that training plain X-ray image. The reference bone density associated with the training image data is the bone density measured for the same person whose bones are visible in the training plain X-ray image, at approximately the same time that the training plain X-ray image was taken.

[0032] The area shown in the training image data (in other words, the area captured in the training image data) may or may not include the area (i.e., bone) for which the reference bone density corresponding to the training image data was measured. In other words, the area shown in the training image data may or may not include the area for which the reference bone density corresponding to the training image data was measured. An example of the former would be a training image data showing a training image of the lumbar region being associated with the reference bone density of the lumbar spine. Another example would be a training image data showing the hip joint being associated with the reference bone density of the proximal femur. On the other hand, an example of the latter would be a training image data showing the chest being associated with the reference bone density of the lumbar spine. Another example would be a training image data showing the knee joint being associated with the reference bone density of the heel.

[0033] Furthermore, the orientation of the area shown in the simple X-ray image of the training image data and the orientation of the X-ray irradiation to the target area in the measurement of reference bone density corresponding to the training image data may be the same or different. In other words, the orientation of the area shown in the simple X-ray image of the training image data and the orientation of the X-ray irradiation to the target area in the measurement of reference bone density corresponding to the training image data may be the same or different. An example of the former is when training image data showing a simple X-ray image of the chest from the front (hereinafter sometimes referred to as a "frontal chest simple X-ray image") is associated with reference bone density measured by irradiating the lumbar spine with X-rays from the front. Another example is when training image data showing a simple X-ray image of the lumbar region from the front (hereinafter sometimes referred to as a "frontal lumbar simple X-ray image") is associated with reference bone density measured by irradiating the proximal femur with X-rays from the front. On the other hand, an example of the latter is when training image data showing a simple lateral X-ray image of the lumbar region (hereinafter sometimes referred to as a "lateral lumbar simple X-ray image") is associated with reference bone density measured by irradiating the lumbar spine with X-rays from the front. Another example is when training image data showing a simple lateral X-ray image of the knee joint (hereinafter sometimes referred to as a "lateral knee simple X-ray image") is associated with reference bone density measured by irradiating the proximal femur with X-rays from the front.

[0034] Furthermore, the multiple training plain X-ray images shown by each of the multiple training image data included in the training data 130 may include plain X-ray images showing the same type of body part as the estimation plain X-ray image, or they may include plain X-ray images showing a different type of body part than the estimation plain X-ray image. An example of the former would be if the estimation plain X-ray image is a frontal chest plain X-ray image, and multiple training plain X-ray images include a frontal chest plain X-ray image. Another example would be if the estimation plain X-ray image is a plain X-ray image showing the knee joint from the front (hereinafter sometimes referred to as a "frontal knee plain X-ray image"), and multiple training plain X-ray images include a lateral knee plain X-ray image. On the other hand, an example of the latter would be if the estimation plain X-ray image is a frontal lumbar plain X-ray image, and multiple training plain X-ray images include a frontal chest plain X-ray image. Another example would be if the estimation plain X-ray image is a lateral lumbar plain X-ray image, and multiple training plain X-ray images include a frontal knee plain X-ray image.

[0035] Furthermore, multiple learning plain X-ray images may include plain X-ray images showing the same orientation as the estimation plain X-ray image, or they may include plain X-ray images showing a different orientation from the estimation plain X-ray image. An example of the former would be if the estimation plain X-ray image is a frontal plain X-ray image of the lumbar spine, and multiple learning plain X-ray images include a frontal plain X-ray image of the lumbar region. Another example would be if the estimation plain X-ray image is a frontal plain X-ray image of the knee, and multiple learning plain X-ray images include a frontal plain X-ray image of the chest. On the other hand, an example of the latter would be if the estimation plain X-ray image is a lateral plain X-ray image of the knee, and multiple learning plain X-ray images include a frontal plain X-ray image of the knee. Another example would be if the estimation plain X-ray image is a lateral plain X-ray image of the lumbar region, and multiple learning plain X-ray images include a frontal plain X-ray image of the chest.

[0036] Furthermore, the training data 140 may include reference bone density measured from areas (bones) included in the area visible in the estimation plain X-ray image, or it may include reference bone density measured from areas (bones) not included in the area visible in the estimation plain X-ray image. An example of the former would be if the estimation plain X-ray image is a frontal plain X-ray image of the lumbar region, in which case the training data 140 may include reference bone density of the lumbar spine. On the other hand, an example of the latter would be if the estimation plain X-ray image is a frontal plain X-ray image of the chest, in which case the training data 140 may include reference bone density of the metacarpal bones.

[0037] Furthermore, the training data 140 may include reference bone density measured by irradiating the target area with X-rays from the same direction as the area shown in the estimation plain X-ray image, or it may include reference bone density measured by irradiating the target area with X-rays from a different direction than the area shown in the estimation plain X-ray image. An example of the former would be if the estimation plain X-ray image is a frontal plain X-ray image of the lumbar region, in which case the training data 140 would include reference bone density measured by irradiating the lumbar spine with X-rays from the front. On the other hand, an example of the latter would be if the estimation plain X-ray image is a lateral plain X-ray image of the lumbar region, in which case the training data 140 would include reference bone density measured by irradiating the proximal femur with X-rays from the front.

[0038] In this example, grayscale image data representing a simple X-ray image obtained from a simple X-ray radiographer (in other words, a general X-ray radiographer or radiographer) is reduced in size and its number of gradations is lowered, and this is used as training image data and estimation image data. For example, consider the case where the number of multiple pixel data constituting the image data obtained from a simple X-ray radiographer is greater than (1024 × 640) and the number of bits in that pixel data is 16 bits. In this case, the number of multiple pixel data constituting the image data obtained from a simple X-ray radiographer is reduced to, for example, (256 × 256), (1024 × 512), or (1024 × 640), and the number of bits in that pixel data is reduced to 8 bits, and this is used as training image data and estimation image data. In this case, the training simple X-ray image and the estimation simple X-ray image each consist of (256 × 256), (1024 × 512), or (1024 × 640) pixels, and the value of the pixel is represented by 8 bits.

[0039] The training image data and estimation image data may be generated by the control unit 10 of the estimation device 1 from image data obtained by a simple X-ray imaging device, or by a device other than the estimation device 1 from image data obtained by a simple X-ray imaging device. In the former case, the image data obtained by the simple X-ray imaging device may be received by the communication unit 30 via a communication network, or stored in a removable memory included in the storage unit 20. In the latter case, the communication unit 30 may receive the training image data and estimation image data from another device via a communication network, and the control unit 10 may store the training image data and estimation image data received by the communication unit 30 in the storage unit 20. Alternatively, the training image data and estimation image data generated by another device may be stored in a removable memory included in the storage unit 20. Furthermore, the teacher data 140 may be received by the communication unit 30 via a communication network, and the control unit 10 may store the teacher data 140 received by the communication unit 30 in the storage unit 20. Alternatively, the teacher data 140 may be stored in a removable memory included in the storage unit 20. However, the number of pixels and bits in the training image data and estimation image data are not limited to those stated above.

[0040] <Example of neural network training> Figure 5 is a diagram illustrating an example of neural network 200 training. When training the neural network 200, the control unit 10 inputs training data 130 to the input layer 210 of the neural network 200, as shown in Figure 5. The control unit 10 then adjusts the variable parameters 110a within the neural network 200 so that the error between the output data 400 output from the output layer 230 of the neural network 200 and the training data 140 is reduced. More specifically, the control unit 10 inputs each training image data in the memory unit 20 to the input layer 210. When inputting training image data to the input layer 210, the control unit 10 inputs the multiple pixel data constituting the training image data to the multiple artificial neurons constituting the input layer 210. The control unit 10 then adjusts the parameters 110a so that the error between the output data 400 output from the output layer 230 and the reference bone density corresponding to the training image data is reduced when training image data is input to the input layer 210. For example, backpropagation can be used to adjust parameter 110a. The adjusted parameter 110a becomes the learned parameter 110 and is stored in the memory unit 20. Parameter 110a includes, for example, the parameters used in the hidden layer 220. Specifically, parameter 110a includes the filter coefficients used in the convolutional layer 240 and the weighting coefficients used in the fully connected layer 260. However, the method for adjusting parameter 110a, or in other words, the method for learning parameter 110a, is not limited to this.

[0041] In this way, the memory unit 20 stores training data 130, which includes image data of multiple training simple X-ray images, and trained parameters 110, which are obtained by training the relationship between the training data 140 (measured bone density) and the training data 140 using a neural network 200.

[0042] In the example above, the estimation device 1 is training the neural network 200, but another device may be training the neural network 200. In this case, the memory unit 20 of the estimation device 1 will store the trained parameters 110 generated by the other device. Furthermore, it will not be necessary for the memory unit 20 to store the training data 130 and the training data 140. The communication unit 30 may receive the trained parameters 110 generated by the other device via a communication network, and the control unit 10 may store the trained parameters 110 received by the communication unit 30 in the memory unit 20. Alternatively, the trained parameters 110 generated by the other device may be stored in a removable memory included in the memory unit 20.

[0043] The neural network 200 trained in the manner described above includes trained parameters 110a, which are learned by inputting image data of multiple training simple X-ray images as training data 130 into the input layer 210, and by using reference bone density as training data 140. As shown in Figure 2 above, the neural network 200 performs calculations based on the trained parameters 110a on the estimation data 120 input to the input layer 210, and outputs an estimated bone density value 300 from the output layer 230. When estimation image data as estimation data 120 is input to the input layer 210, multiple pixel data constituting the estimation image data are input to multiple artificial neurons constituting the input layer 210. The convolutional layer 240 performs calculations using the filter coefficients included in the trained parameters 110a, and the fully connected layer 260 performs calculations using the weighting coefficients included in the trained parameters 110a.

[0044] For example, when estimation image data showing a plain frontal chest X-ray image is input to the input layer 210, the estimated bone density 300 of a person with bones in the chest visible in the plain frontal chest X-ray image shown by the estimation image data is output from the output layer 230. Similarly, when estimation image data showing a plain frontal lumbar X-ray image is input to the input layer 210, the estimated bone density 300 of a person with lumbar vertebrae included in the lumbar region visible in the plain frontal lumbar X-ray image shown by the estimation image data is output from the output layer 230. Furthermore, when estimation image data showing a plain lateral lumbar X-ray image is input to the input layer 210, the estimated bone density 300 of a person with lumbar vertebrae included in the lumbar region visible in the plain lateral lumbar X-ray image shown by the estimation image data is output from the output layer 230. Furthermore, when estimation image data showing a plain frontal X-ray image of the knee is input to the input layer 210, the estimated bone density 300 of a person with bones in the knee joint as seen in the plain frontal X-ray image of the knee shown by the estimation image data is output from the output layer 230. Similarly, when estimation image data showing a plain lateral X-ray image of the knee is input to the input layer 210, the estimated bone density 300 of a person with bones in the knee joint as seen in the plain lateral X-ray image of the knee shown by the estimation image data is output from the output layer 230.

[0045] The estimated value 300 output from output layer 230 represents the bone mineral density (g / cm³) per unit area. 2 ), bone mineral density per unit volume (g / cm³) 3 ), may be expressed by at least one of the following: YAM, T-score, and Z-score. YAM is an abbreviation for "Young Adult Mean" and is sometimes called the average percentage of young adults. For example, from output layer 230, bone mineral density per unit area (g / cm³) 2 The output may be an estimated value of 300 represented by ) and an estimated value of 300 represented by YAM, or it may be an estimated value of 300 represented by YAM, an estimated value of 300 represented by the T-score, and an estimated value of 300 represented by the Z-score.

[0046] The memory unit 20 may store multiple estimation data 120. In this case, the multiple estimation plain X-ray images indicated by each of the multiple estimation data 120 in the memory unit 20 may include multiple plain X-ray images showing the same type of body part, or multiple plain X-ray images showing different types of body parts. Furthermore, the multiple estimation plain X-ray images may include multiple plain X-ray images showing the same body part from the same direction, or multiple plain X-ray images showing the body part from different directions. In other words, the multiple estimation plain X-ray images may include multiple plain X-ray images showing the same orientation of the body part, or multiple plain X-ray images showing different orientations of the body part. The control unit 10 inputs each of the multiple estimation data 120 in the memory unit 20 to the input layer 210 of the neural network 200, and the output layer 230 of the neural network 200 outputs a bone density estimate 300 corresponding to each estimation data 120.

[0047] As described above, in this example, image data from simple X-ray images is used to train the neural network 200 and to estimate bone density using the neural network 200. Image data from simple X-ray images, in other words, image data from radiographs, is readily available as it is used in various examinations and other procedures in many hospitals. Therefore, bone density can be easily estimated without using expensive equipment such as a DEXA scanner.

[0048] Furthermore, by using image data from simple X-ray images taken for medical examinations, etc., as estimation image data, it is possible to easily estimate bone density during those examinations, etc. Therefore, by using estimation device 1, services for hospital users can be improved.

[0049] Furthermore, in frontal plain X-ray images of the chest and other areas, bones may be difficult to visualize due to the influence of organs. On the other hand, frontal plain X-ray images are likely to be taken in many hospitals. In this example, even when frontal plain X-ray images, which may make it difficult to visualize bones, are used as estimation plain X-ray images or learning plain X-ray images, bone density can still be estimated. Therefore, bone density can be easily estimated using readily available frontal plain X-ray image data. In addition, frontal plain X-ray images of the chest are often taken in health checkups and are particularly easy to obtain. By using frontal plain X-ray images of the chest as estimation plain X-ray images or learning plain X-ray images, bone density can be estimated even more easily.

[0050] Furthermore, in this example, even if multiple training plain X-ray images include plain X-ray images showing different types of body parts than the estimation plain X-ray image, bone density can still be estimated from the image data of the estimation plain X-ray image. Therefore, the convenience of the estimation device 1 (in other words, the computer device 1) can be improved.

[0051] Furthermore, in this example, even if multiple training plain X-ray images include plain X-ray images showing areas in a different orientation than the estimation plain X-ray image, bone density can still be estimated from the image data of the estimation plain X-ray image. Therefore, the convenience of the estimation device 1 can be improved.

[0052] Furthermore, in this example, even if the area captured in the training plain X-ray image does not include the area (bone) for which the reference bone density corresponding to that training plain X-ray image was measured, the neural network 200 can estimate the bone density based on the trained parameters 110. Therefore, the convenience of the estimation device 1 can be improved.

[0053] Furthermore, in this example, even if the orientation of the area captured in the training plain X-ray image and the orientation of the X-ray irradiation of the target area in the measurement of the reference bone density corresponding to the training plain X-ray image are different, the neural network 200 can estimate bone density based on the trained parameters 110. Therefore, the convenience of the estimation device 1 can be improved.

[0054] Furthermore, in this example, even if the training data 140 includes reference bone density measured from areas not included in the area visible in the estimation plain X-ray image, bone density can still be estimated from the image data of the estimation plain X-ray image. Therefore, the convenience of the estimation device 1 can be improved.

[0055] The bone density estimate 300 obtained by the estimation device 1 may be displayed on the display unit 40. Furthermore, the bone density estimate 300 obtained by the estimation device 1 may be used in other devices.

[0056] Figure 6 shows an example of a bone density estimation system 600 comprising an estimation device 1 and a processing device 500 that performs processing using the bone density estimate 300 obtained by the estimation device 1. In the example of Figure 6, the estimation device 1 and the processing device 500 can communicate with each other through a communication network 700. The communication network 700 includes, for example, at least one of a wireless network and a wired network. The communication network 700 includes, for example, a wireless LAN (Local Area Network) and the Internet.

[0057] In the estimation device 1, the communication unit 30 is connected to the communication network 700. The control unit 10 causes the communication unit 30 to transmit the bone density estimate 300 to the processing unit 500. The processing unit 500 performs processing using the bone density estimate 300 received from the estimation device 1 via the communication network 700. For example, the processing unit 500 is a display device such as a liquid crystal display device and displays the bone density estimate 300. In this case, the processing unit 500 may display the bone density estimate 300 in a table or in a graph. Also, if multiple estimation devices 1 are connected to the communication network 700, the processing unit 500 may display the bone density estimate 300 obtained by the multiple estimation devices 1. The configuration of the processing unit 500 may be the same as the configuration of the estimation device 1 shown in Figure 1, or it may be different from the configuration of the estimation device 1.

[0058] The processing performed by the processing unit 500 using the estimated bone density value 300 is not limited to the example described above. Furthermore, the processing unit 500 may communicate directly with the estimation device 1 wirelessly or via a wired connection without going through the communication network 700.

[0059] <Other examples of estimation and training data> <Another example from the first point> In this example, training data 130 includes information about the health status of a person with bones visible in the training plain X-ray image shown by each training image data. In other words, training data 130 includes information about the health status of the subject (subject) in the training plain X-ray image shown by each training image data. Hereafter, information about the health status of the subject in the training plain X-ray image may be referred to as "training health-related information." Furthermore, information about the health status of the subject in the training plain X-ray image shown by training image data may be referred to as the training health-related information corresponding to that training image data.

[0060] Health-related information for learning includes, for example, at least one of the following: age information, gender information, height information, weight information, drinking habits information, smoking habits information, and fracture history information. This health-related information is compiled into a database for each individual and generated as a CSV (Comma-Separated Value) file or a text file. Age information, height information, and weight information are each represented, for example, as multi-bit numerical data. Gender information is represented by, for example, "male" or "female" as 1-bit data, drinking habits information by "drinks" or "does not drink" as 1-bit data, smoking habits information by "smoks" or "does not smoke" as 1-bit data, and fracture history information by "fractured" or "no fracture" as 1-bit data. Furthermore, health-related information for learning may also include the subject's body fat percentage or subcutaneous fat percentage.

[0061] If the training data 130 includes training image data and corresponding training health-related information, the reference bone density (see Figure 4) corresponding to the training image data is also associated with the training health-related information corresponding to the training image data. In other words, the training image data showing a training plain X-ray image of a person's bones and the information regarding that person's health status (training health-related information) are associated with the measured bone density of that person (reference bone density). Then, during the training of the neural network 200, the training image data and the corresponding training health-related information are input to the input layer 210 simultaneously. Specifically, the training image data is input to some of the multiple artificial neurons that make up the input layer 210, and the training health-related information is input to the other parts of these multiple artificial neurons. Then, the output data 400 output from the output layer 230 when the training image data and the corresponding training health-related information are input to the input layer 210 is compared with the reference bone density corresponding to the training image data and the training health-related information.

[0062] In this example, the estimation data 120 includes estimation image data and information regarding the health status of a person with bones shown in the estimation plain X-ray image represented by the estimation image data. In other words, the estimation data 120 includes estimation image data and information regarding the health status of the subject of the estimation plain X-ray image represented by the estimation image data. Hereafter, the information regarding the health status of the subject of the estimation plain X-ray image may be referred to as "estimated health-related information" (in other embodiments, this may also be referred to as "individual data"). Furthermore, the information regarding the health status of the subject of the estimation plain X-ray image represented by the estimation image data may be referred to as estimated health-related information corresponding to the estimation image data.

[0063] Estimating health-related information, like learning-related health-related information, includes at least one of the following: age information, gender information, height information, weight information, drinking habits information, smoking habits information, and fracture history information. Estimating health-related information includes the same types of information as learning-related health-related information. In addition, estimation health-related information may include the subject's body fat percentage or subcutaneous fat percentage, similar to learning-related health-related information.

[0064] In this example, when bone density is estimated, estimation image data and corresponding estimation health-related information are simultaneously input to the input layer 210. Specifically, estimation image data is input to some of the multiple artificial neurons that make up the input layer 210, and estimation health-related information is input to the other parts of the same multiple artificial neurons. Once estimation image data and estimation health-related information for a particular person are input to the input layer 210, the output layer 230 outputs an estimated value of that person's bone density.

[0065] Thus, by using not only the image data of a simple X-ray image but also information about the health status of the subject of the simple X-ray image, the accuracy of bone density estimation can be improved.

[0066] <Another second example> In this example, training data 130 contains image data of N (N≧2) training plain X-ray images, each showing a different body part of the same person, but with different orientations of the body parts. Hereafter, these N training plain X-ray images may be collectively referred to as the "training plain X-ray image set".

[0067] A training set of simple X-ray images may include, for example, frontal and lateral images of the same person. Another training set of simple X-ray images may include, for example, a frontal chest X-ray and a lateral lumbar X-ray of a person. The image sizes of the frontal and lateral images included in the training set may differ from each other. For example, the width of the lateral image may be smaller than the width of the frontal image. Hereafter, the image data of each training set of simple X-ray images may be collectively referred to as the "training image dataset."

[0068] The training data 130 includes training image datasets for multiple different individuals. Thus, training data 130 contains multiple training image datasets. Each training image dataset is associated with a single reference bone density. In other words, each training image dataset for a particular individual is associated with that individual's bone density measurement (reference bone density).

[0069] In the training of the neural network 200 in this example, each training image dataset is input to the input layer 210. When one training image dataset is input to the input layer 210, the N training image data that make up that training image dataset are simultaneously input to the input layer 210. For example, suppose a training image dataset consists of a first training image data set and a second training image data set. In this case, the first training image data (e.g., image data of a frontal chest X-ray) is input to some of the multiple artificial neurons that make up the input layer 210, and the second training image data (e.g., image data of a lateral lumbar region X-ray) is input to the other parts of the multiple artificial neurons. Then, the output data 400 output from the output layer 230 when a training image dataset is input to the input layer 210 is compared with the reference bone density corresponding to that training image dataset.

[0070] In this example, the estimation data 120 includes image data of N estimation plain X-ray images, each showing a body part belonging to the same person, but with different orientations of the body parts. Hereafter, these N estimation plain X-ray images may be collectively referred to as the "estimation plain X-ray image set."

[0071] A set of simple X-ray images for estimation may include, for example, frontal and lateral images of the same person. For example, a set of simple X-ray images for estimation may include a frontal X-ray image of the lumbar region and a lateral X-ray image of the knee of a person. The image sizes of the frontal and lateral images included in the set of simple X-ray images for estimation may differ from each other. For example, the width of the lateral image may be smaller than the width of the frontal image. Hereafter, the image data of each simple X-ray image in the set of simple X-ray images for estimation may be collectively referred to as the "estimated image dataset."

[0072] In this example, when bone density is estimated using estimation data 120, the input layer 210 receives N estimation image data sets that make up the estimation image dataset simultaneously. For example, suppose the estimation image dataset consists of a first estimation image data set and a second estimation image data set. In this case, the first estimation image data is input to some of the multiple artificial neurons that make up the input layer 210, and the second estimation image data is input to the other parts of the same multiple artificial neurons. When an estimation image dataset for a person is input to the input layer 210, the output layer 230 estimates the bone density of that person.

[0073] In this way, by using image data from multiple simple X-ray images that show parts of the same subject but with different orientations of those parts, the accuracy of bone density estimation can be improved.

[0074] The training data 130 may include a training image dataset and training health-related information. In this case, during the training of the neural network 200, the training image dataset and training health-related information for the same person are input to the input layer 210 simultaneously. Similarly, the estimation data 120 may include an estimation image dataset and estimation health-related information. In this case, the estimation image dataset and estimation health-related information are input to the input layer 210 simultaneously.

[0075] In each of the above examples, the same trained parameters 110 are used regardless of the type of bone shown in the X-ray image provided by the estimation image data. However, trained parameters 110 may be used depending on the type of bone shown in the X-ray image provided by the estimation image data. In this case, the neural network 200 has multiple trained parameters 110, each corresponding to multiple types of bone. The neural network 200 estimates bone density using trained parameters 110 that correspond to the type of bone shown in the X-ray image provided by the input estimation image data. For example, if the X-ray image provided by the input estimation image data shows the lumbar vertebrae, the neural network 200 estimates bone density using trained parameters 110 for estimating bone density of the lumbar vertebrae. Also, if the X-ray image provided by the input estimation image data shows the proximal femur, the neural network 200 estimates bone density using trained parameters 110 for estimating bone density of the proximal femur. The neural network 200 uses, for example, a pre-trained parameter 110 instructed by the user through the input unit 50, from among several pre-trained parameters 110. In this case, the user instructs the neural network 200 to use the pre-trained parameter 110 according to the type of bone shown in the X-ray image indicated by the estimation image data input to the neural network 200.

[0076] In training the neural network 200, multiple training image data, each showing multiple X-ray images of the same type of bone, are used to generate trained parameters 110 corresponding to the type of bone.

[0077] As described above, the estimation device 1 and the bone density estimation system 600 have been explained in detail, but the above explanation is illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be applied in combination as long as they do not contradict each other. And it is understood that countless examples not illustrated can be conceivable without falling outside the scope of this disclosure.

[0078] Embodiment 2. Figure 7 shows an example of the configuration of the estimation device 1A according to this embodiment. In the estimation device 1A, the approximator 280 further has a second neural network 900. The second neural network 900 can detect fractures based on the learned parameters 910. Note that the estimation device 1A according to this embodiment has the same configuration as the estimation device 1 according to the first embodiment, and the description of the same configuration will be omitted. Also, for the sake of explanation, the neural network 200 described in the above example will be referred to as the first neural network 200. The second neural network 900 has, for example, the same configuration as the first neural network 200.

[0079] The second neural network 900 can detect fractures based on the same estimation image data as the estimation data 120 input to the first neural network 200. That is, from one estimation image data, the first neural network 200 can estimate bone density, and the second neural network 900 can detect fractures. The detection result 920 of the second neural network 900 should be output from the output layer 230 of the second neural network 900, as in the example above.

[0080] In the training of the second neural network 900, training image data showing bones that are not fractured and training image data showing bones that are fractured are used to learn the parameters. In addition, in the training data, each training image data is associated with information indicating whether or not there is a fracture in the bone shown in that training image data, and information indicating the location of the fracture. The training data may also include information indicating past fracture history and information indicating past fracture locations. As a result, the second neural network 900 can detect whether or not there is a fracture in the bone shown in the estimation image data based on the estimation image data, and output the detection result 920.

[0081] Furthermore, the estimation device 1A according to this embodiment may have a determination unit 930 that determines whether or not the subject has osteoporosis, as shown in Figure 8. The determination unit 930 can determine whether or not the subject has osteoporosis by comparing the estimation result 300 of the first neural network 200 and the detection result 920 of the second neural network 900.

[0082] The determination unit 930 may determine osteoporosis based, for example, on its own criteria or already well-known guidelines. Specifically, the determination unit 930 may determine that the patient has osteoporosis if the detection result 920 indicates a fracture in the vertebral body or proximal femur. Furthermore, if the bone density estimate 300 output by the first neural network 200 indicates YAM, the determination unit 930 may determine that the patient has osteoporosis if the YAM value is less than 80% and the detection result 920 indicates a fracture other than in the vertebral body or proximal femur. In addition, the determination unit 930 may determine that the patient has osteoporosis if the YAM value indicated by the bone density estimate 300 is 70% or less.

[0083] Furthermore, in the estimation device 1A according to this embodiment, the approximator 28 may further include a third neural network 950, as shown in Figure 9. The third neural network 950 can distinguish the bones of a subject from the estimation image data included in the estimation data 120 based on the trained parameters 960.

[0084] The third neural network 950 outputs location information for each pixel data of the input estimation image data, indicating the bone location represented by that pixel data. This allows the bones shown in the X-ray image represented by the estimation image data to be separated. Location information is sometimes called segmentation data.

[0085] For example, if the X-ray image shown by the input estimation image data contains lumbar vertebrae, the third neural network 950 outputs location information for each pixel data of the estimation image data, indicating which part of the lumbar vertebrae (L1 to L5) the pixel data represents. For example, if a pixel data in the estimation image data represents L1 of the lumbar vertebrae, the third neural network 950 outputs location information indicating L1 as the location information corresponding to that pixel data.

[0086] The third neural network 950 uses trained parameters 960 corresponding to the types of bones shown in the X-ray image provided by the estimation image data. The third neural network 950 has multiple trained parameters 960, each corresponding to a different type of bone. The third neural network 950 uses the trained parameters 960 corresponding to the types of bones shown in the X-ray image provided by the input estimation image data to classify the bones shown in the X-ray image provided by the estimation image data. For example, if the X-ray image provided by the input estimation image data shows lumbar vertebrae, the third neural network 950 uses trained parameters 960 corresponding to lumbar vertebrae to classify them into L1 to L5. The third neural network 950 uses, for example, trained parameters 960 instructed by the user through the input unit 50 from among the multiple trained parameters 960. In this case, the user instructs the third neural network 950 on the trained parameters 960 to be used by the third neural network 950, depending on the type of bone shown in the X-ray image indicated by the estimation image data input to the third neural network 950.

[0087] The third neural network 950 may divide the bone shown in the X-ray image indicated by the input estimation image data into a first site where an implant is embedded, a second site where a tumor is present, and a third site where a fracture is present. In this case, the third neural network 950 outputs site information for each pixel data of the estimation image data, indicating which of the first, second, or third site the pixel data represents. If the pixel data indicates a site other than the first, second, or third site, the third neural network 950 outputs site information indicating that the pixel data indicates a site other than the first, second, or third site. When the third neural network 950 divides the bone shown in the X-ray image indicated by the estimation image data into a first, second, and third site, it can also be said that the third neural network 950 detects implants embedded in the bone, fractures in the bone, and tumors in the bone shown in the X-ray image indicated by the estimation image data.

[0088] In the training of the third neural network 950, multiple training image data, each showing multiple X-ray images of the same type of bone, are used to generate trained parameters 960 corresponding to the type of bone. When the third neural network 950 classifies the bones shown in the X-ray images shown by the input estimation image data into a first, second, and third region, the multiple training image data includes training image data showing X-ray images of cases with implants, training image data showing X-ray images of cases with bone tumors, and training image data showing X-ray images of cases with fractures. The training data also includes annotation information for each training image data for classifying the bone shown by that training image data. The annotation information includes region information for each pixel data of the corresponding training image data, indicating the bone region shown by that pixel data.

[0089] The first neural network 200 may also estimate bone density for each region divided by the third neural network 950. In this case, as shown in Figure 10, the first neural network 200 receives estimation image data 121 and region information 965, which corresponds to each pixel data of the estimation image data 121 and is output by the third neural network 950 based on the estimation image data 121. Based on trained parameters 110 corresponding to the type of bone shown in the X-ray image indicated by the estimation image data 121, the first neural network 200 outputs an estimated bone density value 300 for each region divided by the third neural network. For example, if the third neural network 950 divides the cervical vertebrae shown in the X-ray image indicated by the estimation image data 121 into L1 to L5, the first neural network 200 will output the bone density estimates 300 for L1, L2, L3, L4, and L5 individually.

[0090] In the training of the first neural network 200, multiple training image data, each showing multiple X-ray images of the same type of bone, are used to generate trained parameters 110 corresponding to the type of bone. The training data also includes reference bone density for each part of the bone shown in each training image data.

[0091] The first neural network 200 uses, for example, a trained parameter 110 instructed by the user through the input unit 50, from among a plurality of trained parameters 110. In this case, the user instructs the trained parameter 110 to be used by the first neural network 200 according to the type of bone shown in the X-ray image indicated by the estimation image data input to the first neural network 200.

[0092] When the third neural network 950 divides the bone shown in the X-ray image indicated by the estimation image data into a first site where an implant is placed, a second site where a tumor is present, and a third site where a fracture is present, the brightness of the first partial image data showing the first site, the second partial image data showing the second site where the tumor is present, and the third partial image data showing the third site may be adjusted. Figure 11 shows an example of the configuration in this case.

[0093] As shown in Figure 11, the adjustment unit 968 receives estimation image data 121 and region information 965 corresponding to each pixel data of the estimation image data 121, which is output by the third neural network 950 based on the estimation image data 121. Based on the region information 965, the adjustment unit 968 identifies the first partial image data, the second partial image data, and the third partial image data included in the estimation image data 121. The adjustment unit 968 then adjusts the brightness of the identified first partial image data, second partial image data, and third partial image data.

[0094] The adjustment unit 968 stores, for example, the brightness of a first region visible in a typical X-ray image as the first reference brightness. The adjustment unit 968 also stores the brightness of a second region visible in a typical X-ray image as the second reference brightness. The adjustment unit 968 then stores the brightness of a third region visible in a typical X-ray image as the third reference brightness. The adjustment unit 968 adjusts the brightness of the first partial image data by subtracting the first reference brightness from the brightness of the first partial image data. The adjustment unit 976 adjusts the brightness of the second partial image data by subtracting the second reference brightness from the brightness of the second partial image data. The adjustment unit 978 then adjusts the brightness of the third partial image data by subtracting the third reference brightness from the brightness of the third partial image data. The adjustment unit 968 then inputs the brightness-adjusted estimation image data of the first partial image data, the second partial image data, and the third partial image data into the first neural network 200. The first neural network 200 estimates the bone density of the bones shown in the X-ray image represented by the estimation image data, based on the estimation image data after brightness adjustment.

[0095] Here, it is not easy to accurately estimate bone density from the first site where the implant is placed, the second site where the tumor is located, and the third site where the fracture is present. As described above, by adjusting and reducing the brightness of the first partial image data showing the first site, the second partial image data showing the second site, and the third partial image data showing the third site, it is possible to more accurately estimate the bone density of the bone shown in the X-ray image data used for estimation.

[0096] The adjustment unit 968 may also input estimation image data, in which the brightness of the first partial image data, the second partial image data, and the third partial image data has been forcibly set to zero, to the first neural network 200 as estimation image data after brightness adjustment.

[0097] Furthermore, the third neural network 950 may detect only one of the implants, fractures, and tumors. Alternatively, the third neural network 950 may detect only two of the implants, fractures, and tumors. In other words, the third neural network 950 may detect at least one of the implants, fractures, and tumors.

[0098] Furthermore, the estimation device 1A may include the first neural network 200 and the third neural network 950 without including the second neural network 900. Alternatively, the estimation device 1A may include at least one of the second neural network 900 and the third neural network 950 without including the first neural network 200.

[0099] As described above, the estimation device 1A has been explained in detail, but the above explanation is illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be combined and applied insofar as they do not contradict each other. And it is understood that countless examples not illustrated can be conceivable without falling outside the scope of this disclosure.

[0100] Embodiment 3. Figure 12 shows an example of the configuration of the estimation device 1B according to this embodiment. The estimation device 1B has a fracture prediction unit 980. The fracture prediction unit 980 can predict the probability of fracture based on, for example, the estimation result 300 of the neural network 200 of the estimation device 1 according to Embodiment 1. Specifically, for example, a calculation formula 990 showing the relationship between estimation results related to bone density (e.g., bone density) and the probability of fracture can be obtained from past literature. The fracture prediction unit 980 stores the calculation formula 990. The fracture prediction unit 980 can predict the probability of fracture based on the input estimation result 300 and the stored calculation formula 990.

[0101] Furthermore, formula 990 may also be a formula that shows the relationship between the estimated results related to bone density and the probability of fracture after bone screw implantation. As a result, it becomes possible to consider the appropriateness of bone screw implantation and to develop a treatment plan that includes drug administration.

[0102] Furthermore, the estimation device 1B may include a second neural network 900. Also, the estimation device 1B may include a third neural network 950.

[0103] As described above, the estimation device 1B has been explained in detail, but the above explanation is illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be combined and applied insofar as they do not contradict each other. And it is understood that countless examples not illustrated can be conceivable without falling outside the scope of this disclosure.

[0104] Embodiment 4. Figure 13 shows a conceptual diagram of the configuration of the estimation system 801 in this embodiment.

[0105] The estimation system 801 of this disclosure can estimate the future bone mass of a subject from images showing the subject's bones, such as X-ray images. The estimation system 801 of this disclosure comprises a terminal device 802 and an estimation device 803. Bone mass is an index related to bone density and is a concept that includes bone density.

[0106] Terminal device 802 can acquire input information I for input to estimation device 803. Input information I may be, for example, an X-ray image. In this case, terminal device 802 may be any device used by a doctor or other professional to take an X-ray image of a subject. For example, terminal device 802 may be a simple X-ray imaging device (in other words, a general X-ray imaging device or radiographer).

[0107] Note that terminal device 802 is not limited to a simple X-ray imaging device. Terminal device 802 may be, for example, an X-ray fluoroscopy device, CT (Computed Tomography), MRI (Magnetic Resonance Imaging), SPECT (Single Photon Emission Computed Tomography)-CT, or tomosynthesis. In this case, the input information I may be, for example, an X-ray fluoroscopy image, a CT (Computed Tomography) image, an MRI (Magnetic Resonance Imaging) image, a bone scintigraphy image, or a tomosynthesis image.

[0108] The estimation system 801 is used, for example, to diagnose osteoporosis in patients visiting a hospital. The estimation system 801 of this disclosure takes an X-ray of the patient using a terminal device 802 installed in an X-ray room, for example. The image data is then transferred from the terminal device 802 to the estimation device 803, and via the estimation device 803, it is possible to estimate not only the patient's current bone mass or bone density, but also the patient's bone mass or bone density in the future from the time of imaging.

[0109] Furthermore, the terminal device 802 does not have to directly transfer the input information I to the estimation device 803. In this case, for example, the input information I acquired by the terminal device 802 may be stored in a storage medium, and the input information I may be input to the estimation device 803 via the storage medium.

[0110] Figure 14 shows a conceptual diagram of the configuration of the estimation device 803 according to this embodiment.

[0111] The estimation device 803 can estimate the subject's future bone mass or bone density based on the input information I acquired by the terminal device 802. The estimation device 803 can estimate the subject's future bone mass or bone density from the image data acquired by the terminal device 802 and output the estimation result O.

[0112] The estimation device 803 includes an input unit 831, an approximator 832, and an output unit 833. The input unit 831 receives input information I from the terminal device 802. The approximator 832 can estimate future bone mass or bone density based on the input information I. The output unit 833 can output the estimation result O predicted by the approximator 832.

[0113] The estimation device 803 has various electronic components and circuits. As a result, the estimation device 803 can form various components. For example, the estimation device 803 can form various functional parts by integrating multiple semiconductor elements to form at least one integrated circuit (e.g., IC: Integrated Circuit or LSI: Large Scale Integration), or by further integrating multiple integrated circuits to form at least one unit.

[0114] The multiple electronic components can be active elements such as transistors or diodes, or passive elements such as capacitors. Furthermore, multiple electronic components, and integrated circuits formed by integrating them, can be formed by conventionally known methods.

[0115] The input unit 831 receives information to be used by the estimation device 803. For example, input information I, which includes an X-ray image acquired by the terminal device 802, is input to the input unit 831. The input unit 831 also has a communication unit, and input information I acquired by the terminal device 802 is directly input from the terminal device 802. Furthermore, the input unit 831 may be equipped with an input device capable of receiving input information I or other information. The input device could be, for example, a keyboard, touch panel, or mouse.

[0116] The approximator 832 estimates the future bone mass or bone density of a subject based on the information input to the input unit 831. The approximator 832 has AI (Artificial Intelligence). The approximator 832 has a program that functions as AI, and various electronic components and circuits for executing that program. The approximator 832 has a neural network.

[0117] The approximator 832 has been pre-trained on the relationship between input and output. That is, by applying machine learning to the approximator 832 using training data and target data, the approximator 832 can calculate an estimation result O from the input information I. Note that the training data or target data only needs to be data that corresponds to the input information I input to the estimation device 803 and the estimation result O output from the estimation device 803.

[0118] Figure 15 shows a conceptual representation of the configuration of the approximator 832 of this disclosure.

[0119] The approximator 832 has a first neural network 8321 and a second neural network 8322. The first neural network 8321 can be any neural network suitable for handling time-series information. For example, the first neural network 8321 can be a ConvLSTM network, which combines LSTM (Long short-term memory) and CNN (Convolutional Neural Network). The second neural network 8322 can be, for example, a convolutional neural network composed of CNNs.

[0120] The first neural network 8321 has an encoding unit E and a decoding unit D. The encoding unit E can extract feature quantities of the time change and position information of the input information I. The decoding unit D can calculate new feature quantities based on the feature quantities extracted by the encoding unit E, the time change of the input information I, and the initial values.

[0121] Figure 16 shows a conceptual representation of the configuration of the first neural network 8321 of this disclosure.

[0122] The encoding unit E has multiple ConvLSTM (Convolutional Long short-term memory) layers E1. The decoding unit D has multiple ConvLSTM (Convolutional Long short-term memory) layers D1. Each of the encoding unit E and the decoding unit D may have three or more ConvLSTM layers E1 and D1. The number of multiple ConvLSTM layers E1 and the number of multiple ConvLSTM layers D1 may be equal.

[0123] Note that multiple ConvLSTM layers E1 may learn different things. Multiple ConvLSTM layers D1 may learn different things. For example, one ConvLSTM layer may learn detailed information such as changes in each individual pixel, while another ConvLSTM layer may learn broader information such as changes in the overall picture.

[0124] Figure 17 shows the conceptual configuration of the second neural network 8322.

[0125] The second neural network 8322 has a conversion unit C. The conversion unit C can convert the feature quantities calculated by the decoding unit D of the first neural network 8321 into bone mass or bone density. The conversion unit C has a plurality of convolutional layers C1, a plurality of pooling layers C2, and a fully connected layer C3. The fully connected layer C3 is located before the output unit 33. In the conversion unit C, the convolutional layers C1 and pooling layers C2 are alternately arranged between the first neural network 8311 and the fully connected layer C3.

[0126] The training data is input to the encoding unit E of the approximator 832 during the training of the approximator 832. The training data is compared with the output data output from the conversion unit C of the approximator 832 during the training of the approximator 832. The training data is data showing values ​​measured using a conventional bone density measurement device.

[0127] The output unit 833 can display the estimation result O. The output unit 833 is, for example, a liquid crystal display or an organic EL display. The output unit 833 can display various types of information such as characters, symbols, and figures. The output unit 833 can display, for example, numbers or images.

[0128] The estimation device 803 of this disclosure further comprises a control unit 834 and a storage unit 835. The control unit 834 can comprehensively manage the operation of the estimation device 803 by controlling the other components of the estimation device 803.

[0129] The control unit 834 includes, for example, a processor. The processor may include, for example, one or more processors, controllers, microprocessors, microcontrollers, application-specific integrated circuits (ASICs), digital signal processing devices, programmable logic devices, or any combination of these devices or calibrations, or combinations of other base devices or calibrations. The control unit 834 includes, for example, a CPU.

[0130] The memory unit 835 includes, for example, a non-temporary recording medium that can be read by the CPU of the control unit 834, such as RAM (Random Access Memory) or ROM (Read-Only Memory). The memory unit 835 stores a control program for controlling the estimation device 803, such as firmware. The memory unit 835 may also store input information I, learning data to be learned, and training data.

[0131] The processor of the control unit 834 can execute one or more data calculation procedures or processes according to the control program of the storage unit 835. The various functions of the control unit 834 are realized by the CPU of the control unit 834 executing the control program in the storage unit 11.

[0132] The control unit 834 may perform other processing as preprocessing before the calculation process, if necessary.

[0133] <Examples of input information, training data, and training data> The input information (hereinafter also referred to as the first input information I1) contains image data of the bone to be estimated for bone mass or bone density. The image data can be, for example, a simple X-ray image. The subject for bone mass or bone density estimation is, for example, a human. In this case, the first input information I1 can be said to be image data of a simple X-ray image of human bone. A simple X-ray image is a two-dimensional image and is also called a general X-ray image or radiograph.

[0134] The first input information I1 is preferably a simple X-ray image, which is relatively easy to obtain, but is not limited to that. For example, using X-ray fluoroscopy images, CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, bone scintigraphy images, or tomosynthesis images as input information may allow for a more accurate estimation of bone mass or bone density.

[0135] Furthermore, the subjects for bone mass or bone density estimation may not be humans. For example, the subjects for bone mass or bone density estimation may be animals such as dogs, cats, or horses. In addition, the bones targeted are mainly biologically derived cortical bone and cancellous bone, but the bones targeted may also include artificial bone mainly composed of calcium phosphate, or regenerated bone artificially produced by regenerative medicine, etc.

[0136] Examples of areas to be photographed with an X-ray include the neck, chest, lumbar region, proximal femur, knee joint, ankle joint, shoulder joint, elbow joint, wrist joint, finger joints, or temporomandibular joint. Note that X-ray images may also include areas other than bone. For example, a simple chest X-ray may include images of the lungs and the thoracic spine. The X-ray image may be a frontal view showing the area from the front, or a lateral view showing the area from the side.

[0137] The training data or training data should be data that corresponds to the input information I input to the estimation device 3 and the estimation result O output from the estimation device 3.

[0138] The training data contains the same type of information as the first input information I1. For example, if the first input information I1 is a simple X-ray image, then the training data should also contain a simple X-ray image. Furthermore, if the first input information I1 is a simple chest X-ray image, then the training data should also contain a simple chest X-ray image.

[0139] The training data includes training image data of multiple plain X-ray images showing bones. The areas captured in the multiple training image data include, for example, at least one of the following: neck, chest, lumbar region, proximal femur, knee joint, ankle joint, shoulder joint, elbow joint, wrist joint, finger joints, and temporomandibular joint. The training data may include some of the 11 types of image data, or all of them. In addition, the multiple training image data may include frontal or lateral views.

[0140] The training data contains images of bones from multiple different individuals. For each of the training image data, the measured bone mass or bone density of the subject in that image data is associated as training data. The measured bone mass or bone density was measured at approximately the same time as the training image data was taken.

[0141] Furthermore, the training image data may be a series of images of the same person taken at different time points. That is, the training image data may include a first training data set having X-ray images of bones, and a second training data set having X-ray images of the same person as the first training data, but taken after the first training data.

[0142] Furthermore, the training image data may consist of images of the same body part of different individuals, but with varying ages and other characteristics. Alternatively, the training image data may consist of a series of images of the same person and the same body part, but taken at different time points.

[0143] The training data and first input information I1 may be obtained by reducing the number of grayscale image data representing a simple X-ray image taken with a simple X-ray imaging device (in other words, a general X-ray imaging device or radiographer) and lowering the number of gradations. For example, consider the case where the number of pixels in the image data is greater than (1024 × 640) and the number of bits in that pixel data is 16 bits. In this case, the number of pixels is reduced to, for example, (256 × 256), (1024 × 512), or (1024 × 640), and the number of bits in that pixel data is reduced to 8 bits, and this is used as the first input information I1 and training data.

[0144] The training data includes, for each of the multiple training image data sets included in the training data, measurements of bone mass or bone density of the bone visible in the training plain X-ray image shown by the training image data. Bone mass or bone density can be measured, for example, by DEXA (dual-energy X-ray absorptiometry) or ultrasound.

[0145] <Example of neural network training> The control unit 834 performs machine learning on the approximator 832 using training data and target data so that the approximator 832 can calculate an estimated result O regarding bone mass or bone density from the input information I. The approximator 832 is optimized by known machine learning using target data. The approximator 832 adjusts its variable parameters so that the difference between the pseudo-estimated result calculated from the training data input to the encoding unit E and output from the conversion unit C and the target data becomes small.

[0146] Specifically, the control unit 834 inputs the training data in the memory unit 835 to the encoding unit E. When inputting training data to the encoding unit E, the control unit 834 inputs multiple pixel data constituting the training image data to multiple artificial neurons constituting the encoding unit E. Then, the control unit 834 adjusts the parameters so that the error between the estimated result O output from the conversion unit C when the training image data is input to the encoding unit E and the measured bone mass or bone density corresponding to the training image data is reduced. The adjusted parameters become trained parameters and are stored in the memory unit 835.

[0147] For example, backpropagation is used as a method for tuning the parameters. The parameters include those used in the encoding unit E, decoding unit D, and transformation unit C. Specifically, the parameters include the weighting coefficients used in the ConvLSTM layers of encoding unit E and decoding unit D, and in the convolutional and fully connected layers of transformation unit C.

[0148] As a result, the approximator 832 performs calculations based on the learned parameters on the input information I input to the encoding unit E, and outputs the estimation result O from the transformation unit C. When X-ray image data is input to the encoding unit E as input information I, multiple pixel data constituting this image data are input to multiple artificial neurons constituting the input unit 831. The ConvLSTM layer, convolutional layer, and fully connected layer then perform calculations using the weighting coefficients included in the learned parameters, and can output the estimation result O.

[0149] As described above, the estimation system 801 uses image data from a simple X-ray image to train the approximator 832 and to estimate bone mass or bone density using the approximator 832. Therefore, by inputting input information I into the estimation system 801, future bone mass or bone density can be output as the estimated result O.

[0150] The estimation result O of the estimation system 801 can be the estimation result for a date in the future from the date the input information I was acquired. For example, the estimation system 801 can estimate bone mass or bone density from 3 months to 50 years after the time of imaging, and more preferably from 6 months to 10 years after.

[0151] The estimated result O should be output as a value. For example, it may be represented by at least one of the following: YAM (Young Adult Mean), T-score, and Z-score. For example, the output unit 833 may output the estimated value expressed in YAM, or it may output the estimated value expressed in YAM, the estimated value expressed in T-score, and the estimated value expressed in Z-score.

[0152] Furthermore, the estimated result O may be output as an image. If the estimated result O is an image, for example, an X-ray image-like image should be displayed. An X-ray image-like image is an image that mimics an X-ray image. Also, if a series of data taken from the same person and the same body part, but at different time points, are trained using ConvLSTM, it is possible to predict the temporal changes in the image. This makes it possible to generate a future image from an X-ray image of a different patient at a single point in time.

[0153] The training data and input information I may include images of internal organs, muscles, fat, or blood vessels in addition to bones. Even in such cases, highly accurate estimation can be performed.

[0154] The first input information I1 may contain the subject's individual data (first individual data). First individual data may include, for example, age information, sex information, height information, weight information, or fracture history. As a result, highly accurate estimations can be performed.

[0155] The first input information I1 may contain second-person data of the subject. This second-person data may include, for example, information on blood pressure, lipids, cholesterol, triglycerides, and blood glucose levels. As a result, highly accurate estimations can be performed.

[0156] The first input information I1 may contain information about the subject's lifestyle habits. This lifestyle information could include details such as drinking habits, smoking habits, exercise habits, or eating habits. As a result, highly accurate estimations can be made.

[0157] The first input information I1 may contain the subject's bone metabolism information. Bone metabolism information may be, for example, bone resorption capacity or bone formation capacity. These can be measured by, for example, at least one of the following bone resorption markers: type I collagen cross-linked N-telopeptide (NTX), type I collagen cross-linked C-telopeptide (CTX), tartrate-resistant acid phosphatase (TRACP-5b), deoxypyridinoline (DPD); bone formation markers: bone-type alkaline phosphatase (BAP), type I collagen cross-linked N-propeptide (P1NP); or bone-related matrix marker: low-carboxylated osteocalcin (ucOC). The bone resorption marker may be measured using serum or urine as the sample.

[0158] In the estimation system 801, second input information I2 concerning the subject's planned future behavior may also be input as input information I. Second input information I2 may be, for example, individual data on planned or improved behavior, or information on planned or improved lifestyle, exercise habits, and dietary habits. Specifically, second input information I2 may be information such as weight data, drinking habits, smoking habits, time spent in the sun, number of steps or walking distance per day, dairy product intake, or intake of foods rich in vitamin D, such as fish and mushrooms, after improvement. As a result, the estimation system 801 can show an estimated result O of improved future bone mass or bone density.

[0159] Furthermore, the second input information I2 may be, for example, information about lifestyle habits that are planned to be worsened. As a result, the estimation system can show a worsened estimated result O of future bone mass or bone density.

[0160] In the estimation system 801, a third input information I3 concerning therapy for the subject may also be input as input information I. The third input information I3 is, for example, information concerning physical therapy or drug therapy. Specifically, the third input information I3 may be at least one of the following: calcium drugs, female hormone drugs, vitamin drugs, bisphosphonate drugs, SERM (Selective Estrogen Receptor Modulator) drugs, calcitonin drugs, thyroid hormone drugs, or denosumab drugs.

[0161] In the estimation system 801, the estimation result O may output a first result O1 based solely on the first input information I1, and a second result O2 based on the first input information I1 and at least one of the second and third input information I2 and I3. As a result, the effects on future planned actions can be compared.

[0162] In the estimation system 801, the estimation result O may output not only future bone mass or bone density, but also current results. As a result, changes in bone mass or bone density over time can be compared.

[0163] Figure 18 shows a conceptual configuration of the approximator 832 in another embodiment of the estimation system 801.

[0164] The estimation device 803 of the estimation system 801 may have a first approximator 832a and a second approximator 832b. That is, in addition to the above approximator 832 (first approximator 832a), it may also have a second approximator 832b. The second approximator 832b may be, for example, a CNN.

[0165] In this case, in the estimation system 801, the first approximator 832a outputs the first image and the first value as the first estimation result O1 to the first output unit 833a. Furthermore, the second approximator 832b outputs the second value as the second estimation result O2 from the first image from the first output unit 833a to the second output unit 833b. As a result, the first value and the second value can be compared as the estimated result O of future bone mass or bone density.

[0166] The estimation system 801 may output a third value based on the first and second values ​​as the estimation result O. As a result, for example, the result of correcting the first value based on the second value (the third value) can be taken as the estimation result O.

[0167] As described above, the estimated system 801 has been explained in detail, but the above explanation is illustrative in all respects, and this disclosure is not limited thereto. Furthermore, the various examples described above can be combined and applied insofar as they do not contradict each other. And it is understood that countless examples not illustrated can be conceivable without falling outside the scope of this disclosure. [Explanation of Symbols]

[0168] 1. Computer device (estimation device) 20 Memory section 100 control programs 110,910,960 trained parameters 120 Estimation data 130 training data 140 training data 200 Neural Networks 210 Input Layers 230 Output Layers 280,832 approximator 500 Processing Units 600 Bone Density Estimation System 801 Estimation System 802 Terminal device 803 Estimation device 831 Input section 833 Output section 834 Control Unit 835 Storage section 900,8322 Second Neural Network 930 Judgment section 950 The Third Neural Network 980 Fracture prediction section 8321 The First Neural Network O Estimation result I Input Information E Encoding Section D Decode Unit C conversion section

Claims

1. A processing unit that outputs classification information relating to the division of parts included in the first skeleton based on first input information including a first simple X-ray image of a first skeleton possessed by a first person and first learned parameters, An estimation unit that estimates the bone density of at least a portion of the first skeleton based on the first input information, the second input information based on the classification information, and the second learned parameter. Equipped with, An information processing system in which the second learned parameters are set based on first training data, which includes a learning simple X-ray image of a second skeleton possessed by a second person, and first training data, which includes the bone density of the second person.

2. The information processing system according to claim 1, The processing unit is an information processing system that outputs classification information dividing the first skeleton into a first site where an implant is embedded, a second site where a tumor is present, and a third site where a fracture is present.

3. The information processing system according to claim 2, The classification information includes segmentation data for the first part, the second part, and the third part, in an information processing system.

4. An information processing system according to claim 2 or claim 3, An information processing system in which the first learned parameters are set based on second training data, which includes a training image containing a first part of a third skeleton belonging to a third person, a training image containing a second part of a fourth skeleton belonging to a fourth person, and a training image containing a third part of a fifth skeleton belonging to a fifth person, and second training data, which includes annotation information for classifying the training images contained in the second training data.

5. An information processing system according to any one of claims 1 to 4, The estimation unit is an information processing system that estimates bone density for each divided area.

6. An information processing system according to any one of claims 1 to 5, An information processing system in which the second learned parameters are parameters corresponding to the type of the first skeleton.

7. An information processing system according to any one of claims 2 to 4, An information processing system in which the second input information includes a first image obtained by adjusting the brightness of the first simple X-ray image based on the classification information.

8. The information processing system according to claim 7, The system further includes an adjustment unit that outputs the second input information, The adjustment unit is an information processing system that outputs a first image based on at least one of the following: a first reference brightness, which is the brightness of the first part as seen in a typical X-ray image; a second reference brightness, which is the brightness of the second part as seen in a typical X-ray image; and a third reference brightness, which is the brightness of the third part as seen in a typical X-ray image, and the classification information.

9. A first step of outputting classification information relating to the division of parts included in the first skeleton based on first input information including a first simple X-ray image of a first skeleton possessed by a first person and first learned parameters, A second step of estimating the bone density of at least a portion of the first skeleton based on the first input information, the second input information based on the classification information, and the second learned parameter. Equipped with, An information processing method in which the second learned parameter is set based on first training data including a simple X-ray image for learning of a second skeleton possessed by a second person, and first training data including the bone density of the second person.

10. A program for causing a computer device to function as an information processing system according to any one of claims 1 to 8.

11. A recording medium for storing the program described in claim 10.

Images