Image generation device, image generation method, and program

Abstract

An image generation device (4) comprises: a storage device (42) for holding first image data obtained by using an X-ray inspection device (1) to perform imaging of a part of an animal that includes a bone inside; and a generation unit (431) that has a trained machine learning model (432), inputs the first image data to the machine learning model (432), and thereby generates second image data indicating the bone and a cartilage region that is estimated from the position of the bone and is continuous with the bone.

Description

Image generation device, image generation method, and program

[0001] The present disclosure relates to an image generating device and the like that generates image data showing a cartilage region.

[0002] Generally, to evaluate the articular cartilage of a patient suffering from osteoarthritis or the like, an examination using an MRI (Magnetic Resonance Imaging) device is performed. For example, an MRI device using an RF coil (radio frequency coil) that can be deformed to fit the patient has been disclosed (see Patent Document 1). This RF coil always achieves a high signal-to-noise ratio regardless of the shape of the coil and the patient, and therefore can achieve high irradiation efficiency with low SAR.

[0003] Patent No. 6005279

[0004] However, even when an examination is performed using the MRI apparatus disclosed in Patent Document 1, it is necessary to suppress the patient's body movement when taking MRI images, which poses a problem that the examination is not easy, particularly for pediatric patients.

[0005] Furthermore, MRI examination devices are generally installed in large hospitals and are often not installed in clinics located in urban areas, which creates the problem that the patient's doctor, who works at a clinic located in urban areas, may miss the fracture line in the cartilage area, delaying treatment.

[0006] The present disclosure has been made to solve such problems, and aims to provide an image generation device, etc. that can generate image data showing the cartilage area through a simpler examination and thereby assist the attending physician in his or her examination.

[0007] An image generating device according to one aspect of the present disclosure includes a storage device that stores first image data obtained by imaging an animal part that has a bone inside using an X-ray inspection device, and a generating unit that has a trained machine learning model and generates second image data that shows the bone and a region of cartilage that is continuous with the bone and that is estimated from the position of the bone by inputting the first image data into the machine learning model.

[0008] An image generation method according to one aspect of the present disclosure is an image generation method executed by an image generation device, the image generation device having a storage device for storing first image data obtained by imaging a part of an animal having a bone inside using an X-ray inspection device, and the image generation method includes a generation step of generating second image data showing the bone and a region of cartilage continuous with the bone estimated from the position of the bone by inputting the first image data into a machine learning model.

[0009] A program according to one aspect of the present disclosure causes a computer to execute the image generation method described above.

[0010] According to the present disclosure, an image generating device and the like are provided that can generate image data showing a cartilage region through a simpler examination and assist a doctor in his or her examination.

[0011] FIG. 1 is a block diagram showing the configuration of a system including an image generating device according to an embodiment. FIG. 2 is a sequence diagram showing operations performed by a system including an image generating device according to an embodiment when training a machine learning model. FIG. 3 is a flowchart showing a detailed flow of step S16 shown in FIG. 2. FIG. 4 is a diagram showing an example of first image data transmitted in step S12 of FIG. 2. FIG. 5 is a diagram showing an example of fourth image data transmitted in step S14 of FIG. 2 and an example of a three-dimensional osteochondral model generated in step S15 of FIG. 2. FIG. 6A is a diagram showing an example of image data obtained in step S21 of FIG. 3. FIG. 6B is a diagram showing an example of image data obtained in step S22 of FIG. 3. FIG. 6C is a diagram showing an example of image data obtained in step S23 of FIG. 3. FIG. 6D is a diagram showing an example of image data obtained in step S24 of FIG. 3. FIG. 7 is a sequence diagram showing operations performed by a system including an image generating device according to an embodiment during operation. FIG. 8 is a diagram showing examples of first image data used as input data and second image data generated as output data by an image generating device according to an embodiment during operation.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. Therefore, the numerical values, shapes, materials, components, component placement positions, connection forms, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Therefore, among the components in the following embodiments, components that are not recited in the independent claims of the present disclosure will be described as optional components.

[0013] Note that each figure is a schematic diagram and is not necessarily an exact illustration. Therefore, the scales and the like do not necessarily match in each figure. Furthermore, in each figure, the same reference numerals are used for substantially the same configurations, and redundant explanations will be omitted or simplified.

[0014] Furthermore, in this specification, the terms "up" and "down" do not necessarily refer to the upward direction (vertically upward) and downward direction (vertically downward) in absolute spatial recognition.

[0015] Furthermore, the image generating device of this embodiment is applicable to animals such as humans, dogs, or cats, and in this specification, we will explain the case where it is applied to humans (hereinafter referred to as patients).

[0016] (Embodiment) An image generating device according to this embodiment is a device used to evaluate a patient's articular cartilage, cartilage other than articular cartilage (such as costal cartilage or auricular cartilage), joint shape, and joint diseases (such as fractures, dislocations, osteoarthritis, and rheumatoid arthritis). For example, image data obtained by imaging (i.e., examination) using an X-ray examination device does not display the articular cartilage region. Therefore, the patient's doctor cannot accurately diagnose fracture lines in the cartilage region using the image data, and therefore cannot provide appropriate treatment. Therefore, to evaluate the condition of the articular cartilage, the doctor uses image data obtained by imaging (i.e., examination) using an MRI examination device. However, imaging using an MRI examination device requires suppressing the patient's body movement, which is a problem, particularly for pediatric patients. Furthermore, MRI examination devices are generally installed in large hospitals and are often not installed in small hospitals such as clinics in urban areas. This can lead to the problem of overlooking fracture lines in the cartilage region and delaying treatment.

[0017] An image generating device that can solve the above problems will be described below.

[0018] [Configuration] First, the configuration of a system including an image generation device according to this embodiment will be described. Fig. 1 is a block diagram showing the configuration of a system including an image generation device 4 according to this embodiment. The system shown in Fig. 1 is made up of an X-ray inspection device 1, an MRI inspection device 2, an image synthesis device 3, and an image generation device 4.

[0019] The X-ray inspection device 1 is a diagnostic inspection device that uses X-rays to image bones present inside a patient's body. The X-ray inspection device 1 is operated by a medical professional to image a part of a patient (i.e., an animal) that has bones therein and acquires the image as two-dimensional first image data. The X-ray inspection device 1 also includes a communication unit 11 and an imaging unit 12.

[0020] The communication unit 11 is a communication circuit for communicating with the image synthesis device 3 and the image generation device 4. The communication unit 11 transmits the first image data to the image synthesis device 3 or the image generation device 4. The communication unit 11 may perform, for example, wireless communication or wired communication. Furthermore, the standard of communication performed by the communication unit 11 is not particularly limited.

[0021] The imaging unit 12 captures an image of a part of the patient's body that has bone therein, and acquires the image as first image data.

[0022] The MRI examination device 2 is a diagnostic examination device that uses nuclear magnetic resonance to image bones, cartilage, and the like present inside a patient's body. Operated by a medical professional, the MRI examination device 2 images a part of the patient's body that contains bones and acquires the image as two-dimensional fourth image data. The fourth image data is composed of multiple image data obtained by slightly changing the imaging area within the part, and contains information that allows the part to be reconstructed three-dimensionally. The MRI examination device 2 also includes a communication unit 21 and an imaging unit 22.

[0023] The communication unit 21 is a communication circuit for communicating with the image synthesis device 3. The communication unit 21 transmits the fourth image data to the image synthesis device 3. The communication unit 21 may perform, for example, wireless communication or wired communication. Furthermore, the standard of communication performed by the communication unit 21 is not particularly limited.

[0024] The imaging unit 22 images a part of the patient that has bone inside it and acquires the image as fourth image data. Note that the part of the patient that is imaged by the imaging unit 22 is the same as or a part of the part of the patient that is imaged by the imaging unit 12 of the X-ray inspection apparatus 1.

[0025] The image composition device 3 is a terminal device such as a personal computer or a tablet that composes third image data based on the first image data transmitted by the X-ray inspection device 1 and the fourth image data transmitted by the MRI inspection device 2. The image composition device 3 also includes a communication unit 31 and a control unit 32. The third image data is two-dimensional data generated by aligning and combining two-dimensional image data obtained by imaging a region having a bone therein using the X-ray inspection device 1 and three-dimensional image data obtained by imaging the same region using the MRI inspection device 2. Specifically, the third image data is two-dimensional image data that indicates a bone and a region of cartilage continuous with the bone.

[0026] The communication unit 31 is a communication circuit for communicating with the X-ray inspection device 1, the MRI inspection device 2, and the image generation device 4. The communication unit 31 receives first image data from the X-ray inspection device 1 and fourth image data from the MRI inspection device 2. The communication unit 31 also transmits third image data to the image generation device 4. The communication unit 31 may perform, for example, wireless communication or wired communication. Furthermore, there are no particular limitations on the standard of communication performed by the communication unit 31.

[0027] The control unit 32 controls the entire image synthesis device 3. The control unit 32 is realized by a microcomputer, a processor, or the like. That is, the functions of the control unit 32 are realized by the microcomputer or the processor executing a program stored in a memory, etc. The control unit 32 also includes a synthesis unit 321.

[0028] The synthesis unit 321 creates a three-dimensional bone and cartilage model, which is three-dimensional image data, from the fourth image data. The three-dimensional bone and cartilage model is three-dimensional image data created by connecting bone and cartilage regions included in the fourth image data.

[0029] Furthermore, the synthesis unit 321 aligns the positions of the bones shown in the first image data with those shown in the created three-dimensional osteochondral model. After alignment, the synthesis unit 321 projects the three-dimensional osteochondral model onto the first image data and synthesizes two-dimensional third image data in which the osteochondral region is displayed on the first image data. The osteochondral region is a region in which the bones and cartilage represented by the three-dimensional osteochondral model are displayed on the first image data. Furthermore, the method of synthesis of the third image data by the synthesis unit 321 will be described in detail below.

[0030] The image generating device 4 is a terminal device such as a personal computer or tablet that generates second image data from the first image data. The second image data is two-dimensional image data that indicates bones included in the first image data and cartilage regions that are continuous with the bone positions estimated from the bone positions. The image generating device 4 also includes a communication unit 41, a storage device 42, a control unit 43, and a display unit 44.

[0031] The communication unit 41 is a communication circuit for communicating with the X-ray inspection apparatus 1 and the image synthesis device 3. The communication unit 41 receives the first image data from the X-ray inspection apparatus 1 and the third image data from the image synthesis device 3. The communication unit 41 may perform, for example, wireless communication or wired communication. Furthermore, the standard of communication performed by the communication unit 41 is not particularly limited.

[0032] The storage device 42 is a memory or the like that stores the first image data and the third image data received by the communication unit 41, programs, and the like.

[0033] The control unit 43 controls the entire image generation device 4. The control unit 43 is realized by a microcomputer, a processor, or the like. That is, the functions of the control unit 43 are realized by the microcomputer or the processor executing a program stored in the storage device 42. The control unit 43 also includes a generation unit 431.

[0034] The generation unit 431 has a trained machine learning model 432, and generates second image data by inputting the first image data into the machine learning model 432. Note that the second image data is two-dimensional image data.

[0035] The machine learning model 432 is, for example, a model based on U-net, a semantic segmentation model developed for biomedicine. The machine learning model 432 is a model that has been trained in advance prior to generating the second image data. Specifically, prior to generating the second image data, the generation unit 431 trains the machine learning model 432 using the first image data as input data and the third image data as output data. Specific details about the training of the machine learning model 432 will be described later.

[0036] The display unit 44 is realized by, for example, a liquid crystal display, etc. The display unit 44 displays the second image data generated by the generation unit 431. Note that the display unit 44 may also display the first image data, the third image data, etc., as necessary.

[0037] [Learning of Machine Learning Model] FIG. 2 is a sequence diagram showing operations executed by a system including an image generating device 4 according to an embodiment when learning a machine learning model 432.

[0038] First, the imaging unit 12 of the X-ray inspection apparatus 1 is operated by a medical professional to image a part of a patient's body that has bones inside (step S11).

[0039] The communication unit 11 of the X-ray inspection apparatus 1 transmits the first image data obtained by the imaging in step S11 to the image synthesis device 3 (step S12).

[0040] The imaging unit 22 of the MRI inspection device 2 is operated by a medical professional to image a part of the patient's body that has bones inside (step S13). Note that the part imaged by the imaging unit 22 in step S13 is the same as or a part of the part imaged by the imaging unit 12 in step S11. In addition, the patient imaged by the MRI inspection device 2 in step S13 is the same patient as the patient imaged by the X-ray inspection device 1 in step S11.

[0041] The communication unit 21 of the MRI examination apparatus 2 transmits the fourth image data obtained by the imaging in step S13 to the image synthesis apparatus 3 (step S14).

[0042] The synthesis unit 321 of the image synthesis device 3 creates a three-dimensional bone and cartilage model from the fourth image data transmitted in step S14 (step S15).

[0043] The synthesis unit 321 synthesizes third image data based on the first image data transmitted in step S12 and the three-dimensional bone and cartilage model created in step S15 (step S16).

[0044] The communication unit 31 of the image synthesis device 3 transmits the third image data synthesized in step S16 to the image generation device 4 (step S17).

[0045] X-ray inspection apparatus 1 transmits the first image data obtained by the imaging in step S11 to image generation device 4 (step S18).

[0046] The generation unit 431 of the image generation device 4 trains the machine learning model 432 using the first image data as input data transmitted in step S18 and the third image data as output data transmitted in step S17 (step S19). Specifically, by repeating steps S11 to S19 for multiple patient sites, the generation unit 431 trains the machine learning model 432 using multiple sets of the first image data and the third image data.

[0047] Note that steps S13 and S14 may be executed before steps S11 and S12.

[0048] Also, step S18 may be executed simultaneously with step S12, for example.

[0049] FIG. 3 is a flowchart showing the detailed flow of step S16 shown in FIG.

[0050] First, the synthesis unit 321 binarizes the first image data and extracts the bone contour pattern from the first image data (step S21).

[0051] The synthesis unit 321 superimposes the three-dimensional bone-cartilage model on the first image data so that the bone contour pattern extracted in step S21 matches the contour pattern of the three-dimensional bone-cartilage model created in step S15 of Fig. 2 (step S22). Note that in step S22, the synthesis unit 321 enlarges, reduces, rotates, or the like the three-dimensional bone-cartilage model to match the bone contour pattern with the contour pattern of the three-dimensional bone-cartilage model.

[0052] The synthesis unit 321 projects the region where the first image data and the three-dimensional bone and cartilage model overlap onto the first image data (step S23).

[0053] The synthesis unit 321 synthesizes third image data that displays the region projected in step S23 as an osteochondral region with the first image data (step S24).

[0054] The sequence diagram illustrated in Fig. 2 and the flowchart illustrated in Fig. 3 will be described below using specific image examples shown in Fig. 4 to Fig. 6D. Note that the following specific examples will describe learning examples of the machine learning model 432 that generates second image data showing bones constituting the hip joint area and the cartilage area continuous with the bones.

[0055] Fig. 4 is a diagram showing an example of the first image data transmitted in step S12 in Fig. 2. The first image data shown in Fig. 4 is image data obtained by capturing an image of a child patient from the waist to the vicinity of the base of the legs.

[0056] As shown in FIG. 4, the first image data is image data obtained by capturing images of pelvises 51a, 51b, 51c, and 51d and femurs 52a and 52b.

[0057] FIG. 5 shows an example of one of the fourth image data transmitted in step S14 of FIG. 2 and an example of a three-dimensional osteochondral model generated in step S15 of FIG. 2. (a) of FIG. 5 shows an example of one of the fourth image data transmitted in step S14 of FIG. 2. Note that the image data shown in (a) of FIG. 5 is image data capturing an area from the waist to the groin of a pediatric patient, similar to the first image data shown in FIG. 4. The pediatric patient who was the subject of the image data shown in (a) of FIG. 5 is the same patient who was the subject of the first image data shown in FIG. 4. The hatching in (a) of FIG. 5 is a pattern for clearly indicating the bone and cartilage regions in that area. (b) of FIG. 5 shows an example of a three-dimensional osteochondral model generated in step S15 of FIG. 2. Note that the three-dimensional osteochondral model shown in (b) of FIG. 5 is a three-dimensional osteochondral model generated from the fourth image data including the image data shown in (a) of FIG. 5.

[0058] As shown in (a) of FIG. 5, the image data is image data obtained by capturing images of pelvises 51a and 51b, femurs 52a and 52b, and cartilages 53a and 53b. Cartilage 53a is cartilage that is continuous with femur 52a and is articular cartilage that exists between pelvises 51a and 51c and femur 52a. Cartilage 53b is cartilage that is continuous with femur 52b and is articular cartilage that exists between pelvises 51b and 51d and femur 52b. Although not shown in FIG. 5, the fourth image data is composed of multiple image data obtained by slightly changing the imaging area within the region shown in (a) of FIG. 5. In other words, the fourth image data is composed of multiple image data captured in the thickness direction of the patient's body, and also includes image data of pelvises 51c and 51d.

[0059] As shown in FIG. 5B, the synthesis unit 321 creates a three-dimensional bone and cartilage model, which is three-dimensional image data showing the pelvises 51a, 51b, 51c, and 51d, the femurs 52a and 52b, and the cartilages 53a and 53b.

[0060] Next, a specific example in which the synthesis unit 321 synthesizes two-dimensional third image data based on the first image data shown in Fig. 4 and the three-dimensional bone and cartilage model shown in Fig. 5(b) will be described with reference to Figs. 6A to 6D. Note that Figs. 6A to 6D describe a case in which the synthesis unit 321 superimposes the femur 52b and cartilage 53b of the three-dimensional bone and cartilage model on the two-dimensional first image data.

[0061] FIG. 6A is a diagram showing an example of image data (that is, binarized first image data) obtained in step S21 of FIG.

[0062] 6A, the synthesis unit 321 binarizes the first image data and extracts bone contour patterns. Specifically, the synthesis unit 321 extracts contour patterns of the pelvises 51 a, 51 b, 51 c, and 51 d and the femurs 52 a and 52 b.

[0063] FIG. 6B is a diagram showing an example of image data obtained in step S22 of FIG. 3 (i.e., the first image data on which the three-dimensional bone and cartilage model is superimposed).

[0064] As shown in Fig. 6B, the synthesis unit 321 superimposes the three-dimensional osteochondral model on the first image data so that the bone contour pattern extracted in Fig. 6A matches the contour pattern of the three-dimensional osteochondral model shown in Fig. 5(b). In the example of Fig. 6B, the synthesis unit 321 superimposes the femur 52b and cartilage 53b of the three-dimensional osteochondral model on the first image data so that the bone contour pattern of the femur 52b extracted in Fig. 6A matches the contour pattern of the femur 52b shown in the three-dimensional osteochondral model.

[0065] Fig. 6C is a diagram showing an example of image data obtained in step S23 of Fig. 3. Note that the diagonal hatching shown in Fig. 6C is a pattern for indicating the area projected onto the first image data.

[0066] As shown in FIG. 6C, the synthesis unit 321 projects the region where the three-dimensional bone and cartilage model shown in FIG. 6B is superimposed onto the first image data.

[0067] FIG. 6D is a diagram showing an example of image data obtained in step S24 of FIG.

[0068] As shown in Figure 6D, the synthesis unit 321 synthesizes third image data that displays the area where the femur 52b and cartilage 53b are projected (i.e., the area hatched with diagonal lines in Figure 6C) as an osteochondral area (the area represented by dashed lines in Figure 6D) with the first image data.

[0069] [Operation of Machine Learning Model] Fig. 7 is a sequence diagram showing operations executed during operation of a system including the image generating device 4 according to an embodiment. Note that the operations shown in Fig. 7 are operations executed after the operations shown in Fig. 2 (i.e., learning of the machine learning model) are completed.

[0070] First, the imaging unit 12 of the X-ray inspection apparatus 1 is operated by a medical professional to image a part of a patient's body that has a bone inside (step S31). Note that the part imaged by the X-ray inspection apparatus 1 in step S31 is the part learned by the machine learning model 432 in Fig. 2, that is, the same part imaged by the X-ray inspection apparatus 1 in step S11.

[0071] The communication unit 11 of the X-ray inspection apparatus 1 transmits the first image data obtained by the imaging in step S11 to the image generation device 4 (step S32).

[0072] The generation unit 431 of the image generation device 4 generates second image data by inputting the first image data transmitted in step S32 to the machine learning model 432 (step S33).

[0073] The display unit 44 of the image generating device 4 displays the second image data generated in step S33 (step S34).

[0074] The sequence diagram illustrated in FIG. 7 will be described using specific image examples shown in FIG. 8 . FIG. 8 is a diagram illustrating examples of first image data used as input data by the image generating device 4 according to the embodiment during operation and second image data generated as output data. The image data illustrated in FIG. 8 is image data capturing the same region as the region captured by the image data illustrated in FIG. 4 . (a) of FIG. 8 is a diagram illustrating an example of first image data used as input data by the image generating device 4 according to the embodiment during operation. (b) of FIG. 8 is a diagram illustrating an example of second image data generated as output data by the image generating device 4 according to the embodiment during operation. The second image data illustrated in (b) of FIG. 8 is an example of image data displayed on the display unit 44. The diagonal hatching illustrated in (b) of FIG. 8 is a pattern indicating a cartilage region.

[0075] As shown in (a) of Figure 8, the first image data is image data in which the pelvis 51a, 51b, 51c, and 51d and the femur 52a and 52b are imaged, and is image data similar to the first image data shown in Figure 4.

[0076] 8(b), the second image data is image data that displays pelvises 51a, 51b, 51c, and 51d, femurs 52a and 52b, and cartilages 53a and 53b. The doctor in charge can examine the patient's articular cartilage and the like by using the second image data generated by the image generating device 4.

[0077] In addition, when the image generation device 4 generates second image data for a part other than the part shown in Figure 8, the system including the image generation device 4 of this embodiment simply executes the sequence diagram shown in Figure 2 again for the other part and trains the machine learning model 432 in advance.

[0078] As explained above, the image generating device 4 according to this embodiment makes it possible to display the cartilage region through an examination that is less burdensome for patients, such as imaging with the X-ray inspection device 1, without requiring a large-scale MRI inspection device, and which is difficult for children to examine. Furthermore, the image generating device 4 according to this embodiment makes it possible for even a doctor working in a small hospital, such as a clinic in a city center, that does not have an MRI inspection device 2 but does have the X-ray inspection device 1 to make an accurate diagnosis and provide appropriate treatment.

[0079] [Performance Evaluation of Machine Learning Model] In order to confirm the effectiveness of the image generating device 4 according to this embodiment, the inventors performed a performance evaluation of the machine learning model 432. The procedure for the performance evaluation performed by the inventors and the results of the performance evaluation will be described below.

[0080] First, the procedure for the performance evaluation carried out by the inventors will be described.

[0081] The inventors prepared 100 sets of training data. Specifically, the inventors prepared sets of image data for 100 patients (in other words, 100 cases), with a set of first image data and third image data obtained from the same patient being one set (in other words, one case). Note that each of the first image data and the third image data is image data of both the left and right hip joints. In other words, the inventors prepared 100 sets of image data, each set capturing two patterns of hip joint images.

[0082] The inventors used 100 sets of prepared training data to train the machine learning model 432 and evaluate the performance of the trained machine learning model 432. Note that the inventors used K-fold cross validation for training the machine learning model 432 and evaluating the performance of the trained machine learning model 432. K-fold cross validation is a technique in which multiple sets of training data are divided into K groups, one of which is used as a performance evaluation dataset and the remaining groups are used as training datasets. In other words, there are K combinations of performance evaluation datasets and training datasets.

[0083] In this specific example, the case where K = 10 will be described. That is, in this specific example, 10 of the 100 sets are used as data sets for performance evaluation, and the remaining 90 sets are used as data sets for training. Note that the value of K can be changed arbitrarily depending on the number of data sets of training data.

[0084] In this specific example, the learning datasets are divided into a training set and a validation set. The training set is a dataset used for actual learning of the machine learning model 432. On the other hand, the validation set is a dataset used to determine whether retraining of the machine learning model 432 that has been trained using the training set is necessary. For example, of the 90 sets used as the learning datasets, 80 sets are used as training sets, and the remaining 10 sets are used as validation sets.

[0085] First, the learning method of the machine learning model 432 will be described.

[0086] First, the generation unit 431 prepares ten machine learning models 432. The generation unit 431 inputs a training set to each of the ten machine learning models 432, and trains the ten machine learning models 432. Note that the generation unit 431 ensures that the training sets input to each of the ten machine learning models 432 are not the same. In this way, the generation unit 431 prepares ten trained machine learning models 432 that have been trained using different training sets. Then, the generation unit 431 inputs a dedicated validation set to each of the ten trained machine learning models 432. Specifically, the generation unit 431 determines, for each of the ten trained machine learning models 432, whether or not the machine learning model 432 needs to be retrained, using a dataset (i.e., a validation set) that was not used when training the machine learning model 432. More specifically, the generation unit 431 determines whether the machine learning model 432, which has been trained using a training set of one pattern (also referred to as a first pattern) among the ten combinations of training sets and validation sets, needs to be retrained using the validation set of the first pattern. When the generation unit 431 determines that a certain machine learning model 432 needs to be retrained, the generation unit 431 adjusts parameters set for the machine learning model 432 and retrains the machine learning model 432 using the training set. Note that the generation unit 431 determines whether the machine learning model 432 needs to be retrained based on, for example, the degree of agreement between second image data generated by the machine learning model 432 based on first image data included in the validation set and third image data included in the validation set. Specifically, the generation unit 431 determines whether the machine learning model 432 needs to be retrained based on the degree of agreement between the osteochondral region indicated in the second image data generated by the machine learning model 432 and the osteochondral region indicated in the third image data.

[0087] Next, performance evaluation of the trained machine learning model 432 will be described.

[0088] The generation unit 431 inputs a dedicated performance evaluation dataset to each of the 10 trained machine learning models 432, and acquires output data output by each of the 10 trained machine learning models 432. Specifically, the generation unit 431 inputs the evaluation performance dataset of a first pattern to the machine learning model 432 that has been trained with a training dataset of one pattern (also referred to as a first pattern) out of 10 patterns of combinations of evaluation performance datasets and training datasets, thereby acquiring output data output by the machine learning model 432. The generation unit 431 performs this process for each of the 10 trained machine learning models 432. The generation unit 431 then averages the acquired 10 pieces of output data to evaluate the performance of the 10 trained machine learning models 432. The inventors used three indices, F1 score, IoU (Intersection over Union), and MAD (Mean Absolute Distance), to evaluate the performance of the 10 trained machine learning models 432. Specifically, each of the 10 trained machine learning models 432 calculates the F1 score, IoU, and MAD, and outputs information on the calculated three indices as output data. Then, the generation unit 431 calculates the average value and standard deviation for each of the three indices based on the acquired output data.

[0089] First, the F1 score, IoU, and MAD will be described.

[0090] The F1 score is an index indicating the accuracy of a machine learning model and is expressed as a number between 0 and 1. The closer the F1 score is to 0, the lower the accuracy of the machine learning model, and the closer it is to 1, the higher the accuracy of the machine learning model. Generally, an F1 score of 0.8 or greater but less than 0.9 is evaluated as a machine learning model (good machine learning model) with high recall and high precision (hereinafter also referred to as precision), and an F1 score of 0.9 or greater is evaluated as a machine learning model (excellent machine learning model) with very high recall and very high precision (precision). The "recall" and "precision" described herein are the ratios calculated from the second image data (predicted data) generated by the machine learning model from the first image data included in the performance evaluation dataset and the third image data (correct data) included in the performance evaluation dataset. Specifically, the "recall" is the ratio of the overlap between the predicted region (specifically, the osteochondral region indicated by the predicted data) and the correct region (specifically, the osteochondral region indicated by the correct data). The "accuracy rate" is the ratio of the overlapping portion between the predicted region and the correct region to the predicted region.

[0091] The F1 score is a numerical value calculated using the above-mentioned "recall rate" and "precision rate" according to the following formula (1).

[0092]

[0093] IoU is an index indicating the degree of overlap between the predicted region (the osteochondral region shown in the predicted data) and the correct region (the osteochondral region shown in the correct data) by the machine learning model, and is expressed as a value between 0 and 1. The closer the IoU value is to 0, the less overlap there is between the predicted region and the correct region, and the closer it is to 1, the more overlap there is between the predicted region and the correct region. Generally, when the IoU is 0.75 or more and less than 0.9, the predicted region and the correct region are evaluated as having a high degree of agreement (i.e., high accuracy), and when the IoU is 0.9 or more, the predicted region and the correct region are evaluated as being very similar (i.e., almost a perfect match). Furthermore, when the IoU is 0.5 or more and less than 0.75, the degree of agreement between the predicted region and the correct region is evaluated as being within an acceptable range for many applications.

[0094] Furthermore, if the predicted region by the machine learning model is A and the correct region is B, IoU is a numerical value calculated by the following formula (2).

[0095]

[0096] The MAD is an index that indicates the average linear distance between the contour of the osteochondral region shown in the predicted data and the contour of the osteochondral region shown in the correct data. In this specific example, the linear distance between each of the multiple points that define the contour of the osteochondral region shown in the predicted data and the contour of the osteochondral region shown in the correct data is calculated, and the average value of these linear distances is calculated as the MAD.

[0097] Next, we will explain the results obtained by using the F1 score, IoU, and MAD indices.

[0098] In a performance evaluation conducted by the inventors, the F1 score was 0.90±0.10. As explained above, an F1 score of 0.8 or greater but less than 0.9 indicates a good machine learning model, and an F1 score of 0.9 or greater indicates an excellent machine learning model. In other words, it was confirmed that the 10 trained machine learning models 432 were models with high precision and high recall.

[0099] Furthermore, in a performance evaluation conducted by the inventors, the IoU was 0.83±0.12. As described above, when the IoU is 0.75 or greater but less than 0.9, the predicted region and the correct region are evaluated as having a high degree of agreement (i.e., high accuracy), and when the IoU is 0.9 or greater, the predicted region and the correct region are evaluated as having a very high degree of agreement (i.e., almost perfect agreement). Furthermore, when the IoU is 0.5 or greater but less than 0.75, the agreement rate between the predicted region and the correct region is evaluated as being within an acceptable range for many applications. In other words, the 10 trained machine learning models 432 were confirmed to be models at a level that is at least acceptable for practical use. Note that, in high-precision applications such as medical image processing (e.g., lesion detection), an IoU of 0.85 or greater may be required as a standard for model use. However, the inventors believe that the machine learning model 432 can meet this condition by increasing the amount of data used for training.

[0100] Furthermore, in a performance evaluation conducted by the inventors, the MAD was 1.33±1.21 mm. When an imaging diagnostic device is clinically applied as a medical device, it is generally required that the error of predicted data relative to ground truth data be approximately 1 mm. In this performance evaluation, the MAD was 1.33±1.21 mm, confirming that the above indicators were generally met. In other words, the 10 trained machine learning models 432 were confirmed to be models with relatively high accuracy. The inventors believe that the machine learning model 432 can meet the above indicators by increasing the amount of data used for training.

[0101] The above performance evaluation of the machine learning model 432 suggests that the image generating device 4 according to the present embodiment is capable of generating second image data with sufficient accuracy from first image data obtained by imaging using the X-ray inspection device 1. As a result, the inventors have confirmed that the image generating device 4 according to the present embodiment can generate image data indicating a cartilage region with sufficiently high accuracy.

[0102] [Effect] The image generating device 4 according to this embodiment includes a storage device 42 that stores first image data obtained by imaging an animal part having a bone inside using the X-ray inspection device 1, and a generating unit 431 that has a trained machine learning model 432 and generates second image data that indicates the bone and an area of ​​cartilage that is continuous with the bone and that is estimated from the position of the bone by inputting the first image data into the machine learning model 432.

[0103] The image generation device 4 generates second image data from first image data obtained by imaging using the X-ray inspection device 1, thereby enabling image data showing the cartilage area to be generated through a simpler examination, thereby assisting the attending physician in their examination.

[0104] In addition, in the image generation device 4 of this embodiment, the storage device 42 further stores third image data, which is image data including the same parts as those included in the first image data, and the third image data is image data showing bones and areas of cartilage continuous with the bones, and the generation unit 431 further trains the machine learning model 432 using the first image data as input data and the third image data as output data prior to generating the second image data.

[0105] The generation unit 431 trains the machine learning model 432 in advance prior to generating the second image data, and therefore the second image data can be generated by inputting the first image data into the machine learning model 432. This allows the image generation device 4 to generate image data showing the cartilage region through a simpler examination, thereby assisting the doctor in his or her examination.

[0106] In addition, in the image generation device 4 according to this embodiment, the third image data is two-dimensional image data generated by aligning and synthesizing two-dimensional image data obtained by imaging the part using the X-ray inspection device 1 and three-dimensional image data obtained by imaging the part using the MRI inspection device 2.

[0107] The third image data is image data generated by aligning and synthesizing image data obtained by imaging with the X-ray inspection device 1 and image data obtained by imaging with the MRI inspection device 2, and therefore displays a region of cartilage that is continuous with the bone. As a result, the generation unit 431 uses the third image data as output data to train the machine learning model 432, and can generate more accurate second image data.

[0108] In the image generating device 4 according to this embodiment, the second image data generated by the generating unit 431 is two-dimensional image data.

[0109] Since the second image data is two-dimensional image data, the image generating device 4 can reduce the data capacity.

[0110] Moreover, the image generating device 4 according to this embodiment further includes a display unit 44 that displays the second image data generated by the generating unit 431 .

[0111] The image generating device 4 includes a display unit 44 that displays the second image data, so that the doctor in charge can examine the patient using the second image data displayed on the display unit 44.

[0112] In addition, the image generation method of this embodiment is an image generation method executed by the image generation device 4, which has a memory device 42 that stores first image data obtained by imaging a part of an animal that has a bone inside using the X-ray inspection device 1, and the image generation method includes a generation step (S33) of generating second image data that shows the bone and a region of cartilage that is continuous with the bone and that is estimated from the position of the bone by inputting the first image data into the machine learning model 432.

[0113] The image generation method generates second image data from first image data obtained by imaging using the X-ray inspection device 1, thereby enabling image data showing the cartilage area to be generated through a simpler examination, thereby assisting the attending physician in their examination.

[0114] Furthermore, the program according to the present embodiment causes a computer to execute the image generating method described above.

[0115] Such a program provides the same effects as the image generating method according to the present embodiment.

[0116] (Other) While the image generating device and the like according to the present disclosure have been described above based on the above embodiment, they are not limited to the above embodiment. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by those skilled in the art to the above embodiment and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects.

[0117] In the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0118] In the above-described embodiments, some or all of the functions of the components may be realized by a processor such as a CPU executing a program.

[0119] In the above-described embodiment, the processing performed by a specific processing unit may be performed by another processing unit. The order of multiple processing operations may be changed, or multiple processing operations may be performed in parallel.

[0120] Furthermore, the method of communication between the devices in the above-described embodiment is not particularly limited, and a relay device (not shown) may be involved in the communication between the devices.

[0121] Furthermore, some or all of the components constituting each of the above devices may be configured as an IC card or a standalone module that can be attached to or detached from each device. The IC card or module is a computer system configured from a microprocessor, ROM, RAM, etc. The IC card or module may include a super multi-function LSI. The IC card or module achieves its functions when the microprocessor operates in accordance with a computer program. The IC card or module may be tamper-resistant.

[0122] The present disclosure may also be realized as a method executed by a computer, or as a program for causing a computer to execute the method. The present disclosure may also be realized as a computer-readable non-transitory recording medium on which such a program is recorded. Such a program includes an application program for causing a general-purpose information terminal (computer) to function as the setting terminal of the above-described embodiment, and an application program installed on the information terminal.

[0123] [Additional Notes] (Technology 1) An image generation device comprising: a storage device that stores first image data obtained by imaging an animal part that has a bone inside using an X-ray inspection device; and a generation unit that has a trained machine learning model and generates second image data that shows the bone and a region of cartilage that is continuous with the bone and that is estimated from the position of the bone by inputting the first image data into the machine learning model.

[0124] (Technology 2) The image generating device described in Technology 1, wherein the storage device further holds third image data, which is image data including the same part as the part included in the first image data, and the third image data is image data showing the bone and a cartilage area continuous with the bone, and the generating unit further trains the machine learning model using the first image data as input data and the third image data as output data prior to generating the second image data.

[0125] (Technology 3) The image generating device described in Technology 2, wherein the third image data is two-dimensional image data generated by aligning and synthesizing two-dimensional image data obtained by imaging the region using an X-ray inspection device and three-dimensional image data obtained by imaging the region using an MRI inspection device.

[0126] (Technology 4) The image generating device according to any one of Technologies 1 to 3, wherein the second image data generated by the generating unit is two-dimensional image data.

[0127] (Technology 5) The image generating device according to any one of Technologies 1 to 4, further comprising a display unit that displays the second image data generated by the generation unit.

[0128] (Technology 6) An image generation method executed by an image generation device, the image generation device having a storage device for storing first image data obtained by imaging a part of an animal that has a bone inside using an X-ray inspection device, the image generation method including a generation step of generating second image data showing the bone and a region of cartilage continuous with the bone that is estimated from the position of the bone by inputting the first image data into a machine learning model.

[0129] (Technology 7) A program for causing a computer to execute the image generating method described in Technology 6.

[0130] An image generating device according to the present disclosure is useful, for example, as an image evaluation support tool for evaluating the condition of a patient's articular cartilage, etc.

[0131] REFERENCE SIGNS LIST 1 X-ray inspection device 11, 21, 31, 41 Communication unit 12, 22 Imaging unit 2 MRI inspection device 3 Image synthesis device 32, 43 Control unit 321 Synthesis unit 4 Image generation device 42 Storage device 431 Generation unit 432 Machine learning model 44 Display unit 51a, 51b, 51c, 51d Pelvis 52a, 52b Femur 53a, 53b Cartilage

Claims

1. An image generation device comprising: a storage device that stores first image data obtained by imaging an animal part containing a bone using an X-ray inspection device; and a generation unit that has a trained machine learning model and generates second image data showing the bone and an area of ​​cartilage continuous with the bone that is estimated from the position of the bone by inputting the first image data into the machine learning model.

2. The image generating device of claim 1, wherein the storage device further stores third image data that is image data including the same area as the area included in the first image data, the third image data being image data showing the bone and an area of ​​cartilage continuous with the bone, and the generating unit further trains the machine learning model using the first image data as input data and the third image data as output data prior to generating the second image data.

3. The image generating device according to claim 2, wherein the third image data is two-dimensional image data generated by aligning and synthesizing two-dimensional image data obtained by imaging the region using an X-ray inspection device and three-dimensional image data obtained by imaging the region using an MRI inspection device.

4. The image generating device according to claim 1, wherein the second image data generated by the generating unit is two-dimensional image data.

5. The image generating device according to any one of claims 1 to 4, further comprising a display unit that displays the second image data generated by the generating unit.

6. An image generation method executed by an image generation device, the image generation device having a storage device for storing first image data obtained by imaging a part of an animal having a bone therein using an X-ray inspection device, the image generation method including a generation step of generating second image data showing the bone and a region of cartilage continuous with the bone estimated from the position of the bone by inputting the first image data into a machine learning model.

7. A program for causing a computer to execute the image generating method according to claim 6.

Images