System, program and method for determining bone formation abnormalities, and method for generating trained model

Abstract

To provide a technology capable of assisting physicians by automatically performing determination relating to bone formation abnormalities.SOLUTION: A system for determining bone formation abnormalities acquires an XP image set including a plurality of XP images relating to the same patient, and performs determination relating to bone formation abnormalities by using a trained model based on the XP image set.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a system, a program, and a method for making a judgment regarding bone dysplasia, and a method for generating a trained model. [Background technology]

[0002] In the medical field, diseases are sometimes diagnosed using medical images such as XP (X-ray Photograph) images. In recent years, a trained model has been constructed using AI (artificial intelligence), and images are input into this model to assist doctors in making diagnoses. For example, Patent Document 1 describes a method of using machine learning to determine whether a medical image has been captured in a position suitable for making a diagnosis. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7404857 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there has been a problem with the difficulty of automatically diagnosing bone dysplasia, especially in the case of osteoporosis. Conventionally, diagnosing bone dysplasia based on XP images has required comprehensive judgment by an experienced physician, which has generally been difficult.

[0005] The present invention has been made to solve these problems, and aims to provide a system, program, and method that can assist doctors by automatically making judgments regarding bone dysplasia, as well as a method for generating a trained model. [Means for solving the problem]

[0006] An example of a system for determining bone dysplasia according to the present invention includes: obtaining an XP image set including multiple XP images relating to the same patient; Based on the XP image set, a trained model is used to make a judgment regarding bone dysplasia.

[0007] In one example, the diagnosis is a diagnosis regarding a disease that results in short stature.

[0008] In one example, the determination is a determination regarding achondroplasia syndrome.

[0009] In one example, the plurality of XP images are of a patient aged 3 years or younger.

[0010] In one example, the XP image set includes: Includes XP images of multiple imaging sites from the head, upper limbs, spine, hand, pelvis, and lower limbs, Includes XP images taken from the front and XP images taken from the side, Includes XP images in multiple positions, including standing, lying down, and sitting, or Contains XP images taken at different times.

[0011] In one example, the system outputs a heat map for each of the plurality of XP images that indicates the importance of each pixel to the decision.

[0012] In one example, the system performs the following for at least one of the plurality of XP images: - extracting a region representing a predetermined site; - generating a modified image by resizing the extracted region to a predetermined size; A judgment regarding bone dysplasia is made using the trained model based on the XP image set including the corrected image.

[0013] In one example, the system generates a modified image by replacing a predetermined region of at least one of the plurality of XP images with a predetermined image; A judgment regarding bone dysplasia is made using the trained model based on the XP image set including the corrected image.

[0014] An example of a program according to the present invention causes a computer to function as the above-described system.

[0015] An example of the method for determining bone dysplasia according to the present invention includes: acquiring, by a computer, an XP image set including a plurality of XP images relating to the same patient; A step in which a computer makes a judgment regarding bone dysplasia using a trained model based on the XP image set; Equipped with.

[0016] An example of a method for generating a trained model for determining bone dysplasia according to the present invention is as follows: a step of acquiring raw data by a computer, the raw data being data in which a raw set including a plurality of XP images as a plurality of raw images is associated with a determination result regarding bone dysplasia; The computer, for at least one of the raw images, - extracting a region representing a predetermined site; - generating a modified image by resizing the extracted region to a predetermined size; A step in which a computer generates a trained model by performing machine learning based on a corrected set including the corrected image and corrected data in which the determination result regarding bone dysplasia is associated; Equipped with.

[0017] An example of a method for generating a trained model for determining bone dysplasia according to the present invention is as follows: a step of acquiring raw data by a computer, the raw data being data in which a raw set including a plurality of XP images as a plurality of raw images is associated with a determination result regarding bone dysplasia; generating a modified image by replacing a predetermined region of at least one of the raw images with a predetermined image; A step in which a computer generates the trained model by performing machine learning based on a corrected set including the corrected image and corrected data in which the determination result regarding bone dysplasia is associated; Equipped with. [Effects of the Invention]

[0018] According to the present invention, it is possible to assist doctors by automatically making a determination regarding bone dysplasia.

[0019] Effects other than those mentioned above will be explained by the embodiments and modifications described in this specification and the drawings. [Brief explanation of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram for explaining an overview of a determination system 10 according to a first embodiment. [Figure 2] 2 shows an example of the configuration of the determination system 10 of FIG. 1. [Figure 3] 3 is a flowchart showing an example of a processing flow when the determination system 10 of FIG. 1 makes a determination. [Figure 4] An example of an XP image set containing XP images of different body parts. [Figure 5] Example XP image sets with different shooting directions. [Figure 6] Example XP image sets in different positions. [Figure 7] Examples of XP image sets taken at different times. [Figure 8] An example of an XP image set. [Figure 9]10 is a flowchart showing an example of the processing flow when the determination system 10 according to the second embodiment generates a determination model M. [Figure 10] 10 shows an example of training data T according to the second embodiment. [Figure 11] A diagram explaining an example of a specific learning method using multi-view CNN. [Figure 12] Breakdown of training data used in experiments in embodiment 2. [Figure 13] Results of the experiment using the training data in Figure 12. [Figure 14] 10 is a flowchart showing an example of the processing flow when a determination system 10 according to a first modification of the second embodiment generates a determination model M. [Figure 15] FIG. 10 is a diagram illustrating an example of pre-cropping of a raw image. [Figure 16] 10 is a flowchart showing an example of the processing flow when a determination system 10 according to a second modification of the second embodiment generates a determination model M. [Figure 17] FIG. 10 is a diagram illustrating an example of partial pre-conversion of a raw image. [Figure 18] 11 is an example of a heat map output by the determination system 10 according to the third embodiment. [Figure 19] Another example of a heatmap. DETAILED DESCRIPTION OF THE INVENTION

[0021] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. [Embodiment 1] FIG. 1 is a diagram illustrating an overview of a determination system 10 according to a first embodiment. The determination system 10 is a system that performs a determination regarding osteodystrophy for a patient P. In particular, the determination system 10 acquires multiple XP images of the same patient P, performs a determination based on these XP images, and outputs a determination result. The determination result indicates, for example, whether or not the patient P has developed a specific disease, and a detailed example will be described later.

[0022] "XP image" means, for example, an image taken using XP (X-ray Photograph, also known as plain X-ray photography or radiography) technology, and particularly refers to, but is not limited to, an image that transparently depicts the internal structure of the human body.

[0023] Although there is no restriction on the patient's age, it is preferable to use XP images of patients under the age of three. This is because many patients diagnosed with osteodystrophy are infants and require low radiation exposure, making it particularly beneficial to use XP images for patients under the age of three. Furthermore, it is difficult for patients under the age of three to autonomously stop moving during imaging, but XP images can be taken even when the patient's body cannot be immobilized for long periods of time, and imaging can also be performed while the radiological technologist is immobilizing the patient.

[0024] FIG. 2 shows an example configuration of a determination system 10 according to the first embodiment. The determination system 10 includes a determination server 20, which stores a determination model M for making a determination. The determination model M is, for example, a trained model that has undergone machine learning to enable appropriate determination according to the present embodiment, but is not limited to this. In the present embodiment, the determination system 10 also includes a client terminal 30. The client terminal 30 is communicably connected to the determination server 20 via a communication network N (for example, the Internet).

[0025] The determination server 20 has a hardware configuration as a known computer, and includes, for example, a calculation means 21, a storage means 22, and a communication means 23. The calculation means 21 includes, for example, a processor. The processor can be manufactured using an integrated circuit, an ASIC, an FPGA, or the like. The storage means 22 includes, for example, a storage medium such as a semiconductor memory device or a magnetic disk device. Part or all of the storage medium may be a non-transitory storage medium.

[0026] The communication means 23 includes a communication device such as a network interface. The communication device can function as both an input device and an output device. The communication device can perform wired communication and / or wireless communication.

[0027] Although not shown in FIG. 2, the determination server 20 may further include input devices such as a keyboard and a mouse, and / or output devices such as a display and a printer.

[0028] The storage means 22 stores the above-mentioned determination model M. The storage means 22 may also store a program (not shown). A processor may execute this program, causing the determination server 20 to perform the functions described in this embodiment. That is, this program causes a computer to function as the determination server 20 according to this embodiment.

[0029] The hardware configuration of the client terminal 30 is not particularly illustrated, but may be the same as that of the determination server 20. That is, the client terminal 30 includes a calculation means, a storage means, and a communication means, and may further include an input device and an output device. The storage means may store a program, and the processor of the client terminal 30 may execute the program, causing the client terminal 30 to perform the functions described in this embodiment.

[0030] The operation of the determination system 10 configured as above will be described below.

[0031] FIG. 3 is a flowchart showing an example of the processing flow when the determination system 10 according to the first embodiment makes a determination. This flowchart shows a method for making a determination regarding osteodystrophy. For example, the execution of the processing according to this flowchart is started when a user of the determination system 10 inputs an instruction to start the determination processing from the client terminal 30. In this case, the user may be, for example, a doctor who makes a diagnosis regarding osteodystrophy or a medical professional who assists the doctor, but is not limited to this. Upon receiving an instruction from the user, the client terminal 30 transmits an instruction to start the processing to the determination server 20, thereby starting the processing of FIG. 3.

[0032] The determination server 20 acquires a set of images including one or more XP images (hereinafter referred to as an "XP image set") (step S1). Note that, because osteodystrophy occurs throughout the patient's body, if the XP image set includes multiple XP images of the same patient, the probability of including an XP image showing symptoms increases, improving accuracy. Below, an example of an XP image set that the determination system 10 handles as input will be described.

[0033] FIG. 4 shows an example of an XP image set including XP images of different regions. In this example, the XP image set includes XP images of multiple imaging regions among the head, upper limbs, spine, hands, pelvis, and lower limbs (or among the head, right upper limb, left upper limb, spine, right hand, left hand, pelvis, right lower limb, and left lower limb). That is, the set includes multiple XP images of the head X1, upper limb X2, spine X3, hand X4, pelvis X5, and lower limb X6. Multiple XP images of a single region may be included. The upper limb XP image X2 may be an XP image of the right upper limb, an XP image of the left upper limb, an XP image including both upper limbs, or one or more of these. Similarly, the XP image X4 of the hand can be an XP image of the right hand, an XP image of the left hand, an XP image including both hands, or one or more of these, and the XP image X6 of the lower limbs can be an XP image of the right lower limb, an XP image of the left lower limb, an XP image including both lower limbs, or one or more of these. Note that the illustrations are merely schematic and do not indicate the exact position or range of the imaging region. For example, the XP image X5 of the pelvis does not need to be an XP image of the pelvis alone or an XP image centered on the pelvis, but rather can be an XP image in which the pelvis is visible.

[0034] FIG. 5 shows a schematic example of an XP image set taken from different shooting directions (or fields of view). In this example, the XP image set includes an XP image X7 taken from the front and an XP image X8 taken from the side. The XP image X7 taken from the front can include an XP image taken from a posterior-anterior (PA) shot, an XP image taken from an anterior-posterior (AP) shot, or both. The XP image X8 taken from the side can include an XP image taken from the right side, an XP image taken from the left side, or both. The set may not include the XP image X7 taken from the front (for example, it may consist of only an XP image taken from the right side and an XP image taken from the left side), or it may include multiple XP images from one shooting direction. Note that the illustration is merely a schematic and does not strictly indicate the shooting directions.

[0035] FIG. 6 shows a schematic example of an XP image set for different body positions. In this example, the XP image set includes XP images for multiple body positions from among standing, lying, and sitting. That is, it includes multiple XP images from among standing XP image X9, sitting XP image X10, and lying XP image X11. Note that the illustration is merely schematic and does not strictly represent the body positions.

[0036] 7 shows an example of an XP image set taken at different times. In this example, the XP image set includes an XP image X12 taken at a first time point and an XP image X13 taken at a second time point later than the first time point. That is, these XP images were taken at different times. There is no particular limitation on the interval between the first and second time points, and the respective times can be determined arbitrarily, but may be, for example, 6 months or 12 months.

[0037] XP images taken at different locations, from different angles, in different positions, and at different times may be mixed. For example, an XP image set may include XP images of the pelvis taken from a frontal perspective at a first time point and XP images of the lower extremities taken from a lateral perspective at a second time point.

[0038] An example of an XP image set is shown in Figure 8. This XP image set includes a total of five XP images of the pelvis taken in different positions and from different imaging directions. Naturally, the number of XP images included in the XP image set is not limited to five, and can be arbitrarily designed or determined as long as it is two or more.

[0039] A method for inputting these XP image sets can be designed as appropriate by a person skilled in the art based on known techniques, etc., but an example is shown below. The user may input the XP image set from the client terminal 30. In that case, the client terminal 30 may transmit the XP image set to the determination server 20, which may receive it and acquire the XP image set. Alternatively, the user may input information (image ID, folder name, file name, hyperlink, etc.) for identifying the XP image set (or each XP image constituting the XP image set) from the client terminal 30 to the determination server 20. In that case, the client terminal 30 may transmit this information to the determination server 20, which may receive it and acquire the XP image set based on that information.

[0040] Furthermore, the information for identifying the XP image set may refer to a computer or storage device that is not included in the determination system 10. Examples of such computers include a computer that controls an XP imaging device, a computer that constitutes a PACS (Picture Archiving and Communication System), a computer that stores electronic medical records, etc. The determination server 20 may acquire the XP image set by receiving it from these computers.

[0041] Returning to Fig. 3, after step S1, the determination server 20 makes a determination regarding bone dysplasia using the determination model M based on the XP image set (step S2). A specific method for generating the determination model M can be designed appropriately by a person skilled in the art based on known technology, and a specific example of the generation method will be described later in relation to embodiment 2. A computer other than the determination server 20 may generate the determination model M, and the determination server 20 may acquire the determination model M (or information necessary to construct it) from that computer.

[0042] After step S2, the determination server 20 outputs the determination result (step S3). The format and content of the determination result can be designed arbitrarily, and the configuration of the determination model M and the machine learning process can also be designed appropriately accordingly. A specific example will be described below. As an example, the determination result includes information indicating the presence or absence of dysplasia. Here, "the presence or absence of dysplasia" may be information expressed as whether or not the patient has dysplasia, whether or not the patient has developed dysplasia, whether or not dysplasia is observed in the patient, whether or not dysplasia has manifested in the patient, etc. Furthermore, "the presence or absence of dysplasia" is not limited to information strictly indicating the presence or absence of a disease, but may also be information for assisting in the diagnosis of a disease, such as information indicating whether or not there is a "suspect" of dysplasia.

[0043] As another example, the determination result includes information indicating the strength of suspicion of dysplasia. The strength of suspicion may be a value called degree, likelihood, confidence, or the like. The strength of suspicion may be expressed, for example, as a real number as an inferred value, and the larger the value, the stronger the suspicion that the patient has dysplasia. Furthermore, the determination server 20 may compare this value with a predetermined threshold and determine that dysplasia is present if the value is equal to or greater than the threshold, or that dysplasia is not present if the value is less than the threshold.

[0044] In addition to or instead of the above information, the determination result may include information indicating the presence or absence or the level of suspicion for each specific disease or group of diseases included in osteogenesis imperfecta. The specific disease may include, for example, among osteogenesis imperfecta, a disease that particularly results in short stature. The specific disease may also include, for example, among osteogenesis imperfecta or diseases that particularly result in short stature, achondroplasia syndrome.

[0045] For example, diseases that result in short stature include osteogenesis imperfecta, type II collagen dysplasia, multiple epiphyseal dysplasia, etc. Furthermore, diseases that fall under the category of achondroplasia syndrome include achondroplasia, hypochondroplasia, thanatophoric dysplasia, etc.

[0046] Specific diseases and disease groups may include one or more of the following in addition to or instead of diseases presenting with short stature and / or achondroplasia syndrome, as listed below, with the disease listed in parentheses after the disease group: -FGFR3 chondrodysplasia group (thanatophoric dysplasia, achondroplasia and hypochondroplasia) - Type 2 and Type 11 collagen group (achondroplasia / hypoplasia, congenital spondyloepiphyseal dysplasia, Kniest dysplasia, Czech dysplasia, Stickler syndrome, otospondylomegaly epiphyseal dysplasia) - Sulfation disorder group (osteogenesis imperfecta, skeletal dysplasia) - Perlecan group (segmental dysplasia, Schwartz-Jample syndrome) - Filamin group and related disorders (Melnick-Needles syndrome, otopalatodactyl syndrome, Larsen syndrome) -TRPV4 group (metamorphic dysplasia, TRPV4 disorders other than metamorphic dysplasia) - Trophoblastic dysplasia with major bony changes (chondroectodermal dysplasia (Ellis-van Creveld), short rib polydactyly syndrome, respiratory insufficiency thoracic dysplasia) -Multiple epiphyseal dysplasia and pseudoachondroplasia group (pseudoachondroplasia, multiple epiphyseal dysplasia) - Metaphyseal dysplasia (various types) - Spondylometaphyseal dysplasia (SMD) (endochondrodysplasia, spondylometaphyseal dysplasia) - Spondylo-epiphyseal (metaphyseal) dysplasia (SE(M)D) (Dyggve-Melchior-Clausen dysplasia (DMC), late-onset spondyloephyseal dysplasia X-linked (SED-XL)) - Distal limb dysplasia (trichorhinophalangeal dysplasia, gerophysical dysplasia, acrodysostosis) - Distal mesomelic dysplasia (distal mesomelic dysplasia Maroteaux type (AMDM)) -Mesomeridae / proximal limb mesomeridae dysplasia (dyschondrosteosis (Leri-Weill), mesomeridae dysplasia Langer type, scapular dysplasia) - Flexor dysplasia group (flexor limb dysplasia) - Osteodysplasia with multiple dislocations (Desbuquois dysplasia, spondyloepiphyseal dysplasia with joint laxity (including Hall type and Beighton type)) - Chondrodysplasia punctata group (chondrodysplasia punctata (all types)) -Neonatal osteosclerotic dysplasia (Caffey disease) - Osteopetrosis and related disorders (osteopetrosis, pyknodysostosis) -Other osteosclerotic bone diseases (bone mottle disease, waxy osteosis (melorheostosis), craniometaphyseal dysplasia, diaphyseal dysplasia Camurati-Engelmann disease, dermatoperiosteal hyperplasia (hypertrophic dermatoperiostosis), Pyle disease) - Groups with osteogenesis imperfecta and decreased bone mineral density (osteogenesis imperfecta, Bruck syndrome, Ehlers-Danlos syndrome) - Abnormal bone mineralization group (hypophosphatasia, hypophosphatemic rickets) - Lysosomal storage diseases with bone changes (mucopolysaccharidosis, mucolipidosis) - Osteolysis group (Hajdu-Cheney syndrome, multicentric carpal-tarsal osteolysis (without nephropathy)) - Developmental anomalies of skeletal components (multiple cartilaginous exostoses, polyostotic fibrous dysplasia (McCune-Albright), metachondromatosis, fibrodysplasia ossificans progressiva, neurofibromatosis type 1 (NF1), hemimeliad epiphyseal dysplasia (Trevor), enchondromatosis (Ollier), enchondromatosis with hemangiomas (Maffucci)) - Overgrowth (tall stature) syndromes involving skeletal lesions (Sotos syndrome, Proteus syndrome, Marfan syndrome, congenital contracture arachnodactyly, Loeys-Dietz syndrome) -Hereditary inflammatory / rheumatoid osteoarthropathy (progressive pseudorheumatoid dysplasia) -Cleidocranial dysplasia and related disorders (cleidocranial dysplasia) - Craniosynostosis syndromes (Pfeiffer syndrome, Apert syndrome, Crouzon syndrome, Antley-Bixler syndrome, Shprintzen-Goldberg syndrome) - Dysostosis mainly involving the spine (spondylocostal dysostosis, Klippel-Feil syndrome) -Patellar dysostosis (nail-patella syndrome) - Brachydactyly (all types of brachydactyly, Rubinstein-Taybi syndrome, pseudohypoparathyroidism) -Limb hypoplasia / deficiency group (Poland syndrome) - Polydactyly, syndactyly, and triphalangeal thumb group (hereditary polydactyly and syndactyly) - Joint dysplasia and synostosis (multiple synostosis syndrome)

[0047] The determination result including such information is transmitted from the determination server 20 to the client terminal 30, and the client terminal 30 receives and outputs it (for example, displays it on a screen), thereby allowing the user to know the determination result.

[0048] As described above, the determination system 10 according to the present embodiment can automatically determine osteodystrophy, thereby assisting physicians. For example, even if a physician is not an experienced physician who is thoroughly familiar with osteodystrophy cases, the diagnosis of osteodystrophy can be made more appropriately. Furthermore, even if a final diagnosis is not made, the possibility of osteodystrophy can be screened.

[0049] [Embodiment 2] In the second embodiment, the determination server 20 further has a function of generating a determination model M. That is, the second embodiment relates to a method for generating the determination model M as a trained model. Hereinafter, a determination system according to the second embodiment will be described, but the description of parts common to the first embodiment may be omitted.

[0050] 9 is a flowchart showing an example of a processing flow when the determination system 10 according to the second embodiment generates a determination model M. For example, the execution of the processing according to this flowchart is started when a user of the determination system 10 inputs an instruction to start the processing from the client terminal 30. The user in this case is, for example, a designer or engineer of the determination system 10, but is not limited to this.

[0051] When the client terminal 30 receives an instruction from the user, it transmits an instruction to start processing to the determination server 20, thereby starting the processing of Fig. 9. In the processing of Fig. 9, the determination server 20 first acquires training data to be used for learning (step S11). The format and content of the training data can be appropriately designed by a person skilled in the art depending on the content of the determination (step S2 in Fig. 3), but an example will be described below.

[0052] Figure 10 shows an example of training data T. Training data T is data in which an XP image set XS (which may be called an image bag) is associated with a label set L. Although Figure 10 shows only a single training data T, multiple sets of training data T are used in actual learning processing.

[0053] The XP image set XS is a set including multiple XP images (trainer XP images) related to the same patient, as exemplified in FIGS. 4 to 8. The label set L is a set including one or more labels indicating the determination results related to osteodystrophy, and includes a label indicating whether or not the patient has one or more specific diseases. In the example of FIG. 10, the label set L includes labels for multiple diseases, but the label set L may also include only a single label (for example, a label indicating the presence or absence of osteodystrophy). The label set may also include a label indicating whether or not the patient has a specific disease group, in addition to or instead of a disease.

[0054] 9, the determination server 20 performs machine learning using the training data T (step S12). For example, the training data T is input to an appropriate learning model, and the model is trained so that a label set L is output in response to an input of an XP image set XS. This generates a determination model M.

[0055] The specific learning method in step S12 can be designed by a person skilled in the art, but for example, a multi-view CNN can be used. A multi-view CNN is a CNN (Convolutional Neural Network) that can integrate information from multiple images and make a comprehensive judgment.

[0056] A specific example of a learning method using a multi-view CNN will be described using FIG. 11. The input of the determination model M is an XP image set XS, and the output is a label set L. The determination model M includes CNNs M1 to M3, a view pooling functional unit M4, and a fully connected layer M5. The CNNs M1 to M3 can be configured as known CNNs. The CNNs M1 to M3 may use the same model for all XP images, or may use different CNNs for any or all of the XP images.

[0057] For an XP image input, CNNs M1 to M3 generate and output feature vectors V1 to V3, respectively. All feature vectors V1 to V3 are vectors of the same dimension.

[0058] The view pooling function unit M4 receives multiple input feature vectors V1 to V3 and generates and outputs a multiview feature vector V4. The multiview feature vector V4 is a vector of the same dimensions as the feature vectors V1 to V3, and each element represents the maximum value among the corresponding elements of the feature vectors V1 to V3. In other words, the view pooling function unit M4 can be said to perform max pooling processing. Note that max pooling processing is just an example, and other methods for combining multiple feature vectors, such as average pooling, can also be used.

[0059] The fully connected layer M5 is, for example, a neural network including multiple layers, in which all neurons are connected to neurons in adjacent layers. The fully connected layer M5 generates and outputs a label set L in response to the input of the multi-view feature vector V4.

[0060] In this way, the multi-view feature vector V4 incorporates elements from the feature vectors of different views (different XP images) for each element, integrating information from multiple views. By classifying (labeling) this using the fully connected layer M5, multiple views can be comprehensively judged, improving judgment accuracy.

[0061] 10, the label set L in this embodiment is not associated with each XP image, but is associated collectively with an XP image set XS that includes multiple XP images. When multiple XP images are used, there may be cases where a symptom appears strongly in one XP image (for example, a certain region) while hardly appearing in another XP image (for example, a different region). However, by labeling on an XP image set basis rather than on an XP image basis, learning can be performed appropriately.

[0062] Below, we show an example of the results of an experiment using actual training data to evaluate the accuracy of the classification. In this example, cross-validation was used.

[0063] Figure 12 shows the breakdown of the training data used in the experiment. A total of 180 sets of training data were prepared, including XP image sets related to achondroplasia syndrome (positive) and XP image sets related to other conditions (negative). All training data were images of the pelvis. The presence or absence of osteodystrophy was used as a label, with 25 positive sets (those with osteodystrophy) and 155 negative sets (those without osteodystrophy). Of these, 144 sets (20 positive sets, 124 negative sets) were used for cross-validation, and 36 sets (5 positive sets, 31 negative sets) were used for testing.

[0064] The 144 pairs for cross-validation were divided into five groups to form teacher groups A, B, C, D, and E. In each of the five trials, a different teacher group was used as the validation group, and the remaining teacher group was used as the training group. That is, in the first trial, teacher group A was used as the validation group, and teacher groups B, C, D, and E were used as the training group. Similarly, in the second trial, teacher group B was used as the validation group, and teacher groups A, C, D, and E were used as the training group. In the third trial, teacher group C was used as the validation group, and teacher groups A, B, D, and E were used as the training group. In the fourth trial, teacher group D was used as the validation group, and teacher groups A, B, C, and E were used as the training group. In the fifth trial, teacher group E was used as the validation group, and teacher groups A, B, C, and D were used as the training group. In each trial, evaluation was performed using the validation group and the test group.

[0065] Figure 13 shows the results of an experiment using the training data in Figure 12. AUC (Area Under the ROC Curve) was used as an evaluation index. Methods for calculating AUC are well known to those skilled in the art, but it represents the area under an ROC (Receiver Operating Characteristics) curve, where ROC refers to a curve obtained by graphing the experimental results with the true positive rate on the vertical axis and the false positive rate on the horizontal axis.

[0066] For the five trials, an average AUC of 0.951 was obtained for the validation group and an average AUC of 0.837 was obtained for the test group. The standard deviation of the AUC for the five trials was 0.079 for the validation group and 0.339 for the test group. The average AUC was high for both the validation group and the test group, and each trial showed a high AUC of 0.837 or higher, except for the test group in the third trial.

[0067] In this experiment, the amount of training data was small, and the possibility of overfitting cannot be ruled out. However, if the amount of training data is further increased, it is expected that the accuracy will be sufficiently useful for practical use for an XP image set consisting of general XP images. Furthermore, since bone dysplasia other than achondroplasia syndrome is thought to appear similarly in XP images, by providing appropriate training data, it is expected that the accuracy will be sufficiently useful for practical use for bone dysplasia (or the disease groups and disorders included therein). In particular, since diseases that result in short stature are thought to appear very similar to achondroplasia syndrome in XP images, high accuracy is expected, at least for diseases that result in short stature.

[0068] As a modification of the second embodiment, the training data before learning may be processed. Such a modification will be described below.

[0069] [Modification 1 of Embodiment 2] In Modification 1 of Embodiment 2, pre-trimming of the training data is performed. A determination system according to Modification 1 of Embodiment 2 will be described below, but descriptions of parts common to Embodiment 1 or Embodiment 2 may be omitted.

[0070] 14 is a flowchart showing an example of the flow of processing when the determination system 10 according to the first modification of the second embodiment generates the determination model M. This processing is executed instead of the processing shown in FIG.

[0071] The determination server 20 acquires training data (step S21). This step corresponds to step S11 in embodiment 2 (FIG. 9). The training data acquired in step S21 is, for example, the same as that acquired in step S11, but is in a state before subsequent pre-trimming is performed. In FIG. 14 and the following description, this is referred to as "raw data" to distinguish it from the data after pre-trimming. Similarly, the XP image set included in the training data acquired in step S21 is referred to as a "raw set," and the training XP image is referred to as a "raw image." In other words, the determination server 20 can be said to acquire raw data in which a raw set including multiple raw images is associated with a determination result regarding bone dysplasia.

[0072] Next, the determination server 20 performs pre-trimming by extracting a region representing a predetermined part from the raw image (step S22). For example, a region representing a desired part can be extracted by performing object detection processing on the image. The part to be extracted may be, for example, the head, upper limbs, hands, pelvis, or lower limbs. The object detection processing can be performed by the determination server 20 or another computer using an appropriate trained model, but since this is a well-known technique, detailed description will be omitted.

[0073] An example of pre-trimming of a raw image will be described using Fig. 15. In raw image X20, a region R1 representing the pelvis is extracted. Note that Fig. 15 shows a schematic representation of region R1 representing the pelvis, and the illustrated range of region R1 is not precise.

[0074] Returning to FIG. 14, the determination server 20 then generates a corrected image (corrected teacher XP image) by changing the size of the extracted region to a predetermined size (step S23). This predetermined size may be, for example, a size whose width is the same as the width of the raw image, or a size whose height is the same as the height of the raw image. Furthermore, when extracting the region in step S22, the width / height ratio (aspect ratio) of the extracted region may be set to be the same as the aspect ratio of the original raw image. In the example of FIG. 15, an image consisting only of region R1 representing the pelvis becomes the corrected image.

[0075] Next, the determination server 20 performs machine learning using an XP image set (corrected set) including the corrected image (step S24). That is, a corrected set is generated by replacing each raw image included in the raw set with a corresponding corrected image. Then, machine learning is performed based on the corrected data in which the corrected set and the label of the original raw data (e.g., the determination result for osteodystrophy) are associated, thereby generating a determination model M. This step corresponds to step S12 in embodiment 2 (FIG. 9).

[0076] According to this modified example, it is possible to eliminate bias between positive XP images and negative XP images and further improve the accuracy of the determination. Generally, there is a possibility that the tendency of the photographing range, magnification rate, etc. may differ between positive XP images and negative XP images, and such bias is particularly likely to occur in diseases with few cases (rare diseases). Even in such cases, by performing a trimming process such as this modified example, it is possible to make the photographing range, magnification rate, etc. more uniform for all teacher XP images, thereby further improving the accuracy of the determination.

[0077] In this modification, it is preferable to perform pre-trimming on all raw images, but it may be omitted for some raw images. For example, it may be determined whether a desired part appears at an appropriate size in a raw image, and if it appears at an appropriate size, the raw image may be used for learning as is. That is, the determination server 20 may perform the above-described pre-trimming on at least one raw image. The determination of whether a desired part appears at an appropriate size can be performed, for example, using the object detection process described above.

[0078] Such pre-trimming can also be performed during inference using actual XP images (step S2 in FIG. 3). For example, the determination server 20 extracts an area representing a predetermined site from at least one of the XP images acquired in step S1. Next, the determination server 20 generates a corrected image by changing the size of the extracted area to a predetermined size. Next, the determination server 20 performs a determination regarding bone dysplasia using the determination model M based on the XP image set including the corrected image.

[0079] It is preferable to apply pre-trimming both during learning and inference, but it may be applied only during learning or only during inference.

[0080] [Modification 2 of Embodiment 2] In Modification 2 of Embodiment 2, partial prior replacement of training data is performed. A determination system according to Modification 2 of Embodiment 2 will be described below, but descriptions of parts common to Embodiment 1 or 2 may be omitted.

[0081] 16 is a flowchart showing an example of the flow of processing when the determination system 10 according to the second modification of the second embodiment generates the determination model M. This processing is executed instead of the processing shown in FIG.

[0082] The determination server 20 acquires raw data (step S31). This step may be the same as step S21, for example. Next, the determination server 20 generates a corrected image by replacing a predetermined area of ​​each raw image with a predetermined image (step S32). For example, a predetermined area (e.g., a square area) in the raw image is replaced with a rectangle of a predetermined color (e.g., white or black). That is, all pixels included in the area are colored white or black. This area is preferably adjusted to be, for example, the area of ​​a protective plate used when capturing XP images (or an appropriate area including the protective plate). A fixed area common to all raw images may be determined in advance, or an area may be determined for each raw image using object detection processing, etc. In this way, preliminary partial replacement of training data is performed.

[0083] An example of partial pre-conversion of a raw image will be described using Figure 17. In raw image X21, all pixels in area R2 including the protective plate are changed to black (the corrected image after the change is not shown). Note that Figure 17 shows a schematic diagram of area R2 including the protective plate, and the area of ​​area R2 shown in the figure is not exact.

[0084] 16, next, the determination server 20 performs machine learning using an XP image set (corrected set) including the corrected image (step S33). This step can be the same as step S24, for example. That is, the determination server 20 generates a determination model M by performing machine learning based on the corrected data in which the corrected set including the corrected image and the label (determination result related to bone dysplasia) are associated.

[0085] According to this modified example, it is possible to eliminate bias between positive XP images and negative XP images, thereby improving the accuracy of the determination. Generally, there is a possibility that the shape and size of the protective plates may differ between positive XP images and negative XP images, and such bias is particularly likely to occur in diseases with few cases (rare diseases). Even in such cases, by performing a partial replacement process such as this modified example, the state of the specified area including the protective plates can be made more uniform for all teacher XP images, thereby improving the accuracy of the determination.

[0086] In this modification, it is preferable to perform the preliminary partial replacement for all raw images, but it may be omitted for some raw images. For example, it may be determined whether or not a protective plate appears in a predetermined size and shape in a raw image, and if it appears in the predetermined size and shape, the raw image may be used for learning as is. In other words, the determination server 20 may perform the preliminary partial replacement described above for at least one raw image. The determination of whether or not a protective plate appears in the predetermined size and shape can be performed, for example, using the object detection process described above.

[0087] Such preliminary partial replacement can also be performed during inference using actual XP images (step S2 in FIG. 3). For example, the determination server 20 generates a corrected image by replacing a predetermined region with a predetermined image for at least one of the XP images acquired in step S1. Next, the determination server 20 performs a determination regarding bone dysplasia using the determination model M based on the XP image set including the corrected image.

[0088] Although it is preferable to apply the partial prior replacement both during learning and during inference, it may be applied only during learning or only during inference.

[0089] [Other Modifications of Embodiment 2] The system may be configured to allow manual pre-processing of raw images. For example, a user (e.g., a designer or engineer of the determination system 10) may view the raw images in advance and modify any areas that the user determines may adversely affect learning. For example, for a raw image that shows a part of a human body other than the patient, the area including the human body other than the patient may be deleted, or pre-trimming may be performed to extract only areas that do not include the human body other than the patient. For example, if the hand of a diagnostic radiologist is shown in the image, the area including the hand may be deleted. Regarding such a task, the determination server 20 or the client terminal 30 may execute a process to delete a part of the XP image or trim the XP image in response to a user operation.

[0090] The appropriateness of the training data can be verified manually. For example, the presence or absence of leaks in the training can be checked, and the training data can be adjusted accordingly. This prevents training using data that would otherwise be unavailable at the time of judgment (inference), improving the accuracy of the judgment.

[0091] When conducting experiments to evaluate accuracy, various methods for grouping training data can be designed. For example, grouping may be performed according to the patient's date of birth, or groups may be formed so that XP images of patients with similar dates of birth belong to the same group. Furthermore, when XP images from multiple imaging facilities are mixed, grouping may be performed taking into account variations between the imaging facilities. For example, groups may be formed for each imaging facility, or groups may be formed so that XP images from all imaging facilities are included as evenly as possible in each group.

[0092] To substantially increase the amount of training data, few-shot learning may be employed. For example, a known few-shot learning method can generate one or more additional XP images from a single XP image, thereby increasing the amount of training data and improving the accuracy of the assessment.

[0093] The above-described variations may be combined. For example, a raw image may be subjected to pre-trimming according to Variation 1 and then subjected to pre-substitution according to Variation 2, and the resulting corrected image may be used as training data. The order in which pre-trimming and pre-substitution are performed may be determined arbitrarily.

[0094] [Embodiment 3] In the third embodiment, the determination server 20 in the first embodiment, the second embodiment, or each of the modifications thereof further includes a function of outputting a heat map of the input XP image. Hereinafter, the determination system according to the third embodiment will be described, but the description of parts common to the first embodiment, the second embodiment, or each of the modifications thereof may be omitted.

[0095] Fig. 18 shows an example of a heat map output by the determination system 10 according to the third embodiment. This example corresponds to the XP image set shown in Fig. 8. The heat map is, for example, information indicating the importance of each pixel included in an XP image when a determination is made on the XP image, but is not limited to this.

[0096] For example, if the original XP image is grayscale, a heat map can be generated by coloring each pixel in a color other than gray. Although Figure 16 shows the image in grayscale, any color combination can be used to display the importance.

[0097] The heat map can be generated using, for example, Class Activation Mapping (CAM) technology or Grad-CAM (Gradient-weighted Class Activation Mapping) technology. These are well-known technologies and will not be described in detail here. For example, see the literature "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization" (Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra,<https: / / arxiv.org / abs / 1610.02391> ) A person skilled in the art can configure the determination server 20 to generate an appropriate heat map based on the description in this specification and the above-mentioned documents.

[0098] Although a flowchart according to the third embodiment is not specifically shown, the determination system 10 can output a heat map together with the determination result in step S3 of FIG. 3, or independently thereafter.

[0099] As described above, according to the determination system 10 of the third embodiment, a heat map is output, so that a doctor can refer to the heat map and check important areas in detail, thereby enabling a more appropriate diagnosis.

[0100] Another example of a heat map is shown in Figure 19. This heat map relates to the XP images used in the experiments described in Figures 12 and 13. In Figures 19(a) to 19(e), the XP images are shown on the left and the heat maps are shown on the right. Figures 19(a), 19(b), and 19(c) relate to XP images that were true positives in a certain trial, while Figures 19(d) and 19(e) relate to XP images that were true negatives in the same trial.

[0101] The XP image in FIG. 19(e) includes region R3, which shows the radiological technologist's hands. For such an XP image, a user (e.g., a designer or engineer of the determination system 10) may pre-process the raw data by deleting region R3 (e.g., overwriting it with a predetermined image, as in the protective plate example described in Modification 2 of Embodiment 2) and generating corrected data including the deleted XP image. In particular, if an inappropriate determination result is output during the learning stage, the designer or engineer can refer to the heat map of the training XP image that gave the inappropriate determination result and take measures such as deleting regions that are considered to be strongly related to the inappropriate determination result, thereby improving the determination accuracy. Furthermore, during the inference stage, if a user refers to the heat map and finds that an inappropriate region, such as the radiological technologist's hands, has a high importance, the user can black out the inappropriate region and perform re-inference to obtain an appropriate determination result.

[0102] [Other embodiments] The specific learning method in step S12 is not limited to that described in embodiment 1. For example, multiple instance learning can also be used. Multiple instance learning is a type of weakly supervised learning, and is a learning method that makes it possible to determine a positive result when at least one positive image is included among multiple images.

[0103] To explain a specific example, each XP image included in the XP image set XS is input to the model, and each label is output. If a label is positive for any XP image (e.g., disease A: present), that label is included in the output label set L as positive. On the other hand, if the labels for all XP images are negative (e.g., disease A: absent), that label is included in the output label set L as negative.

[0104] An example of a specific technique for such multiple instance learning is described in the document "Multiple Instance Learning: Algorithms and Applications" (Boris Babenko, Dept. of Computer Science and Engineering, University of California, San Diego). A person skilled in the art can configure the decision server 20 to appropriately generate the decision model M based on the description in this specification and the above document.

[0105] Those skilled in the art can add, modify, or delete components as desired within the scope of the present invention in each of the above-described embodiments and modifications. For example, the determination system 10 can be configured only with the determination server 20. In this case, the determination server 20 can be designed to also function as the client terminal 30.

[0106] Furthermore, even if a client terminal 30 exists, it can be interpreted that the client terminal 30 is not included in the determination system 10. In that case, the client terminal 30 can be said to be used as an operation terminal provided outside the determination system 10 to use or control the determination system 10. [Explanation of symbols]

[0107] 10. Judging system 20...Judgment server 30...Client terminal L...Label set M... Decision model (trained model) (M1~M3...CNN, M4...view pooling function part, M5...fully connected layer) N: Communication network P…Patient T...teaching data R1: Area representing the pelvis (area representing a specific part) R2: Area including protective plates (predetermined area) R3: Area showing the radiological technologist's hands V1~V3...Feature vectors V4: Multi-view feature vector X1~X13...XP images X20, X21...Raw images

Claims

1. A system for making a determination regarding bone dysplasia, obtaining an XP image set including multiple XP images relating to the same patient; A judgment regarding bone dysplasia is made using the trained model based on the XP image set. system.

2. The determination is a determination regarding a disease causing short stature. The system of claim 1 .

3. The diagnosis is a diagnosis regarding achondroplasia syndrome. The system of claim 2 .

4. The system of claim 1 , wherein the plurality of XP images are of patients 3 years of age or younger.

5. The XP image set is Includes XP images of multiple imaging sites from the head, upper limbs, spine, hands, pelvis, and lower limbs, Includes XP images taken from the front and XP images taken from the side, XP images in multiple positions among standing, lying, and sitting, or XP images taken at different times are included. The system of claim 1 .

6. The system of claim 1 , wherein the system outputs a heat map for each of the plurality of XP images, the heat map representing the importance of each pixel to the determination.

7. The system, for at least one of the plurality of XP images, - extracting a region representing a predetermined site; - generating a modified image by resizing the extracted region to a predetermined size; making a judgment regarding bone dysplasia using the trained model based on an XP image set including the corrected image; The system of claim 1 .

8. The system generates a modified image by replacing a predetermined region of at least one of the plurality of XP images with a predetermined image; making a judgment regarding bone dysplasia using the trained model based on an XP image set including the corrected image; The system of claim 1 .

9. A program that causes a computer to function as the system according to claim 1.

10. A method for determining bone dysplasia, comprising: acquiring, by a computer, an XP image set including a plurality of XP images relating to the same patient; A step in which a computer makes a judgment regarding bone dysplasia using a trained model based on the XP image set; A method comprising:

11. A method for generating a trained model for making a judgment regarding bone dysplasia, a step of acquiring raw data by a computer, the raw data being data in which a raw set including a plurality of XP images as a plurality of raw images is associated with a determination result regarding bone dysplasia; The computer, for at least one of the raw images, - extracting a region representing a predetermined site; - generating a modified image by resizing the extracted region to a predetermined size; A step in which a computer performs machine learning based on corrected data in which a corrected set including the corrected image and the determination result regarding bone dysplasia are associated, thereby generating a trained model; A method for generating a trained model comprising:

12. A method for generating a trained model for making a judgment regarding bone dysplasia, a step of acquiring raw data by a computer, the raw data being data in which a raw set including a plurality of XP images as a plurality of raw images is associated with a determination result regarding bone dysplasia; generating a modified image by replacing a predetermined region of at least one of the raw images with a predetermined image; A step in which a computer generates the trained model by performing machine learning based on a corrected set including the corrected image and corrected data in which the determination result regarding bone dysplasia is associated; A method for generating a trained model comprising:

Images