Method for outputting bone age information

Abstract

Disclosed is a method, according to various embodiments, by which a computing device: inputs a bone image into a bone age prediction model so as to acquire, from the bone age prediction model, prediction information including predicted bone age, information about at least one region-of-interest, and opinion information; generates, on the basis of the predicted bone age, reference image information including at least one reference bone image retrieved from a reference image database; and outputs bone age information on the basis of the prediction information and the reference image information.

Description

How to output bone age information

[0001] This relates to a method for outputting bone age information and a device therefor.

[0002] Assessing bone age is a common pediatric procedure. X-rays are taken of a child's left hand, and bone age is assessed based on the image. Bone age assessment is an important indicator of normal development in children. If there is a significant discrepancy between the chronological age and the measured bone age, it can be considered a sign of abnormal bone growth (such as diabetes or a genetic disorder), allowing for appropriate treatment.

[0003] Common methods for assessing bone age from X-ray images of bones in the human body include the GP method proposed by Greulich and Pyle (Greulich, William Walter, and Sarah Idell Pyle. Radiographic atlas of skeletal development of the hand and wrist. Stanford university press, 1959), the TW (or TW1, TW2, TW3) method by Tanner and Whitehouse, the Sauvegrain method, and the Risser sign method.

[0004] Among these, the GP method is a method of assessing / calculating bone age based on a book containing X-ray images of the Atlas pattern group, which organizes X-ray images of the left hand, etc. according to age and sex. In this method, the doctor evaluates the bone age by finding the image that most closely resembles the overall shape of the child's X-ray image. Currently, 76% of doctors who assess bone age use this method. However, the GP method is very likely to have errors in the process of assessing bone age due to the difference in the doctor's skill level and the degree of bone growth between people.

[0005] The problem to be solved is to provide a method and a device for improving the reliability of bone age predicted by an artificial neural network-based bone age prediction model for bone images.

[0006] The technical challenges are not limited to the technical challenges mentioned above, and other technical challenges not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below.

[0007] A method for a computing device to output bone age information according to one aspect may include: inputting a bone image into a bone age prediction model to obtain prediction information including a predicted bone age, finding information, and information about at least one region of interest included in the bone image from the bone age prediction model; retrieving at least one reference bone image based on the predicted bone age from a reference image database, and generating reference image information including the retrieved at least one reference bone image; and outputting bone age information for the bone image based on the prediction information and the reference image information.

[0008] Alternatively, the at least one reference bone image is characterized in that it comprises at least one of a first reference bone image for a bone age earlier than the predicted bone age, a second reference bone image for the predicted bone age, and a third reference bone image for a bone age later than the predicted bone age.

[0009] Alternatively, the method further comprises a step of extracting at least one sub-reference goal image corresponding to the at least one region of interest from the at least one reference goal image; and a step of generating sub-reference image information including the at least one sub-reference goal image.

[0010] Alternatively, the method further comprises: receiving a user input for a specific region of interest among the at least one region of interest; and outputting a sub-reference goal image for the specific region of interest among the at least one sub-reference goal images in response to the user input.

[0011] Alternatively, the sub-reference goal image is characterized in that it includes at least one of a first sub-reference goal image for a bone age earlier than the predicted goal age, a second sub-reference goal image for the predicted goal age, and a third sub-reference goal image for a bone age later than the predicted goal age.

[0012] Alternatively, the opinion information is characterized in that it includes at least one opinion related to each of the at least one region of interest among the opinions on the predicted bone age.

[0013] Alternatively, the bone age information is characterized in that it includes at least one of an output bone image indicating at least one region of interest in the bone image, the predicted bone age, and the reference image information.

[0014] Alternatively, the at least one region of interest is characterized as being a portion of the bone image that influences bone age prediction.

[0015] A computing device for outputting bone age information according to another aspect includes a communication unit connected to external devices; a display; and a processor connected to the communication unit and the display, wherein the processor inputs a bone image into a bone age prediction model to obtain prediction information including a predicted bone age, information on at least one region of interest, and finding information from the bone age prediction model, searches for a reference bone image for at least one bone age based on the predicted bone age from a reference image database, generates reference image information including the searched reference bone image, and outputs bone age information for the bone image on the display based on the prediction information and the reference image information.

[0016] According to various embodiments, a bone age prediction model based on an artificial neural network can significantly improve the explanatory power and reliability of the predicted bone age predicted by the bone age prediction model by additionally providing additional information related to the prediction of the predicted bone age in addition to the information about the predicted bone age for the input bone image.

[0017] The effects that can be obtained in various embodiments are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the description below.

[0018] The drawings attached to this specification are intended to provide an understanding of the present invention, illustrate various embodiments of the present invention, and together with the description of the specification serve to explain the principles of the present invention.

[0019] Figure 1 is a diagram explaining the structure of a CNN, an artificial neural network.

[0020] Figure 2 illustrates a computing device that trains a classification / prediction model based on an artificial neural network.

[0021] Figure 3 is a diagram illustrating a method for a computing device to train a bone age prediction model.

[0022] FIG. 4 is a diagram illustrating a method for a computing device to generate reference image information based on prediction information of a bone age prediction model.

[0023] FIGS. 5 to 11 are diagrams for explaining a method for a computing device to output bone age information based on prediction information and reference image information.

[0024] FIG. 12 is a diagram illustrating a method for a computing device to output bone age information using a bone age prediction model.

[0025] The detailed description of the present invention, which follows, refers to the accompanying drawings, which illustrate specific embodiments in which the present invention may be implemented, to clearly illustrate the purposes, technical solutions, and advantages of the present invention. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention.

[0026] The term "image" or "image data" as used throughout the detailed description and claims of this specification refers to multidimensional data comprised of discrete image elements (e.g., pixels in a two-dimensional image, voxels in a three-dimensional image).

[0027] For example, an "image" may mean a two-dimensional image corresponding to a slide of a given tissue observed using a microscope, but the "image" is not limited thereto, and may be a medical image of a subject collected by (cone-beam) computed tomography, magnetic resonance imaging (MRI), ultrasound, or any other medical imaging system known in the art. An image may also be provided in a non-medical context, for example, a remote sensing system, an electron microscopy, etc.

[0028] Throughout the detailed description and claims of this specification, the term “image” refers to a visible image (e.g., displayed on a screen) or a digital representation of an image.

[0029] For convenience of explanation, the drawings provided herein illustrate slide image data as exemplary image modalities. However, those skilled in the art will appreciate that the image modalities utilized in various embodiments of the present invention include, but are not limited to, the exemplary modalities listed herein, including X-ray images, MRI, CT, PET (positron emission tomography), PET-CT, SPECT, SPECT-CT, MR-PET, 3D ultrasound images, and the like.

[0030] Medical images described throughout the detailed description and claims of this specification may comply with the DICOM (Digital Imaging and Communications in Medicine) standard. The DICOM standard is a general term for several standards used for digital image representation and communication in medical devices, and the DICOM standard is published by a joint committee formed by the American College of Radiology (ACR) and the National Electrical Manufacturers Association (NEMA).

[0031] In addition, the medical images described throughout the detailed description and claims of this specification may be stored or transmitted through a 'Picture Archiving and Communication System (PACS)', and the medical image storage and transmission system may be a system that stores, processes, and transmits medical images in accordance with the DICOM standard. Medical images acquired using digital medical imaging equipment such as X-rays, CTs, and MRIs are stored in DICOM format and can be transmitted to terminals inside and outside the hospital via a network, and observation results and treatment records may be added to these.

[0032] And throughout the detailed description and claims of this specification, the term "learning" or "learning" refers to performing machine learning through procedural computing, and is not intended to refer to mental operations such as human educational activities, and the term "training" is used in a generally accepted sense with respect to machine learning. For example, "deep learning" refers to machine learning using a deep artificial neural network. A deep neural network is a machine learning model that automatically learns the characteristics of each data by training a large amount of data in a structure composed of multi-layer artificial neural networks, and thereby performs learning in a way that minimizes the error of the objective / loss function, that is, classification accuracy, and can extract and classify features at various levels, from low-level features such as points, lines, and planes to complex and meaningful high-level features.

[0033] Throughout the detailed description and claims of this specification, the word "comprise" and variations thereof are not intended to exclude other technical features, additives, components, or steps. Furthermore, "a" or "an" means one or more, and "another" is limited to at least a second or more.

[0034] Other objects, advantages, and features of the present invention will become apparent to those skilled in the art, partly from this specification, and partly from practice of the invention. The examples and drawings below are provided for illustrative purposes and are not intended to limit the present invention. Accordingly, the details disclosed herein regarding specific structures or functions should not be construed in a limiting sense, but rather as representative basic material that provides guidance to those skilled in the art to variously implement the present invention using any suitable detailed structures.

[0035] Furthermore, the present invention encompasses all possible combinations of the embodiments set forth herein. It should be understood that the various embodiments of the present invention, while different, are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the present invention. Furthermore, it should be understood that the positions or arrangements of individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the present invention. Accordingly, the following detailed description is not intended to be limiting, and the scope of the present invention is defined only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly described. Like reference numerals in the drawings designate the same or similar functions throughout the several aspects.

[0036] Unless otherwise indicated or clearly contradicted by context, items referred to in the singular encompass the plural unless the context otherwise requires. Furthermore, when describing the present invention, detailed descriptions of known components or functions will be omitted if they are deemed to obscure the gist of the present invention.

[0037] Hereinafter, in order to enable those skilled in the art to easily practice the present invention, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

[0038] Figure 1 is a diagram explaining the structure of a CNN, an artificial neural network.

[0039] A Convolutional Neural Network (CNN) is a type of Artificial Neural Network (ANN) that utilizes convolution operations. To extract image features, a CNN calculates convolutions by traversing an input image (or input data) through filters, and can use the results of these convolutions to generate feature maps or activation maps.

[0040] The above CNN maintains the shape of the input / output data of each layer, effectively recognizes features from adjacent images while maintaining the spatial information of the image, can extract and learn features of the image using multiple filters, can optionally include a pooling layer that collects and strengthens the features of the extracted image, and has the advantage of having very few learning parameters compared to a general artificial neural network because it uses the filter as a shared parameter.

[0041] Specifically, referring to FIG. 1, the CNN may include a feature extraction region that extracts features from an input image (or input data) and an image classification region that classifies the extracted features. The feature extraction region may include at least one convolutional layer that finds features of an image while minimizing the number of shared parameters using a filter. Alternatively, the feature extraction region may further include at least one pooling layer that enhances and aggregates the features. That is, the pooling layer may be omitted.

[0042] The convolution layer is a layer that applies a filter to an input image (or input data) and then reflects an activation function. One or more filters may be applied to the input image fed into the convolution layer. The single filter may constitute a channel of the feature map. For example, when n filters are applied to the convolution layer, the feature map or the activation map (or output data) will have n channels.

[0043] The above-mentioned pooling layer is an optional layer located after the above-mentioned convolutional layer. The above-mentioned pooling layer can be used as a layer to reduce the size of the output data or to emphasize specific data by receiving the output data of the convolutional layer as input. Methods for processing the above-mentioned pooling layer include max pooling, average pooling, and min pooling. The pooling layer has no learning target parameters, can reduce the size of the matrix, and does not change the number of channels.

[0044] The above CNN can adjust the shape or size of the output data according to the filter size, stride, whether padding is applied, and max pooling size, and can determine the channels through the number of filters.

[0045] The above FC layer is a layer for recognition and classification operations, and is a pre-connected layer used for connecting each layer in a conventional neural network. The FC layer can perform a one-dimensional flattening operation on the two-dimensional array form of the output data in the feature extraction area. The one-dimensional flattened output data can be classified using the SoftMAx function.

[0046] A CNN-based model (hereinafter, a classification model) like this can be trained to perform a predetermined diagnosis or reading from medical image data such as MRI, CT, etc. For example, the CNN-based model can be trained by adjusting parameter values ​​for the above-described layers so that the difference between the classification information / output information for the image data and the label information is minimized using training data including the image data and label information for the image data. Hereinafter, a method for training an artificial neural network-based classification model to perform a specific classification, prediction, and / or diagnosis (e.g., Nodule detection, FPR, Classification, Segmentation) based on medical images, which are medical image data such as MRI, CT, and X-ray, will be described in detail.

[0047] Figure 2 illustrates a computing device that trains a classification / prediction model based on an artificial neural network.

[0048] Referring to FIG. 2, a computing device (20) includes a communication unit (21) and a processor (22), and can directly or indirectly communicate with an external computing device (not shown) through the communication unit (21). Here, the communication unit (21) may correspond to or include a transceiver capable of transmitting and receiving requests and responses with another computing device.

[0049] Specifically, the computing device (20) may achieve desired system performance by using a combination of typical computer hardware (e.g., devices that may include computer processors, memory, storage, input devices and output devices, and other components of conventional image processing devices; electronic communication devices such as routers, switches, etc.; electronic information storage systems such as network-attached storage (NAS) and storage area networks (SAN)) and computer software (i.e., instructions that cause the computing device to function in a specific manner).

[0050] Such a communication unit (21) can transmit and receive requests and responses with other linked image processing devices. As an example, such requests and responses may be made through the same TCP (transmission control protocol) session, but are not limited thereto. For example, they may be transmitted and received as UDP (user datagram protocol) datagrams. In addition, in a broad sense, the communication unit (21) may include a keyboard, a pointing device such as a mouse, other external input devices, a printer, a display, or other external output devices for transmitting commands or instructions.

[0051] In addition, the processor (22) of the computing device (20) may include hardware configurations such as a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU) or a tensor processing unit (TPU), a cache memory, and a data bus. In addition, the processor (22) may further include software configurations of an operating system and an application that performs a specific purpose. The processor (22) may execute instructions for performing the functions of a neural network described below.

[0052] The processor (22) can train the classification / prediction model based on the artificial neural network to perform a predetermined task. For example, the processor (22) can train the classification / prediction model by adjusting the parameters of the classification / prediction model or an encoder included in the classification / prediction model through training data including medical images and label information corresponding to each medical image so that the difference between the label information and the output information is minimized. Here, the classification / prediction model can be based on an artificial neural network capable of extracting features or feature vectors from the medical images. For example, the artificial neural network can be at least one of various types of neural networks such as a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), Long Short-Term Memory models (LSTM), a Bidirectional Recurrent Deep Neural Network (BRDNN), and Convolutional Neural Networks (CNN).

[0053] According to one example, the processor (22) may train a bone age prediction model, which is a classification / prediction model that predicts bone age, based on a bone image (e.g., a bone image of a hand of a subject or a bone image of an elbow of a subject) of a medical image subject (or patient). For example, the processor (22) may train the bone age prediction model to extract features based on a CNN for the input bone image and to predict the bone age of the subject based on the extracted features. In addition, the processor (22) may train the bone age prediction model to output a region of interest (ROI), which is a region that played an important role in calculating the bone age, based on the features extracted from the bone image. In this case, the trained bone age prediction model may output a predicted bone age based on the input bone image of the subject and information on a region of interest that had a major influence on bone age prediction in the bone image.

[0054] Meanwhile, due to the advancement of the classification / prediction model based on the artificial neural network, it may be difficult to explain how the prediction value of the classification / prediction model was derived. Furthermore, the classification / prediction model based on the artificial neural network may have limitations in ensuring the reliability of medical professionals for the output values, such as bone age, due to the existence of black box areas, such as hidden layers. For example, even if the bone age prediction model additionally provides information, such as a region of interest related to bone age prediction, if it does not provide an explanation as to which part of the region of interest in the bone image influenced the bone age prediction, the reliability of the bone age prediction may not be high.

[0055] Therefore, in order to improve the reliability of the classification / prediction information (i.e., predicted bone age) of the bone age prediction model, the computing device needs to additionally provide information on not only the region of interest that had a major influence on the prediction of bone age, but also information on how the region of interest had an influence on the prediction of bone age. In addition, in order to further improve the reliability of the bone age prediction, the computing device may also provide a standard / reference bone image related to the prediction of bone age and standard / reference bone images for bone ages before and after the predicted bone age. While providing the predicted bone age and information on the region of interest, it is necessary to additionally provide the predicted bone age and related finding information.

[0056] For example, the processor (22) may input a bone image into a bone age prediction model to obtain prediction information including a predicted bone age, information on a region of interest, and finding information from the bone age prediction model, generate reference image information including reference bone images retrieved from a reference image database based on the predicted bone age, and output bone age information on the display based on the prediction information and the reference image information. At this time, the reference image information may include reference bone images for different bone ages. Alternatively, a non-transitory computer-readable storage medium recording a program / command for performing the operation of the above-described processor (22) may be configured.

[0057] Hereinafter, a method for providing additional information, such as information on a region of interest, finding information, and reference bone images, to improve the reliability of the predicted bone age for a bone image of the subject using the bone age prediction model by a computing device (20) is described in detail.

[0058] Figure 3 is a diagram illustrating a method for a computing device to train a bone age prediction model.

[0059] The computing device can train the bone age prediction model so that it can additionally provide not only a predicted bone age for the input bone image, but also at least one region of interest, which is a portion of the bone image that influenced the prediction of the bone age, and finding information, as described above.

[0060] Referring to FIG. 3 (a), the computing device can train the bone age prediction model (210) to predict bone age based on an input bone image, and to extract a region of interest (ROI), which is at least one partial region that has a major influence on bone age prediction, from the bone image. Meanwhile, the bone image may be an image of a wrist bone corresponding to the left hand of the subject or an image taken of the elbow bone of the subject.

[0061] Specifically, the computing device can train the bone age prediction model (210) to predict bone age, etc., using learning data (X). Here, the learning data (X) for training the bone age prediction model (210) may include a plurality of learning bone images, and labeled bone age information (or label information) for each of the plurality of learning bone images. In this case, the computing device can input each of the plurality of learning bone images into the bone age prediction model (210), and train the bone age prediction model (210) so that the difference (or loss function) between the output value of the bone age prediction model (210) and the bone age according to the label information is minimized. For example, the computing device can train the bone age prediction model (210) by adjusting parameters of the bone age prediction model (210) (e.g., parameters of layers described with reference to FIG. 1) so that the difference between the output value and the label information is minimized. In addition, the computing device can extract at least one region of interest (ROI), which is a region that has a major influence on bone age prediction, from the bone image by performing segmentation on the bone image, and train a bone age prediction model (210) to output information on the at least one region of interest.

[0062] Meanwhile, the learning data (X) may include, but is not limited to, GP data, TW data, Sauvegrain data, Risser sign data based on pelvic bone images (or GP database, TW database, Sauvegrain database, Risser sign database based on pelvic bone images) used to predict bone age. Here, the GP data may include an atlas (or GP atlas) of sample carpal bones images (hereinafter, reference carpal bones images) that serve as references for each bone age based on the GP method, the degree of ossification (a phenomenon in which bones harden) of the carpal bones and the detailed bones constituting the carpal bones according to the bone age, and bone age-specific finding information on the change pattern of the detailed bones (between) according to the bone age. Specifically, the GP data may include reference images of the wrist bones (or GP atlas) that serve as a reference and standard for the wrist bones listed by age group, and finding information related to the judgment of bone maturity by bone age (see Greulich, William Walter, and Sarah Idell Pyle. Radiographic atlas of skeletal development of the hand and wrist. Stanford university press, 1959).

[0063] The Sauvegrain method is a method of determining bone age based on the results of summing up scores assigned based on the shapes of four or more identified areas, which are determined by checking the shapes of the humerus, lateral condyle and epicondyle, and trochlea in a frontal image of the elbow, and checking the shapes of the olecranon and radial epiphysis in a lateral image of the elbow. The Sauvegrain data or the Sauvegrain database may include Sauvegrain finding information and a Sauvegrain atlas (or, Sauvegrain standard bone image, Sauvegrain reference bone image), which are finding information based on the Sauvegrain method.

[0064] The above TW method is a method of measuring bone age by grading the bones of the wrist and hand according to the bone maturity of each part in the TW atlas and adding up the scores accordingly. The above TW method may include the TW1 method, which is a bone age measurement method developed by Tanner, Whitehouse, and Haley in the UK, the TW2 method which is an improvement on the TW1 method, and the TW3 method which is a further improvement on the TW2 method. The above TW data may include standard findings based on the TW method or TW atlases (TW standard bone images, TW reference bone images). The above TW data may include standard findings based on the TW method or TW atlases (TW standard bone images, TW reference bone images).

[0065] Meanwhile, the Risser sign method may be a method of predicting bone age by dividing the degree of bone growth into five stages based on the maturity of the pelvis or iliac crest (the upper curve of the broad butterfly-shaped bone that makes up the hip). The Risser sign data may include finding information for each stage according to the Risser sign method and a Risser sign atlas for each stage (Risser sign standard bone image, Risser sign reference bone image).

[0066] For example, the bone age prediction model (210) may include at least one of a first sub-model learned based on GP data, a second sub-model learned based on the Sauvegrain data, a third sub-model learned based on the TW data, and a fourth sub-model learned based on the Risser sign data.

[0067] Hereinafter, for convenience of explanation, the learning method of the bone age prediction model (210) and / or the bone age prediction method based on the GP data will be mainly described, but as described above, a method of predicting bone age based on Sauvegrain data, Risser sign data and / or the TW data using the bone age prediction model (210) using at least one of the first to fourth sub-models may also be included in the scope of the proposed invention.

[0068] Specifically, the GP method is a method of predicting the bone age of a wrist image (e.g., an X-ray image of the left hand and wrist) for which bone age is required based on a GP atlas of reference wrist (or wrist) images for each pre-configured bone age and finding information for each wrist image. More specifically, the GP method is a method of predicting bone age through comparison between a reference wrist image included in the GP atlas and a wrist image for which bone age is required. The GP atlas may include at least one sample X-ray image for each bone age (at intervals of 6 months or 1 year). In this case, the bone age or bone maturity of the subject may be predicted through comparison of the wrist image of the subject with the GP atlas.

[0069] For example, the bone age prediction model (210) can be trained to predict the bone age for a bone image input to the model based on training data (X) including the GP data. In this case, the bone age prediction model (210) can be trained to reflect a correlation or correlation coefficient between a reference bone image included in the GP data and a bone age corresponding to each reference bone image (or reference carpal image). In addition, the bone age prediction model (210) can be trained to select / extract at least one region of interest that has a major influence on the prediction of bone age from the GP atlas based on the finding information included in the GP atlas. For example, the GP data can include finding information on whether bone age was predicted based on a specific bone part for each of the reference carpal images included in the GP atlas. In this case, the computing device can train the bone age prediction model (210) so that at least one region of interest (at least one ROI) predicted or extracted by the bone age prediction model (210) as having a major influence on the prediction of bone age and some region of the bone image presented in the finding information (including the GP data or learning data) can correspond to each other.

[0070] Referring to FIG. 3 (b), the computing device can train the bone age prediction model (220) to additionally output finding information related to the predicted bone age for the input bone image.

[0071] Specifically, the computing device can train the bone age prediction model (220) to additionally output finding information related to the predicted bone age predicted by the bone age prediction model (220) using the learning data (X) described above. As described above, the learning data (X) can include information on findings describing the osteophyte phenomenon and appearance of detailed bones related to the prediction of bone age in the reference bone image by bone age.

[0072] For example, the computing device can input a learning bone image included in learning data (X) into a bone age prediction model (220), and train the bone age prediction model (220) so that the difference between the finding information output by the bone age prediction model (220) and the finding information corresponding to the bone age included in the learning data (X) is minimized.

[0073] Alternatively, the bone age prediction model (220) can predict bone age for the input learning bone image and extract a region of interest, thereby constructing a candidate finding set for findings related to the predicted bone age among the finding information for each bone age included in the learning data (X), and can be trained to select at least one finding among the findings included in the candidate finding set based on the region of interest to determine a final finding. For example, the bone age prediction model (220) can be trained to select a finding for a portion of the bone image (e.g., a detailed bone region) corresponding to the at least one region of interest among the findings included in the candidate finding set. In this case, the bone age prediction model (220) can generate finding information for the finding selected based on the at least one region of interest extracted from the candidate finding set. In addition, when the region of interest is plural, the bone age prediction model (220) can also be trained to output finding information including a finding for each of the regions of interest.

[0074] Alternatively, the at least one region of interest may be grouped into at least one region of interest group based on the finding information. For example, if a single finding is related to two or more regions of interest, the two or more regions of interest may be grouped into one region of interest group. Alternatively, if the finding information for two or more regions of interest is similar, the two or more regions of interest may be grouped into one region of interest group. For example, if the finding information for the two or more regions of interest indicates similar bone change patterns, the two or more regions of interest may be grouped into one region of interest group. In this case, the finding information may be mapped to each region of interest group and / or region of interest.

[0075] In this way, the bone age prediction model (220) can output prediction information including a predicted bone age for an input bone image, information on at least one region of interest that has a major influence on bone age prediction, and the above-mentioned finding information.

[0076] Hereinafter, a method is described in which the computing device additionally provides reference bone images related to the predicted bone age based on prediction information obtained from the bone age prediction model.

[0077] FIG. 4 is a diagram illustrating a method for a computing device to generate reference image information based on prediction information of a bone age prediction model.

[0078] Referring to FIG. 4, the computing device can input a bone image (I) of a subject into the learned bone age prediction model (210, 220) as described in FIG. 3. The computing device can obtain prediction information including a predicted bone age predicted for the bone image (I) from the bone age prediction model (210, 220), information on at least one region of interest, and finding information (including a finding corresponding to each region of interest). Here, the finding information may be provided in text form and may include at least one finding related to the GP method, the TW method, the Risser sign method, and / or the Sauvegrain method. Meanwhile, in FIG. 4, the computing device is illustrated as including a reference image information generator (230), but this is for convenience of explanation, and operations corresponding to the reference image information generator (230) may be a function of the computing device. That is, operations of the reference image information generator (230) may be operations of the computing device.

[0079] The computing device may input information on the predicted goal age and / or at least one region of interest included in the predicted information into a reference image information generator (230), and obtain reference image information from the reference image information generator (230). The reference image information generator (230) may search for at least one reference goal image related to the predicted goal age from a reference image database based on the predicted goal age and / or the region of interest. Here, the reference image database may include reference goal images by bone age included in the above-described GP atlas, and / or separately configured reference goal images that are identical / similar to the reference goal images by bone age included in the GP atlas (i.e., reference goal images that are similar by a predefined degree of similarity or higher).

[0080] Specifically, the reference image information generator (230) can retrieve a reference bone image for at least one bone age based on the predicted bone age, and generate reference image information including the reference bone image. For example, the reference image information generator (230) can retrieve a first reference bone image for a bone age earlier than the predicted bone age, a second reference bone image for the predicted bone age, and a third reference bone image for a bone age later than the predicted bone age from the reference image database (see FIG. 6 to be described later). That is, the reference image information generator (230) can retrieve reference bone images for different bone ages based on the predicted bone age. The reference image information generator (230) can generate reference image information including the retrieved at least one reference bone image.

[0081] Alternatively, the reference image information generator (230) may further generate sub-reference image information by additionally considering information about the at least one region of interest. Specifically, the reference image information generator (230) may extract sub-reference bone images for a portion of the region of interest from the reference bone image (see FIG. 7). The reference image information generator (230) may generate first sub-reference image information including sub-reference bone images extracted from the reference bone image. In this case, like the above-described reference bone image, the sub-reference bone images may be at least one sub-reference bone image for at least one different bone age for the region of interest. For example, the first sub-reference image information may include, for each of the at least one region of interest, a first sub-reference bone image extracted from the first reference bone image for a bone age earlier than the predicted bone age, a second sub-reference bone image extracted from a second reference bone image for the predicted bone age, and a third sub-reference bone image extracted from a third reference bone image for a bone age later than the predicted bone age.

[0082] Meanwhile, in another embodiment, the reference image information generator (230) may extract the corresponding sub-reference goal image for each region of interest (for each region of interest). For example, when the at least one region of interest includes a first region of interest and a second region of interest, the reference image information generator (230) may extract at least one sub-goal image corresponding to the first region of interest from the reference goal image, and may extract at least one sub-goal image corresponding to the second region of interest. In this case, the reference image information generator (230) may generate the first sub-reference image information including at least one sub-goal image corresponding to the first region of interest and at least one sub-goal image corresponding to the second region of interest. As described above, each of the sub-goal image corresponding to the first region of interest and the sub-goal image corresponding to the second region of interest may include at least one sub-goal image corresponding to at least one bone age.

[0083] Alternatively, the computing device or bone age prediction model (210, 220) may group the region of interest into at least one region of interest group based on the finding information as described with reference to FIG. 3. In this case, the reference image information generator (230) may receive information (or, region of interest information) on the at least one region of interest group, and additionally generate second sub-reference image information based on the at least one region of interest group. For example, the reference image information generator (230) may extract a group sub-goal reference image for a portion of the region corresponding to each of the at least one region of interest groups from the reference bone image, and generate second sub-reference image information including the group sub-goal reference image for each region of interest group (see FIG. 9).

[0084] For example, the reference image information generator (230) can generate the second sub-reference image information based on the first region of interest group and the second region of interest group that group the at least one region of interest. The reference image information generator (230) can extract a first group sub-goal reference image for a portion of the reference bone image that includes all of the first region of interest group from the reference bone image included in the reference image database, and can extract a second group sub-goal reference image for a portion of the reference bone image that includes all of the second region of interest group. The reference image information generator (230) can generate second sub-reference image information that includes the first group sub-goal reference image extracted for the first region of interest group and the second group sub-goal reference image extracted for the second region of interest group. In this way, the reference image information generator (230) can generate the reference image information, the first sub-reference image information, and / or the second sub-reference image information from the reference image database based on the predicted bone age and / or the information on the at least one region of interest. In embodiments of the present invention, the first sub-reference image information and the second sub-reference image information may be included in the reference image information.

[0085] In this case, the computing device can generate bone age information based on the prediction information output from the bone age prediction model (210, 220), the reference image information generated by the reference image information generator (230), the first sub-reference image information, and / or the second sub-reference image information. Hereinafter, a method for the computing device to output some / all of the generated bone age information on a display will be described in detail.

[0086] FIGS. 5 to 11 are diagrams for explaining a method for a computing device to output bone age information based on prediction information and reference image information.

[0087] The computing device can generate / output bone age information based on prediction information obtained by inputting a bone image into the bone age prediction model and the reference image information. That is, the computing device can generate the bone age information including the prediction information and the reference image information. Specifically, the computing device can generate bone age information including at least one of the predicted bone age, the finding information, the output bone image in which at least one region of interest is displayed in the bone image, and the reference bone image.

[0088] In addition, as illustrated in FIGS. 5 to 11, the computing device can output some / all of the information included in the bone age information on a display based on a predetermined UI (User Interface). Meanwhile, the output methods described below are each an example and, without limitation, the bone age information can be output on the display through various forms of UI (User Interface).

[0089] Referring to FIG. 5, the computing device can output the predicted goal age, the output goal image, and the reference goal image for the predicted goal age from among the bone age information on a display. That is, the computing device can output the output goal image and the reference goal image to the display through a single window so that they can be compared with each other.

[0090] For example, the computing device can display the output bone image and the reference bone image (Ref. 7Y 10M) for the predicted bone age, in which at least one region of interest (ROI; ①, ②, ③, ④, ⑤, ⑥, ⑦) is displayed (overlaid) as shown in FIG. 5 based on the bone age information. In addition, the computing device can output information (information separately input as metadata) about the predicted bone age (Bone Age; BA) and the age (Chronical Age; CA) of the subject of the input bone image on the display. Meanwhile, although each region of interest is displayed in the shape of a rectangle in FIG. 5, it is not limited thereto and may be displayed in various shapes such as a circle or a polygon.

[0091] Referring to FIG. 6, the computing device can output the reference bone image and the finding information included in the reference image information on the display through a single window. Here, the reference bone image can be output on the display so as to be sequentially arranged according to bone age as illustrated in FIG. 6, and the finding information can be output on the display so as to be arranged by region of interest. Meanwhile, the computing device can also output a window displaying the above-described output bone image and the reference bone image, and a window displaying the reference bone image and the finding, on a single screen.

[0092] Alternatively, the computing device may receive a user input for a specific region of interest among the at least one region of interest from an input device (mouse, keyboard, etc.). In this case, the computing device may respond to the user input based on the first sub-reference image information (see FIG. 4) included in the bone age information. Specifically, when the computing device receives a user input for a specific region of interest among the at least one region of interest from a user or an input device, the computing device may output a sub-goal image for the specific region of interest (i.e., a partial region image for the specific region of interest in the input bone image) and a sub-reference bone image corresponding to the specific region of interest among the sub-reference bone images included in the first sub-reference image information so as to be sequentially arranged according to bone age on the display.

[0093] Meanwhile, the computing device can enlarge the sub-goal image for the specific area and output it on the display together with the sub-reference goal image.

[0094] For example, as illustrated in FIG. 7, a sub-reference bone image corresponding to a bone age (Ref. 6Y 9M) earlier than the predicted bone age (7Y 10M), a sub-reference bone image which is a part of the bone image input to the bone age prediction model, a sub-reference bone image (Ref. 7Y 10M) corresponding to the predicted bone age, and a sub-reference bone image corresponding to a bone age (Ref. 8Y 10M) later (behind) the predicted bone age (7Y 10M) may be sequentially arranged and output on the display. In this case, the user can not only examine in more detail a specific area that has a major influence on the bone age prediction, but also more accurately and easily determine the change pattern of the bone according to the change in bone age through comparison / contrast with the sub-reference bone images corresponding to the bone ages before and after the predicted bone age.

[0095] Referring to FIG. 8, the computing device may output the predicted bone age, the output bone image displaying the at least one region of interest, and the finding information from among the bone age information on a display. At this time, the at least one region of interest may be grouped into at least two or more groups of regions of interest, such as the second sub-reference image information described with reference to FIG. 4, and displayed as included in the output bone image.

[0096] For example, the computing device may output the output bone image (overlaid) in which the first region of interest group (①) and the second region of interest group (②) are displayed on the bone image based on the bone age information, the predicted bone age (Ref. 12Y 6M), and the finding information on the display. Here, the finding information may be output on the display for each region of interest group.

[0097] Alternatively, the computing device may receive a user input for a specific region of interest among the at least one region of interest from an input device such as a keyboard or a mouse. In this case, the computing device may respond to the user input based on the second sub-reference image information included in the bone age information. Specifically, the computing device may receive a user input for a specific region of interest among the at least one region of interest from a user or an input device. In this case, the computing device may output group sub-goal reference images for a region of interest group corresponding to the specific region of interest among a plurality of group sub-goal reference images included in the second sub-reference image information on the display.

[0098] Specifically, the computing device may, in response to the user input, output, on a display, a portion of the output goal image corresponding to a region of interest group including the specific region of interest, and a group sub-goal reference image for the region of interest group. For example, referring to FIG. 9, the group sub-goal reference image for the region of interest group may include a first group sub-goal reference image for a bone age (11Y 0M) earlier than the predicted goal age (Ref. 12Y 6M), a second group sub-goal reference image for the predicted goal age (Ref. 12Y 6M), and a third group sub-goal reference image for a bone age (13Y 6M) later (behind) than the predicted goal age (Ref. 12Y 6M). The computing device may output, on a display, the first group sub-goal reference image, the second group sub-goal reference image, and the third group sub-goal reference image, which are sequentially arranged by bone age, and a portion of the output goal image corresponding to the region of interest group. In this case, the user can not only examine in more detail a portion of a group of regions of interest including the specific region of interest that had a major influence on bone age prediction, but also more accurately and easily determine the change pattern of the bone according to the change in bone age through comparison / contrast with the group sub-reference bone images corresponding to bone ages before and after the predicted bone age.

[0099] Alternatively, the computing device may output a sub-reference goal image that further enlarges a specific region of interest based on the first sub-reference image information in response to an additional user input while outputting the second sub-reference image information as illustrated in FIG. 9. For example, the computing device may receive an additional user input for selecting a specific region of interest among three regions of interest included in the first region of interest group. In response to the additional user input, the computing device may output sub-reference goal images for the specific region of interest from the first sub-reference image information on the display. In this case, the user may compare one specific region of interest among at least one region of interest in the region of interest group and sub-reference goal images related to the one specific region of interest in more detail.

[0100] Alternatively, referring to FIG. 10, the computing device may output to the display not only the predicted bone age and output bone information, but also related finding information for each region of interest that had a major influence on bone age prediction and information on a reference bone image related to bone age prediction.

[0101] Alternatively, referring to FIG. 11, the computing device may output the bone age information on the display based on the above-described region of interest group. Specifically, the computing device may output, on the display, not only the output bone image and predicted bone age, but also related finding information for each region of interest group described above, and group sub-reference bone images for each region of interest group, as the bone age information.

[0102] The bone age information based on the above bone age information can be provided in the form of DICOM (Digital Imaging and Communications in Medicine) format so that it can be linked with PACS (Picture Archiving Communication System).

[0103] FIG. 12 is a diagram illustrating a method for a computing device to output bone age information using a bone age prediction model.

[0104] Referring to FIG. 12, a computing device may input a bone image into the bone age prediction model and obtain prediction information about the bone image from the bone age prediction model (S121). As described above, the prediction information of the bone age prediction model may include a predicted bone age, information about at least one region of interest, and finding information related to the region of interest.

[0105] The above-described finding information may be at least one finding information related to the region of interest among the findings for the predicted bone age included in a database for bone age calculation methods (GP (Greulich & Pyle) database, TW database, Risser sign database, and / or Sauvegrain database) as described in FIGS. 3 and 4. For example, the bone age prediction model may be pre-trained to output finding information for at least one finding related to (or corresponding to) the predicted bone age and the region of interest among a plurality of findings included in the GP database. The at least one region of interest may be some regions that have a major influence on bone age prediction in a bone image as described above. Alternatively, the at least one region of interest may be grouped into at least one region of interest group based on the similarity of bone change patterns (or similarity of finding information) according to changes in bone age as described above. Meanwhile, the bone image input to the bone age prediction model may be an X-ray image of the subject's left hand, such as a wrist bone or wrist bone image, or an image of the subject's elbow bone or pelvis.

[0106] Next, the computing device may retrieve a reference bone image for at least one different bone age from a reference image database based on the predicted bone age, and generate reference image information including the retrieved reference bone image (S123). The reference image information may include reference bone images for bone ages before and after the predicted bone age (i.e., for a plurality of different bone ages). For example, the reference image information may include a first reference bone image for a bone age earlier than the predicted bone age, a second reference bone image for the predicted bone age, and a third reference bone image for a bone age later than the predicted bone age. As described above, the reference image database may include a plurality of reference bone images (e.g., a GP atlas, a Sauvegrain atlas, a Risser sign atlas, a TW atlas, and / or separate reference bone images similar thereto) that are reference images for a plurality of bone ages based on the GP method, the TW method, the Risser sign method, and / or the Sauvegrain method.

[0107] Alternatively, the computing device may generate first sub-reference image information and / or second sub-reference image information based on the at least one region of interest. As described above, the first sub-reference image may include sub-reference bone images for each of the at least one region of interest (i.e., sub-reference bone images for anterior and posterior bone ages based on the predicted bone age, such as the reference bone image) (see FIG. 7). The second sub-reference image information may include group sub-reference bone images for each of at least one region of interest group (see FIG. 9). The computing device may generate bone age information including at least one of the prediction information, the reference image information, the first sub-reference image information, and the second sub-reference image information.

[0108] Next, the computing device may output some / all of the bone age information for the bone image on the display based on the predicted information and the reference image information (S125). For example, before providing the above-described bone age information, so that the user can effectively review / analyze information related to the predicted bone age, the computing device may output, on the display, not only the predicted bone age for the bone image, but also an output bone image in which at least one region of interest that is a key region for predicting the bone age in the bone image is additionally displayed, finding information related to each of the at least one region of interest, and the reference bone image. For example, the computing device may output some / all of the bone age information on the display as described with reference to FIGS. 5 to 11. In addition, when the user's review / analysis is completed, the computing device may output, on the display, bone age information as illustrated in FIG. 10 and / or FIG. 11 based on the bone age information.

[0109] Alternatively, when a user input for selecting a specific region of interest from among the at least one region of interest is received, the computing device may additionally output a group sub-reference goal image for a region of interest group including the specific region of interest to the display based on the second sub-reference image information. Thereafter, when an additional user input for selecting one region of interest from among at least one region of interest included in the region of interest group is received, the computing device may additionally output a sub-reference image for the one region of interest based on the first sub-reference image information.

[0110] In this way, the computing device can significantly improve the reliability of bone age prediction by outputting bone age information including not only bone age prediction information for the bone image input to the bone age prediction model, but also region of interest and finding information that have a major influence on bone age prediction. In addition, the computing device can sequentially arrange at least one reference bone image related to bone age prediction and / or sub-reference bone images (or group sub-reference bone images) related to at least one region of interest that have a major influence on bone age prediction according to bone age, thereby providing the user with additional information on the change pattern and change process of (detailed) bone according to bone age. By providing such additional information, the reliability of the predicted bone age predicted by the bone age prediction model can significantly improve.

[0111] Based on the description of the above embodiments, it will be apparent to those skilled in the art that the methods and / or processes of the present invention, and their steps, may be implemented by hardware, software, or any combination of hardware and software suitable for a specific application. The hardware may include a general-purpose computer and / or a dedicated computing device, or a specific computing device or a specific aspect or component of a specific computing device. The processes may be implemented by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, or other programmable devices having internal and / or external memory. Additionally, or alternatively, the processes may be implemented by an application specific integrated circuit (ASIC), a programmable gate array, a programmable array logic (PAL), or any other device or combination of devices that can be configured to process electronic signals. Furthermore, the objects of the technical solution of the present invention or the parts contributing to prior art can be implemented in the form of program commands that can be executed by various computer components and recorded on a machine-readable recording medium. The machine-readable recording medium can include program commands, data files, data structures, etc., either singly or in combination. The program commands recorded on the machine-readable recording medium may be those specifically designed and constructed for the present invention or may be known and usable by those skilled in the art of computer software.Examples of machine-observable storage media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical recording media such as CD-ROMs, DVDs, and Blu-ray; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROMs, RAMs, and flash memories. Examples of program instructions include, but are not limited to, any of the aforementioned devices, as well as a processor, a heterogeneous combination of processor architectures or different combinations of hardware and software, or any other program instructions that can be stored and compiled or interpreted using a structured programming language such as C, an object-oriented programming language such as C++, or a high-level or low-level programming language (assembler, hardware description languages, and database programming languages ​​and technologies), and include not only machine code and byte code, but also high-level language code that can be executed by a computer using an interpreter, etc.

[0112] Accordingly, in one aspect according to the present disclosure, when the methods and combinations thereof described above are performed by one or more computing devices, the methods and combinations thereof may be implemented as executable code performing the respective steps. In another aspect, the methods may be implemented as systems performing the steps, and the methods may be distributed in various ways across the devices, or all functions may be integrated into a single dedicated, standalone device or other hardware. In yet another aspect, the means for performing the steps associated with the processes described above may comprise any of the hardware and / or software described above. All such sequential combinations and arrangements are intended to fall within the scope of the present disclosure.

[0113] For example, the hardware device may be configured to operate as one or more software modules to perform processing according to the present specification, and vice versa. The hardware device may include a processor, such as an MPU, a CPU, a GPU, or a TPU, coupled with a memory, such as a ROM / RAM, for storing program instructions and configured to execute the instructions stored in the memory, and may include a communication unit capable of sending and receiving signals with an external device. In addition, the hardware device may include a keyboard, a mouse, or other external input devices for receiving instructions written by developers.

[0114] Although the present invention has been described above with specific details such as specific components and limited examples and drawings, these are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above examples, and a person having ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from this description.

[0115] Therefore, the idea of ​​the present invention should not be limited to the embodiments described above, and all things that are modified equally or equivalently to the following claims as well as the claims are considered to fall within the scope of the idea of ​​the present invention.

[0116] Such equivalent or equivalent modifications would include, for example, logically equivalent methods capable of producing the same results as those obtained by performing the method according to the present specification. The spirit and scope of the present invention should not be limited by the examples set forth above, but should be understood in the broadest sense permissible by law.

[0117] As described above, the embodiments may be applied in whole or in part to a bone age prediction device and system. Those skilled in the art may make various modifications or variations to the embodiments within the scope of the embodiments. The embodiments may include modifications and variations, and the modifications and variations do not depart from the scope of the claims and their equivalents.

Claims

1. A method for a computing device to output bone age information, A step of inputting a bone image into a bone age prediction model and obtaining prediction information including a predicted bone age, finding information, and information about at least one region of interest included in the bone image from the bone age prediction model; A step of retrieving at least one reference bone image from a reference image database based on the predicted bone age, and generating reference image information including the retrieved at least one reference bone image; and A method comprising the step of outputting bone age information for the bone image based on the prediction information and the reference image information.

2. In paragraph 1, A method, characterized in that the at least one reference bone image comprises at least one of a first reference bone image for a bone age earlier than the predicted goal age, a second reference bone image for a bone age later than the predicted goal age.

3. In paragraph 1, A step of extracting at least one sub-reference goal image corresponding to the at least one region of interest from the at least one reference goal image; and A method, characterized in that it further comprises a step of generating sub-reference image information including at least one sub-reference goal image.

4. In paragraph 3, A step of receiving user input for a specific region of interest among at least one region of interest; and A method, characterized in that it further comprises the step of outputting, in response to the user input, a sub-reference goal image for the specific region of interest from among the at least one sub-reference goal image.

5. In paragraph 4, A method, characterized in that the sub-reference goal images include at least one of a first sub-reference goal image for a bone age earlier than the predicted goal age, a second sub-reference goal image for a bone age later than the predicted goal age.

6. In paragraph 1, A method, characterized in that the above finding information includes at least one finding related to each of the at least one region of interest among the findings for the predicted bone age.

7. In paragraph 1, A method, characterized in that the bone age information includes at least one of an output bone image indicating at least one region of interest in the bone image, the predicted bone age, and the reference image information.

8. In paragraph 1, A method, characterized in that said at least one region of interest is a region of said bone image that influences bone age prediction.

9. A non-transitory computer-readable storage medium having recorded thereon commands for performing the method described in paragraph 1.

10. In a computing device that outputs bone age information, Communication unit connected to external devices; display; and Including a processor connected to the above communication unit and the above display, A computing device wherein the processor inputs a bone image into a bone age prediction model to obtain prediction information including a predicted bone age, information on at least one region of interest, and finding information from the bone age prediction model, searches for a reference bone image for at least one bone age based on the predicted bone age from a reference image database, generates reference image information including the searched reference bone image, and outputs bone age information for the bone image on the display based on the prediction information and the reference image information.

Images