Method and electronic device for multifaceted integrated analysis of musculoskeletal diseases on basis of heterogeneous medical images

Abstract

A method for an electronic device to perform multifaceted integrated analysis of musculoskeletal diseases on the basis of heterogeneous medical images according to one embodiment of the present application comprises the steps of: obtaining a first type of medical image and a second type of medical image; determining at least one first individual status indicator by inputting the first type of medical image into a first neural network; determining at least one second individual status indicator by inputting the second type of medical image into a second neural network; and determining an integrated status indicator by inputting at least a part of the first type of medical image, the second type of medical image, the first individual status indicator, or the second individual status indicator into a third neural network. According to one embodiment of the present invention, musculoskeletal diseases can be analyzed in a multifaceted integrated manner by analyzing heterogeneous medical images.

Description

Method and electronic device for multi-faceted integrated analysis of musculoskeletal diseases based on heterogeneous medical images

[0001] The present application relates to a method for multi-faceted integrated analysis of musculoskeletal diseases, and relates to a method for multi-faceted integrated analysis of musculoskeletal diseases by integrating and analyzing heterogeneous medical images.

[0002] Previously, to diagnose musculoskeletal disorders based on images, individual indicators were analyzed based on each individual image. For example, for one patient, the stage of knee osteoarthritis was analyzed based on the cross-sectional state of knee X-ray images, such as joint space reduction, degree of osteophyte formation, and skeletalization. Alternatively, the alignment of the lower extremity skeletal system was analyzed using full-length lower extremity X-ray images. Alternatively, the disease status was analyzed with a focus on soft tissue, such as knee sole damage and abnormalities, using knee MRI images. Alternatively, the precise status of knee osteoarthritis and sarcopenia were analyzed using knee CT images.

[0003] Analyses like the above suffer from low accuracy due to the lack of comprehensive medical data analysis alongside non-imaging clinical data. Despite this, there is no solution in the medical field that allows for comprehensive analysis based on individual analysis results. Therefore, medical professionals analyze each image individually and then, based on clinical data and their own experience and knowledge, arrive at an overall diagnosis.

[0004] As such, the lack of a comprehensive diagnostic analysis solution for each patient's knee condition makes objective and quantitative integrated analysis difficult. Therefore, there is a need for a method capable of comprehensively analyzing musculoskeletal disorders, and a method that can guide appropriate treatment methods based on this integrated analysis.

[0005] The problem that the present invention seeks to solve is to provide a method capable of multi-faceted integrated analysis of musculoskeletal diseases.

[0006] In addition, a task to be solved by the present invention is to provide a method for determining an integrated condition index based on individual condition indexes individually analyzed from heterogeneous medical images.

[0007] In addition, a task to be solved by the present invention is to provide a method for predicting the prognosis of musculoskeletal diseases by performing a multifaceted integrated analysis of musculoskeletal diseases.

[0008] In addition, a task that the present invention seeks to solve is to provide a treatment guide for musculoskeletal diseases by conducting a multifaceted integrated analysis of musculoskeletal diseases.

[0009] The problems to be solved by the present invention are not limited to the problems described above, and problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention pertains from this specification and the attached drawings.

[0010] A method for performing a multi-faceted integrated analysis of a musculoskeletal disease based on heterogeneous medical images by an electronic device according to one embodiment of the present invention may include the steps of acquiring a first type of medical image and a second type of medical image, inputting the first type of medical image into a first neural network to determine at least one first individual condition indicator, inputting the second type of medical image into a second neural network to determine at least one second individual condition indicator, and inputting at least some of the first type of medical image, the second type of medical image, the first individual condition indicator, or the second individual condition indicator into a third neural network to determine an integrated condition indicator.

[0011] An electronic device for multi-faceted integrated analysis of musculoskeletal diseases based on heterogeneous medical images according to one embodiment of the present invention includes a communication unit, a display, a memory storing at least one instruction, and a processor executing the at least one instruction, wherein the processor acquires a first type of medical image and a second type of medical image, inputs the first type of medical image into a first neural network to determine at least one first individual condition indicator, inputs the second type of medical image into a second neural network to determine at least one second individual condition indicator, and inputs at least some of the first type of medical image, the second type of medical image, the first individual condition indicator, or the second individual condition indicator into a third neural network to determine an integrated condition indicator.

[0012] The solutions to the problems of the present invention are not limited to the solutions described above, and solutions that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.

[0013] According to one embodiment of the present invention, a multi-faceted integrated analysis of musculoskeletal diseases can be performed based on heterogeneous medical images.

[0014] In addition, according to one embodiment of the present invention, the prognosis of a musculoskeletal disease can be predicted by performing a multifaceted integrated analysis of the musculoskeletal disease.

[0015] Additionally, according to one embodiment of the present invention, a treatment guide for musculoskeletal diseases can be provided by performing a multifaceted integrated analysis of musculoskeletal diseases.

[0016] FIG. 1 is a block diagram briefly showing the configuration of an electronic device according to one embodiment of the present application.

[0017] FIG. 2 is a block diagram briefly showing the configuration of a processor according to one embodiment of the present application.

[0018] FIG. 3 is a drawing for explaining a method for analyzing individual medical images according to one embodiment of the present application.

[0019] FIG. 4 is a drawing for explaining a method for analyzing individual medical images according to one embodiment of the present application.

[0020] FIG. 5 is a drawing for explaining a method for analyzing individual medical images according to one embodiment of the present application.

[0021] FIG. 6 is a drawing for explaining a method for analyzing an integrated medical image according to one embodiment of the present application.

[0022] FIG. 7 is a flowchart illustrating a method for multi-faceted integrated analysis of musculoskeletal diseases based on heterogeneous medical images according to one embodiment of the present application.

[0023] The above-described purposes, features, and advantages of the present application will become more apparent through the following detailed description, taken in conjunction with the accompanying drawings. However, as the present application is susceptible to various modifications and various embodiments, specific embodiments will be illustrated in the drawings and described in detail below.

[0024] Throughout the specification, identical reference numbers, in principle, indicate identical components. Furthermore, components with identical functions within the scope of the same concept shown in the drawings of each embodiment are described using the same reference numbers, and redundant descriptions thereof will be omitted.

[0025] If a detailed description of a known function or configuration related to this application is deemed to unnecessarily obscure the gist of this application, such detailed description will be omitted. Furthermore, numbers (e.g., "first," "second," etc.) used throughout the description of this specification are merely identifiers used to distinguish one component from another.

[0026] In addition, the suffixes "module" and "part" for components used in the following examples are given or used interchangeably only for the convenience of writing the specification, and do not have distinct meanings or roles in themselves.

[0027] In the examples below, singular expressions include plural expressions unless the context clearly indicates otherwise.

[0028] In the examples below, terms such as “include” or “have” mean that a feature or component described in the specification is present, and do not preclude the possibility that one or more other features or components may be added.

[0029] For convenience of explanation, the sizes of components in the drawings may be exaggerated or reduced. For example, the sizes and thicknesses of each component shown in the drawings are arbitrarily shown for convenience of explanation, and the present invention is not necessarily limited to what is shown.

[0030] In some embodiments, where implementations are otherwise feasible, the order of specific processes may differ from the order described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the order described.

[0031] In the following examples, when components are said to be connected, this includes not only cases where the components are directly connected, but also cases where components are interposed between the components and are indirectly connected.

[0032] For example, when it is said in this specification that components, etc. are electrically connected, it includes not only cases where the components, etc. are directly electrically connected, but also cases where components, etc. are interposed in between and are indirectly electrically connected.

[0033] Meanwhile, the present invention is applicable to various fields. While the present specification will describe the technology using knee medical imaging, the present invention is not limited to the body area included in the medical image, and can be utilized in various medical image analyses for multifaceted, integrated analysis of musculoskeletal disorders such as the spine, shoulders, and pelvis.

[0034] In addition, the division of the first to third deep neural network models in the present invention is merely for the purpose of enhancing understanding of the invention, and in reality, there may be one or more models.

[0035]

[0036] Hereinafter, with reference to FIGS. 1 to 11, a method and electronic device for multi-faceted integrated analysis of musculoskeletal diseases based on heterogeneous medical images of the present application will be described.

[0037]

[0038] FIG. 1 is a block diagram briefly illustrating the configuration of an electronic device according to one embodiment of the present application. Referring to FIG. 1, the electronic device (100) may include a communication unit (110), a processor (120), and a memory (120).

[0039]

[0040] The communication unit (110) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (100) and an external electronic device, and the performance of communication through the established communication channel. The communication unit (110) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication.

[0041] The communication unit (110) can receive medical images from an external device. In addition, the communication unit (110) can transmit individual condition indicators determined from a single medical image and integrated condition indicators determined from two or more medical images to the external device.

[0042]

[0043] The processor (120) may execute software to control at least one other component (e.g., hardware or software component) of the electronic device (100) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., the communication unit (110)) in a volatile memory, process the command or data stored in the volatile memory, and store the resulting data in a non-volatile memory. According to one embodiment, the processor (120) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a neural processing unit (NPU)) that can operate independently or together therewith.

[0044] According to one embodiment, the auxiliary processor (e.g., a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, in the electronic device (100) itself where the artificial intelligence is performed, or may be performed through a separate server. The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network, a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

[0045] The processor (120) can obtain a first type of medical image and a second type of medical image through the communication unit (110). The medical image may refer to an image taken of a body part from which a musculoskeletal disease can be determined through an image, such as an X-ray image, an MRI image, a CT image, etc. For example, the medical image may be a knee X-ray image, a lower extremity full-length X-ray image, a knee MRI image, a knee CT image, a spine X-ray image, a spine MRI image, a spine CT image, a cervical X-ray image, a cervical MRI image, a cervical CT image, a thoracic X-ray image, a thoracic MRI image, a thoracic CT image, a shoulder X-ray image, a shoulder MRI image, a shoulder CT image, a pelvic X-ray image, a pelvic MIR image, a pelvic CT image, etc. The first type of medical image and the second type of medical image may be one of the types of medical images described, and the first type and the second type may refer to different types. For example, the processor (120) may acquire a first type of medical image, which is a knee X-ray image, and a second type of medical image, which is a lower extremity full-length X-ray image. In addition, the first type of medical image may be composed of multiple medical images of the same type. For example, the knee X-ray image may mean one or more of an anteroposterior (AP) knee X-ray image, a posteroanterior (PA) knee X-ray image, or a lateral knee X-ray image.

[0046] The processor (120) may input a first medical image into a first neural network to determine at least one individual condition indicator. Specifically, the processor (120) may train an appropriate neural network model to determine an individual condition indicator according to the type of the medical image, and may determine at least one individual condition indicator from the medical image using the trained neural network. For example, when the medical image is a knee X-ray image, the processor (120) may train a neural network (e.g., a CNN or a vision transformer) to extract a KL grade from the knee X-ray image, and may determine the KL grade from the knee X-ray image as an individual condition indicator using the trained neural network. In addition, the processor (120) may train a neural network to extract quantitative features for joint space reduction from the knee X-ray image, and may determine the quantitative features for joint space reduction from the knee X-ray image as an individual condition indicator using the trained neural network. In this manner, the processor (120) can determine individual condition indicators such as KL grade, joint space reduction, degree of imbalance between the medial and lateral sides of joint space reduction, presence and severity of osteophyte formation, and sclerosis and severity of subchondral bone in the knee X-ray image.

[0047] The processor (120) can input a second type of medical image into a second neural network to determine at least one individual condition index. For example, when the medical image is a lower extremity full-length X-ray image, the processor (120) can train the neural network to measure lower extremity alignment in the lower extremity full-length X-ray image, and can determine the degree of varus and valgus knees as individual condition indexes in the lower extremity full-length X-ray image using the trained neural network. In addition, the processor (120) can train the neural network to extract muscle mass in the lower extremity full-length X-ray image, and can determine the muscle mass as an individual condition index in the lower extremity full-length X-ray image using the trained neural network. In this manner, the processor (120) can determine individual condition indexes such as varus, valgus, muscle mass, bone density, and osteoporosis risk according to the lower extremity alignment angle in the lower extremity full-length X-ray image. Here, the second type of medical image is different from the first type of medical image only in type, and is the same as inputting the first type of medical image into the first neural network to determine at least one individual condition indicator.

[0048] The processor (120) may input at least some of the first type of medical image, the second type of medical image, the first individual condition index, or the second individual condition index into the third neural network to determine the integrated condition index. Specifically, the processor (120) may train the neural network or the machine learning model to determine the interrelationship of the individual condition indexes, and may correct the results of the individual indexes using the learned neural network or the machine learning model. Specifically, the processor (120) may correct at least some of the first individual condition index or the second individual condition index based on the first individual condition index and the second individual condition index, or may classify at least some of the first individual condition index or the second individual condition index into detailed steps. For example, when the first type of medical image is a knee X-ray image and the second type of medical image is a lower extremity full-length X-ray image, the processor (120) may subdivide the KL grade based on at least one of a lower extremity alignment measurement angle or a muscle mass.

[0049] The processor (120) can output an integrated condition index by determining an integrated condition index based on individual condition indices input to the third neural network. For example, if individual condition indices for a knee X-ray image, individual condition indices for a lower extremity full-length X-ray image, and individual condition indices for a knee MRI image are input to the third neural network to determine an integrated condition index, the processor (120) can output an integrated analyzed KL grade, lower extremity alignment status, and a quantitative evaluation of knee cartilage damage together.

[0050] The processor (120) can determine the prognosis of the first individual condition indicator or the second individual condition indicator based on at least some of the first type of medical image, the second type of medical image, the first individual condition indicator, or the second individual condition indicator. For example, if the first type of medical image is a knee X-ray image and the second type of medical image is a lower extremity full-length X-ray image, the processor (120) can predict the possibility of worsening of the KL grade based on at least one of the lower extremity alignment measurement angle or muscle mass. Alternatively, the processor (120) can predict the prognosis by determining the possibility of worsening of genu varum or valgus according to the lower extremity alignment measurement angle based on at least one of the KL grade, osteophytes, or osteosclerosis.

[0051] The processor (120) can obtain clinical data, such as the subject's age, gender, weight, or BMI, which reflects weight and height information. Furthermore, the processor (120) can additionally consider clinical data when determining the integrated status indicator. For example, the processor can set groups of clinical data and determine the likelihood of deterioration differently depending on the group to which the clinical data corresponds.

[0052] The processor (120) may provide treatment guidance based on individual condition indicators and / or integrated condition indicators. Specifically, the processor (120) may provide treatment guidance using a neural network trained to determine treatment guidance based on individual condition indicators and / or integrated condition indicators using follow-up observation data of multiple subjects as learning data.

[0053] The processor (120) may display at least one of the first type of medical image or the second type of medical image together with the integrated status indicator. For example, if the first type of medical image is a knee X-ray image and the second type of medical image is a lower extremity full-length X-ray image, the processor (120) may display the knee X-ray image and / or the lower extremity full-length X-ray image together with the integrated status indicator.

[0054]

[0055] The memory (130) can store various data used by at least one component (e.g., the processor (120)) of the electronic device (100). The data can include, for example, input data or output data for software and commands related thereto. The memory (130) can include volatile memory or non-volatile memory.

[0056]

[0057] FIG. 2 is a block diagram briefly illustrating the configuration of a processor according to one embodiment of the present application. FIG. 2 will be described in more detail with reference to FIGS. 3 to 6.

[0058] FIG. 3 is a drawing for explaining a method for analyzing an individual medical image according to one embodiment of the present application, FIG. 4 is a drawing for explaining a method for analyzing an individual medical image according to one embodiment of the present application, FIG. 5 is a drawing for explaining a method for analyzing an individual medical image according to one embodiment of the present application, and FIG. 6 is a drawing for explaining a method for analyzing an integrated medical image according to one embodiment of the present application.

[0059]

[0060] Referring to FIG. 2, the processor (120) may include an individual medical image analysis unit (121) and an integrated medical image analysis unit (123). The individual medical image analysis unit (121) and the integrated medical image analysis unit (123) may include one or more neural networks.

[0061]

[0062] The individual medical image analysis unit (121) can determine at least one individual condition indicator from a type of medical image. Specifically, the individual medical image analysis unit (121) can determine at least one individual condition indicator from a medical image using a neural network trained to extract individual indicators from a medical image according to the type of medical image.

[0063]

[0064] In one embodiment, when the medical image is a knee X-ray image, the individual medical image analysis unit (121) can determine condition indices according to the severity of arthritis in the knee X-ray image. For example, when the medical image is a knee X-ray image, the individual condition indices can include at least some of osteoarthritis condition indices such as KL grade, joint space, osteophyte analysis, and osteosclerosis analysis, quantitative condition indices for joint space reduction, condition indices for imbalance between the medial and lateral sides of joint space reduction, condition indices for the presence and severity of osteophyte formation, and condition indices for the severity of subchondral bone sclerosis.

[0065] The individual medical image analysis unit (121) can train a plurality of neural networks to determine at least one condition indicator from a knee X-ray image. For example, referring to FIG. 3, the individual medical image analysis unit (121) can train a neural network to extract a region of interest (b) for a knee area from a knee X-ray image (a), extract a joint gap region (c) from the region of interest, and determine a joint gap. The individual medical image analysis unit (121) can determine at least one individual condition indicator from a knee X-ray image using the trained neural network.

[0066]

[0067] In one embodiment, when the medical image is a lower extremity full-length X-ray image, the individual medical image analysis unit (121) can measure the alignment of the lower extremities in the lower extremity full-length X-ray image to determine condition indices for genu varum and genu valgus. For example, when the medical image is a lower extremity full-length X-ray image, the individual condition indices can include at least some of the following: a lower extremity alignment angle, condition indices for genu varum and genu valgus, condition indices for muscle mass and risk of sarcopenia according to muscle mass, and condition indices for bone density in the pelvic region and risk of osteoporosis according to bone density.

[0068] The individual medical image analysis unit (121) can train a plurality of neural networks to determine at least one condition indicator from a lower extremity full-length X-ray image. For example, referring to FIG. 4, the individual medical image analysis unit (121) can train a neural network to extract a plurality of feature points (a) from a lower extremity full-length X-ray image, and to connect the feature points to measure lower extremity alignment (HKA (Hip-Knee-Ankle), JLCA (Joint Line Convergence), LDFA (Lateral Distal Femoral Angle), MPTA (Media Proximal Tibial Angle)). In addition, the individual medical image analysis unit (121) can train a neural network to measure a joint line convergence angle (JLCA) (b) based on the lower extremity alignment measurement, and can train a neural network to determine the degree of varus and valgus based on a varus and valgus judgment criterion (c). The individual medical image analysis unit (121) can determine at least one individual condition indicator from a lower limb full-length X-ray image using a learned neural network.

[0069]

[0070] In one embodiment, when the medical image is a knee MRI image, the individual medical image analysis unit (121) can determine status indicators for the degree of damage to the knee soft tissue from the knee MRI image. For example, when the medical image is a knee MRI, the individual status indicators can include at least some of status indicators for the degree of damage (normal, partial loss, total loss) by cartilage location (e.g., medial / lateral, tibial / femoral, anterior / posterior) and status indicators for the degree of meniscus protrusion.

[0071] The individual medical image analysis unit (121) can train multiple neural networks to determine at least one condition index from a knee MRI image. For example, referring to FIG. 5, the criteria indicating the degree of damage to the knee soft tissue can be classified according to the location and degree of damage. The individual medical image analysis unit (121) can train a neural network to determine a condition index for the degree of damage to the medial / lateral, tibia / femur, anterior / posterior regions of the cartilage, or a condition index for the degree of semicircular cartilage protrusion from the knee MRI image. The individual medical image analysis unit (121) can determine at least one individual condition index from the knee MRI image using the trained neural network.

[0072]

[0073] In one embodiment, when the medical image is a knee CT image, the individual medical image analysis unit (121) can determine status indicators for 3D knee skeletal analysis from the knee CT image. For example, when the medical image is a knee CT image, the individual status indicators can include at least some of the status indicators for 3D skeletal analysis, the status indicators for joint space precision analysis, the status indicators for muscle mass precision analysis, and the status indicators for bone density and osteoporosis according to bone density.

[0074] The individual medical image analysis unit (121) can train multiple neural networks to determine at least one condition indicator from a knee CT image. For example, the individual medical image analysis unit (121) can train a neural network to determine a condition indicator for joint space, a condition indicator for muscle mass, or a condition indicator for bone density and thus osteoporosis from a knee CT image. The individual medical image analysis unit (121) can determine at least one individual condition indicator from a knee CT image using the trained neural network.

[0075]

[0076] The integrated medical image analysis unit (123) can determine an integrated condition index from different types of medical images. Specifically, the integrated medical image analysis unit (123) can determine an integrated condition index using a neural network trained to extract an integrated individual index based on different types of medical images and their respective individual condition indexes. Meanwhile, the integrated medical image analysis unit (123) can also determine individual condition indexes for different types of medical images.

[0077] Additionally, the integrated medical image analysis unit (123) can predict the prognosis of individual condition indicators based on different types of medical images and each individual condition indicator. Specifically, the integrated medical image analysis unit (123) can predict the prognosis of individual condition indicators by using a neural network trained to determine the possibility of deterioration of individual condition indicators using follow-up observation data of multiple subjects as learning data.

[0078]

[0079] In one embodiment, when different types of medical images are a knee X-ray image and a lower extremity full-length X-ray image, the integrated medical image analysis unit (123) can determine an integrated condition index based on the individual condition indexes for the knee X-ray image and the individual condition indexes for the lower extremity full-length X-ray image, and predict the prognosis of the individual condition indexes. Here, the integrated condition index can be a value that corrects the individual condition indexes or is a value that is a subdivision of the individual condition indexes.

[0080] The integrated medical image analysis unit (123) can train a neural network to correct or refine individual condition indicators based on individual condition indicators for knee X-ray images and individual condition indicators for lower extremity full-length X-ray images. The integrated medical image analysis unit (123) can use the trained neural network to comprehensively analyze the severity of knee osteoarthritis based on the degree of genu varum, valgus, or muscle mass. For example, if the degree of genu varum or valgus has progressed to a certain degree, the integrated medical image analysis unit (123) can refine KL Grade 3 into early, middle, and late stages and adjust it to KL Grade 3 late stage.

[0081] The integrated medical image analysis unit (123) can train a neural network to determine the possibility of deterioration of individual condition indicators based on individual condition indicators for a knee X-ray image and individual condition indicators for a lower extremity full-length X-ray image. The integrated medical image analysis unit (123) can use the trained neural network to predict the prognosis of whether knee osteoarthritis will worsen or be maintained based on the degree of varus or valgus knees or muscle mass. For example, the integrated medical image analysis unit (123) can predict the possibility that a KL grade 3 determined from a knee X-ray image will worsen to a KL grade 4 within 48 months when the muscle mass is below a certain amount. The integrated medical image analysis unit (123) can use the trained neural network to predict the prognosis of a lower extremity alignment angle based on condition indicators for osteoarthritis. For example, the integrated medical image analysis unit (123) can predict that a degree of varus determined from a lower extremity full-length X-ray image will worsen when osteoarthritis has progressed to a certain degree or more.

[0082]

[0083] In one embodiment, when different types of medical images are a knee X-ray image and a knee MRI image, the integrated medical image analysis unit (123) can determine an integrated condition index based on individual condition indices for the knee X-ray image and individual condition indices for the knee MRI image.

[0084] The integrated medical image analysis unit (123) can train a neural network to correct or subdivide individual condition indicators based on individual condition indicators for knee X-ray images and individual condition indicators for knee MRI images. The integrated medical image analysis unit (123) can use the trained neural network to comprehensively analyze the severity of knee osteoarthritis based on the degree of damage to the knee cartilage. For example, the integrated medical image analysis unit (123) can correct the KL 3 grade, which is an individual condition indicator determined from the knee X-ray image, to KL 4 grade if the degree of wear of the knee cartilage is above a certain level based on the degree of damage to the knee cartilage, or can subdivide the KL 3 grade into early, middle, and late stages and adjust it to KL 3 grade late stage.

[0085] The integrated medical image analysis unit (123) can train a neural network to determine the possibility of deterioration of individual condition indices based on individual condition indices for knee X-ray images and individual condition indices for knee MRI images. The integrated medical image analysis unit (123) can use the trained neural network to predict the prognosis of whether knee osteoarthritis will worsen or persist based on the degree of damage to the knee cartilage. For example, the integrated medical image analysis unit (123) can predict the possibility that the KL 3 grade determined in the knee X-ray image will worsen to KL 4 grade within 18 months if the damaged area of ​​the knee cartilage is a specific area or the degree of damage is above a certain level. The integrated medical image analysis unit (123) can use the trained neural network to predict the prognosis of the degree of knee cartilage damage based on the condition indices for knee osteoarthritis. For example, the integrated medical image analysis unit (123) can predict that the degree of cartilage damage determined in the knee MRI image will worsen if osteoarthritis has progressed to a certain degree.

[0086]

[0087] In one embodiment, when different types of medical images are a full-length X-ray image of a lower extremity and a knee MRI image, the integrated medical image analysis unit (123) can determine an integrated condition index based on individual condition indexes for the full-length X-ray image of a lower extremity and individual condition indexes for the knee MRI image, and predict the prognosis of the individual condition indexes.

[0088] The integrated medical image analysis unit (123) can train a neural network to correct or refine individual condition indicators based on individual condition indicators for lower extremity full-length X-ray images and individual condition indicators for knee MRI images. The integrated medical image analysis unit (123) can use the trained neural network to perform an integrated analysis of the lower extremity alignment angle based on individual indicators for knee cartilage damage.

[0089] The integrated medical image analysis unit (123) can train a neural network to determine the possibility of deterioration of individual condition indices based on individual condition indices for lower extremity full-length X-ray images and individual condition indices for knee MRI images. The integrated medical image analysis unit (123) can predict the prognosis for the lower extremity alignment angle based on individual indices for knee cartilage damage using the trained neural network. For example, the integrated medical image analysis unit (123) can determine whether the degree of varus will worsen or remain constant depending on the location or degree of damage to the knee cartilage. The integrated medical image analysis unit (123) can predict the prognosis for knee cartilage damage based on condition indices for the lower extremity alignment angle, condition indices for muscle mass, and condition indices for bone density using the trained neural network. For example, the integrated image analysis unit (123) can predict that the degree of knee cartilage damage will worsen when the bone density is below a certain level.

[0090]

[0091] In one embodiment, when different types of medical images are a knee X-ray image, a lower extremity full-length X-ray image, and a knee MRI image, the integrated medical image analysis unit (123) can determine an integrated condition index based on the individual condition indexes for the knee X-ray image, the individual condition indexes for the lower extremity full-length X-ray image, and the individual condition indexes for the knee MRI image, and can predict the prognosis of the individual condition indexes.

[0092] The integrated medical image analysis unit (123) can train a neural network to correct or subdivide the individual condition indices based on the individual condition indices for the knee X-ray image, the individual condition indices for the lower extremity full-length X-ray image, and the individual condition indices for the knee MRI image. The integrated medical image analysis unit (123) can use the trained neural network to comprehensively analyze the severity of knee osteoarthritis based on the individual indices for knee cartilage damage and the individual indices for the lower extremity alignment angle. For example, if the damaged area of ​​the knee cartilage is a specific area and the degree of varus or valgus knees has progressed to a certain degree, the integrated medical image analysis unit (123) can subdivide the individual condition indices KL Grade 3 determined from the knee X-ray image into early, middle, and late stages and adjust them to KL Grade 3 late stages.

[0093] The integrated medical image analysis unit (123) can train a neural network to determine the possibility of deterioration of individual condition indices based on individual condition indices for knee X-ray images, individual condition indices for lower extremity full-length X-ray images, and individual condition indices for knee MRI images. The integrated medical image analysis unit (123) can use the trained neural network to predict the prognosis of whether knee osteoarthritis will worsen or be maintained based on individual indices for knee cartilage damage and individual indices for lower extremity alignment angles. For example, the integrated medical image analysis unit (123) can predict the possibility that KL Grade 3 determined in a knee X-ray image will worsen to KL Grade 4 within 48 months when muscle mass is below a certain amount and the degree of knee cartilage damage is above a certain degree.

[0094]

[0095] In one embodiment, when different types of medical images are a knee X-ray image and a knee CT image, the integrated medical image analysis unit (123) can determine an integrated condition index based on individual condition indexes for the knee X-ray image and individual condition indexes for the knee CT image, and predict the prognosis of the individual condition indexes.

[0096]

[0097] As shown in FIG. 6, the integrated medical image analysis unit (123) can determine an integrated condition index based on at least two types of medical images among a knee X-ray image, a lower extremity full-length X-ray image, a knee MRI image, or a knee CT image, and / or individual condition indexes determined from the medical images.

[0098]

[0099] FIG. 7 is a flowchart illustrating a method for multifaceted integrated analysis of musculoskeletal diseases based on heterogeneous medical images according to one embodiment of the present application. The operations in FIG. 7 are not limited in order, and other operations may be performed between two adjacent operations. In addition, at least some of the operations in FIG. 7 may be omitted. In the present invention, the expression that the electronic device (100) performs a specific operation may mean that the processor (120) of the electronic device (100) performs the specific operation, or that the processor (120) controls other hardware to perform the specific operation.

[0100]

[0101] Referring to FIG. 7, the electronic device (100) can acquire a first type of medical image and a second type of medical image (S1000). In this case, the first type of medical image and the second type of medical image may refer to different types of medical images.

[0102]

[0103] An electronic device (100) can input a first type of medical image into a first neural network to determine at least one first individual indicator (S2000). Specifically, the electronic device (100) can train an appropriate neural network model to determine at least one individual condition indicator depending on the type of medical image, and can use the trained neural network to determine at least one individual condition indicator from the first type of medical image.

[0104]

[0105] The electronic device (100) can input a second type of medical image into a second neural network to determine at least one second individual indicator (S3000). Specifically, the electronic device (100) can train an appropriate neural network model to determine at least one individual condition indicator depending on the type of medical image, and can use the trained neural network to determine at least one individual condition indicator from the second type of medical image.

[0106]

[0107] The electronic device (100) can input at least some of the first type of medical image, the second type of medical image, the first individual indicator, or the second individual indicator into the third neural network to determine the integrated condition indicator (S4000). Specifically, the electronic device (100) can train the neural network or machine learning model to correct at least one individual condition indicator based on the interrelationship between the individual condition indicators, and can correct the individual indicators using the trained neural network or machine learning model.

[0108]

[0109] According to one embodiment of the present invention, there is an advantage in that a multi-faceted integrated analysis of musculoskeletal diseases can be performed based on heterogeneous medical images.

[0110]

[0111] The features, structures, effects, etc. described in the embodiments above are included in at least one embodiment of the present invention, and are not necessarily limited to just one embodiment. Furthermore, the features, structures, effects, etc. exemplified in each embodiment can be combined or modified in other embodiments by those skilled in the art to which the embodiments pertain. Therefore, the contents related to such combinations and modifications should be construed as falling within the scope of the present invention.

[0112] In addition, although the above description focuses on the embodiments, these are merely examples and do not limit the present invention. Those skilled in the art to which the present invention pertains will appreciate that various modifications and applications not exemplified above are possible without departing from the essential characteristics of the present embodiments. In other words, each component specifically shown in the embodiments can be modified and implemented. In addition, differences related to such modifications and applications should be interpreted as being included within the scope of the present invention defined in the appended claims.

Claims

1. A method for multi-faceted integrated analysis of musculoskeletal diseases based on heterogeneous medical images using an electronic device, A step of acquiring a first type of medical image and a second type of medical image; A step of inputting the first type of medical image into a first neural network to determine at least one first individual condition indicator; A step of inputting the second type of medical image into a second neural network to determine at least one second individual condition indicator; and A step of inputting at least some of the first type of medical image, the second type of medical image, the first individual condition indicator, or the second individual condition indicator into a third neural network to determine an integrated condition indicator; Multifaceted integrated analysis method.

2. In paragraph 1, A step of determining a prognosis of the first individual condition indicator or the second individual condition indicator based on at least a part of the first type of medical image, the second type of medical image, the first individual condition indicator or the second individual condition indicator; further comprising; Multifaceted integrated analysis method.

3. In paragraph 1, A step of displaying at least one of the first type of medical image or the second type of medical image and the integrated status indicator together; further comprising; Multifaceted integrated analysis method.

4. In paragraph 1, The step of determining the above integrated status indicator is: Correcting at least some of the first individual condition indicator or the second individual condition indicator based on the first individual condition indicator and the second individual condition indicator Multifaceted integrated analysis method.

5. In paragraph 1, The step of determining the above integrated status indicator is: Classifying at least some of the first individual condition indicator or the second individual condition indicator into detailed steps based on the first individual condition indicator and the second individual condition indicator. Multifaceted integrated analysis method.

6. In paragraph 1, The step of determining at least one first individual condition indicator is: A step of detecting a region of interest in the first medical image; and a step of determining the severity of a disease in the area of ​​interest; Multifaceted integrated analysis method.

7. In paragraph 1, The first type of medical image is a knee X-ray image, and the second type of medical image is a lower extremity full-length X-ray image. The step of determining the above first individual status indicator is: Inputting the first type of medical image into the first neural network to determine an individual condition indicator for at least one of KL grade, osteophyte, or osteosclerosis, The step of determining the second individual status indicator is: Inputting the second type of medical image into the second neural network to determine an individual condition index for at least one of genu varum, genu valgus, or muscle mass according to the lower limb alignment measurement angle. Multifaceted integrated analysis method.

8. In paragraph 7, The step of determining the above integrated status indicator is: The KL grade is subdivided based on at least one of the above leg alignment measurement angle or muscle mass. Multifaceted integrated analysis method.

9. In paragraph 8, A step of predicting the prognosis by determining the possibility of deterioration of the KL grade based on at least one of the alignment measurement angle or muscle mass of the lower extremities; further comprising; Multifaceted integrated analysis method.

10. In paragraph 7, A step of predicting the prognosis by determining the possibility of worsening of genu varum or valgus knees according to the lower limb alignment measurement angle based on at least one of the above KL grade, osteophytes or osteosclerosis; further comprising; Multifaceted integrated analysis method.

11. In a non-transitory computer-readable recording medium including a program for executing a method of controlling an electronic device, The method of controlling the above electronic device is as follows: A step of acquiring a first type of medical image and a second type of medical image; A step of inputting the first type of medical image into a first neural network to determine at least one first individual condition indicator; A step of inputting the second type of medical image into a second neural network to determine at least one second individual condition indicator; and A step of inputting at least some of the first type of medical image, the second type of medical image, the first individual condition indicator, or the second individual condition indicator into a third neural network to determine an integrated condition indicator; Computer readable recording medium.

12. In an electronic device for multi-faceted integrated analysis of musculoskeletal diseases based on heterogeneous medical images, Department of Communications; display; memory that stores at least one instruction; and a processor for executing at least one instruction; The above processor, Acquire type 1 medical images and type 2 medical images, Inputting the first type of medical image into the first neural network to determine at least one first individual condition indicator, Inputting the second type of medical image into the second neural network to determine at least one second individual condition indicator, Inputting at least some of the first type of medical image, the second type of medical image, the first individual condition indicator, or the second individual condition indicator into the third neural network to determine an integrated condition indicator. Electronic devices.

Images