Method, device, and program-recording storage medium for providing information about whether knee is abnormal

Abstract

A method for providing information about whether a knee is abnormal according to an embodiment comprises: a preprocessing step for extracting a first image set including skin contour information by processing each of a plurality of pieces of medical image data obtained by imaging a knee region; a step for generating a knee abnormality determination model by performing training on the basis of the first image set; a step for inputting, to the abnormality determination model, a diagnostic image including skin contour information in a knee region of a subject; and a step for outputting, from the abnormality determination model, information about whether a knee of the subject is abnormal.

Description

Storage medium recording a method, device and program for providing information on whether there is an abnormality in the knee

[0001] Embodiments of the present disclosure relate to a method, device, and storage medium recording a program for providing information on abnormalities in the knee. Specifically, the present disclosure includes a device and method for early diagnosis of knee osteoarthropathy using surface anatomy and deep learning. Furthermore, the present disclosure includes a device and a method for operating the same for early diagnosis of knee osteoarthropathy based on surface anatomy using a convolutional neural network (CNN) and an image of the knee popliteus.

[0002] Convolutional neural networks (CNNs) are a type of deep learning model capable of classifying and / or making decisions based on images. Using this AI, classification and / or decisions can be made on new images based on trained images and labeled labels.

[0003] Total knee arthroplasty (TKA) is a major surgical procedure that replaces a damaged or worn-out knee joint with an artificial one. The lifespan of the artificial joint limits the age of patients eligible for the procedure. Regardless of age, the most important factor in deciding whether to proceed with surgery is the condition of the patient's knee, which can be diagnosed using X-rays, CT scans, or MRI. However, X-rays and CT scans involve radiation exposure, and CT and MRI are expensive to perform. For these reasons, X-rays, which are inexpensive and have minimal radiation exposure, are often used for initial diagnosis. However, it is often difficult to make an accurate decision on whether to proceed with TKA based solely on X-ray images. In other words, additional imaging may be required for an accurate diagnosis, resulting in additional radiation exposure and costs. To address these issues, methods that can determine whether to proceed with TKA based solely on the appearance of the knee are being discussed, one of which is a surface anatomy-based approach. Surface anatomy, a branch of anatomy that studies the external features of the body, offers insights into human function through observable anatomical areas (e.g., shape, proportions, landmarks, etc.) and even allows for disease prediction, opening up new avenues for diagnosis. In particular, in clinical medicine, such as orthopedics, where surgical procedures are the primary focus, surface anatomy is gaining recognition as a crucial research area capable of significantly improving patients' quality of life through non-invasive procedures and diagnostics.

[0004] Knee surgeries, such as total knee replacement, often incur high imaging costs to determine whether surgery is warranted, and often result in high surgical costs. Furthermore, if unnecessary surgery is performed due to a misdiagnosis, the lifespan of the artificial joint can necessitate future revision surgery. Consequently, knee patients are often exposed to unnecessary financial burdens and potential health risks. Various embodiments of the present disclosure provide a method for accurately diagnosing knees at a low cost.

[0005] For example, the present disclosure deals with a method of analyzing the condition of a knee captured by a camera or smart device using artificial intelligence and providing information on whether there is an abnormality in the health of the knee.

[0006] A method for providing information on whether there is an abnormality in a knee according to one embodiment includes a preprocessing step of processing a plurality of medical image data photographing a knee area to extract a first image set including skin contour information; a step of generating a model for determining whether there is an abnormality in a knee by performing learning based on the first image set; a step of inputting a diagnostic image including skin contour information in a knee area of ​​a subject into the abnormality determination model; and a step of outputting information on whether there is an abnormality in the knee of the subject from the abnormality determination model.

[0007] A device for providing information on whether there is an abnormality in a knee according to one embodiment includes: a storage unit for storing a plurality of medical image data photographing a knee area; and a control unit for processing each of the plurality of medical image data to extract a first image set including skin contour information, performing learning based on the first image set to generate a model for determining whether there is an abnormality in the knee, inputting a diagnostic image including skin contour information in a knee area of ​​a subject into the abnormality determination model, and outputting information on whether there is an abnormality in the knee of the subject from the abnormality determination model.

[0008] A computer-readable storage medium recording a program for providing information on whether there is an abnormality in a knee according to one embodiment records a program for causing a computer to execute the following operations: a preprocessing operation for processing a plurality of medical image data photographing a knee region to extract a first image set including skin contour information; an operation for generating a model for determining whether there is an abnormality in a knee by performing learning based on the first image set; an operation for inputting a diagnostic image including skin contour information in a knee region of a subject into the model for determining whether there is an abnormality; and an operation for outputting information on whether there is an abnormality in the knee of the subject from the model for determining whether there is an abnormality.

[0009] The present invention enables diagnosis of the health of a knee using only a knee photograph taken at low or no additional cost. For example, if the diagnosed health status of a knee is output as only two values, "Healthy" or "Abnormal," the patient can easily determine whether it is time to visit a hospital or not. As another example, if the diagnosed health status of a knee includes information regarding the need for total knee replacement, the patient can save on the cost and time of examinations (e.g., imaging) for diagnosing the knee joint, and the doctor can improve the accuracy of the diagnosis through an artificial intelligence model trained on labeled data, using information measured from the patient's medical images.

[0010] Figure 1 is a flowchart illustrating a process for providing information on whether a knee is abnormal according to one embodiment.

[0011] Fig. 2 shows a configuration diagram that embodies each device of the processing process in the embodiment of Fig. 1.

[0012] Figure 3 is a flowchart showing a deep learning training process of a knee joint disease diagnosis device according to one embodiment.

[0013] FIG. 4 illustrates a flow chart of a method for providing information on whether a knee is abnormal according to one embodiment.

[0014] Fig. 5 shows a configuration diagram of a device that provides information on whether a knee is abnormal according to one embodiment.

[0015] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. When adding reference numerals to components in each drawing, identical components may have the same numerals as much as possible even if they are shown in different drawings. In addition, when describing the present embodiments, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the technical idea of ​​the present invention, the detailed description may be omitted. When "includes," "has," "consists of," etc. are used in this specification, other parts may be added unless "only" is used. When a component is expressed in the singular, it may include a plural unless there is a special explicit description.

[0016] Additionally, terms such as first, second, A, B, (a), (b), etc. may be used to describe components of the present disclosure. These terms are only intended to distinguish the components from other components, and the nature, order, sequence, or number of the components are not limited by the terms.

[0017] In a description of the positional relationship of components, when it is described that two or more components are "connected," "combined," or "connected," it should be understood that the two or more components may be directly "connected," "combined," or "connected," but that the two or more components may also be further "interposed" with another component to be "connected," "combined," or "connected." Here, the other component may be included in one or more of the two or more components that are "connected," "combined," or "connected" to each other.

[0018] In the description of the temporal flow relationship related to components, operation methods, or manufacturing methods, for example, when the temporal or flow relationship is described as “after”, “following”, “next to”, “before”, etc., it may also include cases where it is not continuous, unless “immediately” or “directly” is used.

[0019] Meanwhile, when numerical values ​​or corresponding information (e.g., reference values, etc.) for components are mentioned, even if there is no separate explicit description, the numerical values ​​or corresponding information may be interpreted as including an error range that may occur due to various factors (e.g., process factors, internal or external impact, noise, etc.).

[0020] The embodiments are described in detail with reference to the drawings below.

[0021] Figure 1 is a flowchart illustrating a process for providing information on whether a knee is abnormal according to one embodiment.

[0022] In step 110, a knee photograph is taken of the subject (patient). This can be done using a camera, digital camera, cell phone, smartphone, or wearable device. The knee photograph may be taken by the subject while looking in a mirror, or by someone nearby. The knee photograph may be taken in a location unrelated to a medical institution.

[0023] At step 120, the knee image may be stored in a data storage device. The data storage device may be the memory of a smartphone or wearable device or the storage of a separate device (server).

[0024] At step 130, the knee image may be processed by a main processing unit. The main processing unit may be a processor of a smartphone or wearable device, or a processor of a separate device (server). Processing here may include inputting the knee image into a trained artificial intelligence model and receiving knee condition information.

[0025] At step 140, the output of the AI ​​model can be transmitted to software. The software may be standalone software, a portion of a specific software program, a web application, or one or more applications. The software can be accessed by the patient or the physician. The physician can diagnose the subject's knee condition based on the output transmitted to the software.

[0026] Fig. 2 shows a more detailed configuration diagram of each device in the embodiment of Fig. 1.

[0027] A knee image (220) captured by a camera (210), etc., is input into the image input unit of the main processing device (250), processed by the image processing unit, and also stored in the data storage of the data storage device (240). The knee image, on which noise removal, etc. has been performed in the image processing unit of the main processing device (250), may be transmitted to the diagnosis unit of the main processing device (250), so that a weight file for the corresponding knee image may be generated. This weight file may be transmitted to the weight file update unit of the learning server (230), so that the weight file stored by the learning server (230) may be updated. The knee image stored in the data storage of the data storage device (240) may be classified by the data classification unit of the data storage device (240) and transmitted to the deep learning model of the learning server (230).

[0028] The diagnosis unit of the main processing device (250) can transmit a knee image to the classification unit of the main processing device (250) and transmit the result of classifying the knee condition indicated by the knee image to the result processing unit of the main processing device (250). The result processing unit of the main processing device (250) can transmit information about the result of classifying the knee condition to a web server and / or standalone software (280). The web server can transmit and receive information with a database located within or outside the same device, and the database can transmit and receive information with an operation server located within or outside the same device. The web server can transmit information about the result of classifying the knee condition to a web app or application installed in another device upon a user request.

[0029] The operations illustrated in the embodiments illustrated in FIGS. 1 and 2 may be omitted, at least in part, and at least some of the operations performed by each server or each device may be entirely performed on a single device (e.g., a smartphone). For example, all operations except those of the learning server may be implemented as being performed on a single device.

[0030] With the expansion of the Common Data Model (CDM), the collection of medical image data has become easier. For example, in Korea, to support healthcare research and development, the CDM, a nationwide medical information database, is open to researchers who meet certain criteria. The CDM includes CT images, and various embodiments of this disclosure can perform deep learning training using a large volume of CT images.

[0031] Figure 3 is a flowchart showing a deep learning training process of a knee joint disease diagnosis device according to one embodiment.

[0032] The device of FIG. 3 reconstructs a 3D image, such as a CT image (310), into an image (320) having two or more types and performs measurements. The reconstructed image (320) may include, for example, a first image type (right) including skin contour information and a second image type (left) including bone information. In various embodiments of the present disclosure, the second image type may be an X-ray image or a similar form.

[0033] At least a portion of the reconstructed image (320) can be input to an artificial neural network for deep learning. The embodiment of FIG. 3 illustrates an example of inputting image data (330) focused on the occipital region of the second image type of the reconstructed 3D image (320) into a CNN together with labeling data (340).

[0034] The popliteal line is a crease line that appears in the concave area behind the knee when the knee is bent. Recently, scientific evidence has been presented that can determine the condition of the knee joint based on the angle and shape of the popliteal line, and it has been revealed that the popliteal line can be used as a surface anatomical landmark to diagnose the need for total knee arthroplasty according to the condition of the knee joint. For example, by training a convolutional neural network with the surface anatomical features of the knees of patients who underwent total knee arthroplasty and also training a convolutional neural network with the surface anatomical features of the knees of people with normal knees, it is possible to diagnose whether or not to proceed with total knee arthroplasty using knee images captured by a camera. Alternatively, a method or algorithm may be provided for preparing knee photos of a patient who underwent total knee replacement surgery, knee photos of a normal person with a normal knee joint, and knee photos according to the aging state of the knee joint, training a dataset in which the knee photos and the corresponding images of the patient's knee joint status are labeled using deep learning, and inputting a knee photo taken of a new patient into the trained model to diagnose the knee joint status and determine whether or not total knee replacement surgery is required.

[0035] Among surface anatomical features, not only the angle and shape of the popliteal line, but also the degree of knee swelling (and thus aging status) and / or the shape of the knee can be used as indicators for diagnosing the health of the knee joint. For example, deep learning can be used to learn the correlation between knee swelling or shape and knee joint health, and information about the health of the knee joint can be obtained from new patient knee images.

[0036] In various embodiments of the present disclosure, the CNN includes an input layer, a convolutional layer, a pooling layer, and a fully connected layer, and can sequentially perform softmax and classification operations.

[0037] FIG. 4 illustrates a flow chart of a method for providing information on whether a knee is abnormal according to one embodiment.

[0038] In step 410, a plurality of medical image data photographed in the knee region are processed to extract a first image set including skin contour information. This preprocessing step may further include a step of processing the plurality of medical image data to extract a second image set including bone information, and performing labeling on the second image set. In various embodiments of the present disclosure, the plurality of medical image data photographed in the knee region may be CT images photographed including the knees of multiple patients. In various embodiments of the present disclosure, the first image set may include information about the shape of the popliteal line and / or swelling of the knee.

[0039] In step 420, a model for determining whether a knee is abnormal is created by performing learning based on the first image set. For example, the model for determining whether a knee is abnormal can be created by performing learning based on multiple first images and multiple labeled data. Alternatively, the model for determining whether a knee is abnormal can be created by performing learning based on the first image set and the labeled second image set.

[0040] In step 430, a diagnostic image containing skin contour information of the subject's knee area is input into the abnormality determination model. The diagnostic image is an image without any medical information labeling or annotation. The diagnostic image may be taken directly by the subject using a mirror or similar device, or may be taken by someone nearby.

[0041] In step 440, a step is included in which information regarding the presence or absence of an abnormality in the subject's knee is output from the abnormality determination model. Here, the information regarding the presence or absence of an abnormality in the subject's knee represents information regarding the health status of the knee and may include information regarding the condition of the subject's knee joint. The information regarding the condition of the subject's knee joint may include information regarding the need for total knee replacement surgery.

[0042] Fig. 5 shows a configuration diagram of a device that provides information on whether a knee is abnormal according to one embodiment.

[0043] The storage unit (510) can store a plurality of medical image data taken of the knee area. The storage unit (510) can include a non-transitory storage medium of at least one type among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD / XD memory, etc.), a RAM (Random Access Memory), a SRAM (Static Random Access Memory), a ROM (Read-Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a PROM (Programmable Read-Only Memory), a magnetic memory, a magnetic disk, and an optical disk, and there is no limitation on the type thereof.

[0044] The control unit (520) processes each of the plurality of medical image data to extract a first image set including skin contour information, performs learning based on the first image set to generate a model for determining whether there is an abnormality in the knee, inputs a diagnostic image including skin contour information in the knee area of ​​the subject into the abnormality determination model, and outputs information on whether there is an abnormality in the knee of the subject from the abnormality determination model. The control unit (520) is configured by one or more processors, such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or a core including the same, and controls the entire device for providing information by executing a computer program stored in the storage unit (510). Alternatively, the control unit (520) may control the entire device for providing information through cooperation between the computer program stored in the storage unit (510) and an OS (Operating System).

[0045] Additionally, the control unit (520) can generate data or signals to be transmitted in communication with other devices. In this case, although not illustrated in FIG. 5, the device providing information on knee abnormalities may further include a communication unit. The communication unit (1220) can connect to other devices and transmit and receive data using a communication module such as Bluetooth or a wired or wireless LAN (Local Area Network).

[0046] The embodiments described above may be implemented through various means. For example, the embodiments may be implemented through hardware, firmware, software, or a combination thereof.

[0047] In the case of hardware implementation, the method for providing information on whether there is an abnormality in the knee according to the present embodiments may be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), processors, controllers, microcontrollers, or microprocessors.

[0048] When implemented via firmware or software, the method for providing information on knee abnormalities according to the present embodiments may be implemented in the form of a device, procedure, or function that performs the functions or operations described above. The software code may be stored in a memory unit and executed by a processor. The memory unit may be located within or outside the processor and may exchange data with the processor via various known means.

[0049] Additionally, terms such as "system," "processor," "controller," "component," "module," "interface," "model," or "unit" as described above may generally refer to a computer-related entity, such as hardware, a combination of hardware and software, software, or software in execution. For example, the aforementioned components may be, but are not limited to, a process driven by a processor, a processor, a controller, a control processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a controller or a processor and the controller or the processor may be components. One or more components may be within a process and / or thread of execution, and the components may be located on a single device (e.g., a system, a computing device, etc.) or distributed across two or more devices.

[0050] Meanwhile, another embodiment provides a computer program stored on a computer storage medium that performs the method for providing information on the presence of a knee abnormality described above. Furthermore, another embodiment provides a computer-readable storage medium storing a program for implementing the method for providing information on the presence of a knee abnormality described above. The program recorded on the storage medium can be read, installed, and executed by a computer, thereby executing the steps described above.

[0051] In this way, in order for the computer to read the program recorded on the recording medium and execute the functions implemented as the program, the above-mentioned program may include code coded in a computer language such as C, C++, JAVA, or machine language that the computer's processor (CPU) can read through the computer's device interface. The method for extracting the ventricle from the CT image described above can be classified as an image processing algorithm and can be implemented through a function of a computer vision processing package such as Python's Numpy, OpenCV, Scikit-image, or SimpleITK.

[0052] Such code may include functional code related to functions defining the aforementioned functions, and may also include control code related to execution procedures required for the computer's processor to execute the aforementioned functions according to a predetermined procedure.

[0053] Additionally, such code may further include memory reference related code regarding where in the internal or external memory of the computer the additional information or media required for the computer's processor to execute the aforementioned functions should be referenced.

[0054] Additionally, if the computer's processor needs to communicate with another computer or server located remotely in order to execute the functions described above, the code may further include communication-related code regarding how the computer's processor should communicate with another computer or server located remotely using the computer's communication module, and what information or media should be sent and received during the communication.

[0055] The computer-readable recording medium that records the program as described above includes, for example, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical media storage device, etc., and may also include one implemented in the form of a carrier wave (e.g., transmission via the Internet).

[0056] Additionally, computer-readable recording media can be distributed across network-connected computer systems, allowing computer-readable code to be stored and executed in a distributed manner.

[0057] In addition, the functional program for implementing the present invention and the code and code segments related thereto may be easily inferred or changed by programmers in the technical field to which the present invention belongs, taking into consideration the system environment of the computer that reads the recording medium and executes the program.

[0058] The method for providing information on the presence of abnormalities in the aforementioned knee can also be implemented in the form of a recording medium containing computer-executable instructions, such as an application or program module executed by a computer. The computer-readable medium can be any available medium that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Furthermore, the computer-readable medium can include all computer storage media. The computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0059] The method for providing information on whether there is an abnormality in the knee can be executed by an application that is installed by default on the terminal (which may include a program included in the platform or operating system installed by default on the terminal), or by an application (i.e., a program) that the user directly installs on the master terminal through an application providing server such as an application store server, an application, or a web server related to the service. In this sense, the method for providing information on whether there is an abnormality in the knee described above can be implemented by an application (i.e., a program) that is installed by default on the terminal or directly installed by the user, and can be recorded on a computer-readable recording medium such as on the terminal.

[0060] The foregoing description of the present invention is for illustrative purposes only, and those skilled in the art will readily appreciate that the present invention can be readily modified into other specific forms without altering the technical spirit or essential characteristics of the present invention. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single entity may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined manner.

[0061] The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention.

[0062] The above description is merely an illustrative example of the technical idea of ​​the present disclosure, and those skilled in the art to which the present disclosure pertains will appreciate that various modifications and variations can be made without departing from the essential characteristics of the technical idea of ​​the present disclosure. In addition, the present embodiments are not intended to limit the technical idea of ​​the present disclosure but rather to explain it, and therefore the scope of the technical idea of ​​the present disclosure is not limited by these embodiments. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included within the scope of the rights of the present disclosure.

[0063]

[0064] CROSS-REFERENCE TO RELATED APPLICATION

[0065] This patent application claims priority under 35 USC §119(a) to Korean Patent Application No. 10-2024-0009260, filed in Korea on January 22, 2024, the entire contents of which are incorporated herein by reference. This patent application also claims priority in countries other than the United States for the same reasons, the entire contents of which are incorporated herein by reference.

Claims

1. In a method of providing information on whether there is an abnormality in the knee, A preprocessing step of extracting a first image set including skin contour information by processing multiple medical image data captured in the knee area; A step of creating a model for determining whether or not there is an abnormality in the knee by performing learning based on the first image set; A step of inputting a diagnostic image including skin contour information of the knee area of the subject into the above abnormality judgment model; and A step of outputting information on whether the knee of the subject is abnormal from the abnormality judgment model; including; How to provide information about whether there is an abnormality in the knee.

2. In claim 1, The above medical image data is a CT image, and the method provides information on whether there is an abnormality in the knee.

3. In claim 1, A method for providing information on whether or not a knee is abnormal, wherein the above-mentioned abnormality judgment model is generated by performing learning based on a plurality of the first images and a plurality of label data.

4. In claim 1, The above preprocessing step further includes a process of processing each of the plurality of medical image data to extract a second image set including bone information, and performing labeling on the second image set. A method for providing information on whether there is an abnormality in the knee, wherein the above abnormality judgment model is generated by performing learning based on the first image set and the labeled second image set.

5. In claim 1, A method for providing information on whether the subject has an abnormality in the knee, the information including information on the need for total knee replacement surgery for the subject.

6. A computer-readable storage medium recording a program that provides information on whether there is an abnormality in the knee, A preprocessing operation for extracting a first image set including skin contour information by processing multiple medical image data captured in the knee area; An operation of creating a model for determining whether or not a knee is abnormal by performing learning based on the first image set above; An operation of inputting a diagnostic image including skin contour information of the subject's knee area into the above abnormality judgment model; and An action of outputting information on whether the knee of the above subject is abnormal from the above abnormality judgment model; A computer-readable storage medium that records a program that executes on a computer.

7. In claim 6, The above medical image data is a computer-readable storage medium that records a program, which is a CT image.

8. In claim 6, A computer-readable storage medium recording a program, wherein the above-mentioned abnormality judgment model is generated by performing learning based on a plurality of the first images and a plurality of labeling data.

9. In claim 6, The above preprocessing operation further includes a process of processing each of the plurality of medical image data to extract a second image set including bone information, and performing labeling on the second image set. A computer-readable storage medium recording a program, wherein the above-mentioned abnormality judgment model is generated by performing learning based on the first image set and the labeled second image set.

10. In claim 6, A computer-readable storage medium having recorded thereon a program, which includes information on whether the subject has an abnormality in the knee, and information on the need for total knee replacement surgery for the subject.

11. In a device that provides information on whether there is an abnormality in the knee, A storage unit storing multiple medical image data taken of the knee area; and A control unit that processes each of the plurality of medical image data to extract a first image set including skin contour information, performs learning based on the first image set to create a model for determining whether there is an abnormality in the knee, inputs a diagnostic image including skin contour information in the knee area of the subject into the abnormality determination model, and outputs information on whether there is an abnormality in the knee of the subject from the abnormality determination model; A device that provides information on whether there is an abnormality in the knee, including:

Images