Method and device for diagnosing knee osteoarthritis based on multi-modal data
By inputting the patient's X-ray image and age information into the multimodal classification network model, combining feature extraction and age information, the problem of inaccurate diagnosis of knee osteoarthritis in the prior art is solved, and a more accurate knee bone is achieved Diagnosis of arthritis.
Patent Information
- Application Number
- CN202411012430.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-13
AI Technical Summary
The prior art cannot effectively diagnose knee osteoarthritis in combination with patient age factors, resulting in inaccurate diagnosis.
By obtaining the patient's positive and lateral X-ray images, and converting the patient's age value into image form, inputting the preset knee multimodal classification network model, the diagnosis of knee osteoarthritis is performed, combining image feature extraction and The age information is used to output the diagnostic results.
A more accurate diagnosis of knee osteoarthritis is achieved, which can effectively distinguish between normal, arthritis and degenerative lesions, and improve the accuracy of the diagnosis.
Smart Images
Figure CN119132559A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of diagnosis of knee osteoarthritis, and in particular to a diagnosis method, device, equipment and computer-readable storage medium for knee osteoarthritis based on multimodal data. Background Art
[0002] Knee osteoarthritis is a degenerative joint disease with a prevalence of 4 to 12%. The prevalence of knee osteoarthritis is increasing worldwide due to an aging population and obesity. This subsequently leads to an increasing demand for knee replacements. Therefore, the correct diagnosis and classification of knee osteoarthritis is very important for follow-up and planning of conservative or surgical treatment.
[0003] However, the current diagnosis of knee osteoarthritis in related technologies cannot comprehensively diagnose knee osteoarthritis in combination with the patient's age factor.
[0004] Therefore, how to diagnose knee osteoarthritis more accurately is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention
[0005] The embodiments of the present application provide a diagnosis method, device, equipment and computer-readable storage medium for knee osteoarthritis based on multimodal data, which can more accurately diagnose knee osteoarthritis.
[0006] In a first aspect, an embodiment of the present application provides a method for diagnosing knee osteoarthritis based on multimodal data, comprising:
[0007] Obtain anteroposterior and lateral X-ray images of the patient's knee joint;
[0008] Convert the patient's age value into an image form to obtain an age image;
[0009] The frontal X-ray image, lateral X-ray image and age image are input into a preset knee joint multimodal classification network model to output the diagnosis result of knee osteoarthritis;
[0010] Among them, the diagnosis results of knee osteoarthritis include: normal, arthritis and degenerative lesions.
[0011] Optionally, the patient's age value is converted into an image form to obtain an age image, including:
[0012] The patient's age value is converted into all pixel values in image form to obtain an age image.
[0013] Optionally, the patient's age value is converted into all pixel values in image form to obtain an age image, including:
[0014] Create a blank image with all pixel values initially set to 0;
[0015] Map the patient's age value to the blank image to obtain the age image.
[0016] Optionally, the frontal X-ray image, the lateral X-ray image and the age image are input into a preset knee joint multimodal classification network model to output the diagnosis results of knee osteoarthritis, including:
[0017] After superimposing the data of the frontal X-ray image, the lateral X-ray image and the age image, feature extraction is performed to obtain a first feature map;
[0018] Based on the first feature map, a diagnosis result of knee osteoarthritis is output.
[0019] Optionally, based on the first characteristic graph, a diagnosis result of knee osteoarthritis is output, including:
[0020] In branch one, feature extraction is performed on the first feature map and the frontal X-ray image to obtain a second feature map;
[0021] In branch 2, feature extraction is performed on the first feature map and the lateral X-ray image to obtain a third feature map;
[0022] In branch three, feature extraction is performed on the first feature map and the age image to obtain a fourth feature map;
[0023] The second feature map, the third feature map and the fourth feature map are fused to output the diagnosis result of knee osteoarthritis.
[0024] Optionally, the second feature map, the third feature map and the fourth feature map are fused to output a diagnosis result of knee osteoarthritis, including:
[0025] The second feature map, the third feature map and the fourth feature map are fused by pixel-by-pixel operation to obtain a fused image;
[0026] The fused image is input into the classifier and the diagnosis result of knee osteoarthritis is output.
[0027] Optional, calculation of the overall loss function, including:
[0028] Determine three loss functions and their corresponding weights respectively; wherein the three loss functions include: frontal image loss function, lateral image loss function and age image loss function;
[0029] Based on the three loss functions and their corresponding weights, the overall loss function is calculated.
[0030] In a second aspect, an embodiment of the present application provides a diagnostic device for knee osteoarthritis based on multimodal data, comprising:
[0031] An image acquisition module, used for acquiring an anteroposterior X-ray image and a lateral X-ray image of a patient's knee joint;
[0032] An image conversion module is used to convert the patient's age value into an image form to obtain an age image;
[0033] The osteoarthritis diagnosis module is used to input the frontal X-ray image, the lateral X-ray image and the age image into the preset knee joint multimodal classification network model, and output the diagnosis result of knee joint osteoarthritis;
[0034] Among them, the diagnosis results of knee osteoarthritis include: normal, arthritis and degenerative lesions.
[0035] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: a processor and a memory storing computer program instructions;
[0036] When the processor executes the computer program instructions, a diagnosis method for knee osteoarthritis based on multimodal data is implemented.
[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implements a method for diagnosing knee osteoarthritis based on multimodal data.
[0038] The diagnosis method, device, equipment and computer-readable storage medium for knee osteoarthritis based on multimodal data of the embodiments of the present application can diagnose knee osteoarthritis more accurately.
[0039] The diagnosis method for knee osteoarthritis based on multimodal data includes:
[0040] Obtain anteroposterior and lateral X-ray images of the patient's knee joint;
[0041] Convert the patient's age value into an image form to obtain an age image;
[0042] The frontal X-ray image, lateral X-ray image and age image are input into a preset knee joint multimodal classification network model to output the diagnosis result of knee osteoarthritis;
[0043] Among them, the diagnosis results of knee osteoarthritis include: normal, arthritis and degenerative lesions. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0045] Figure 1 It is a flowchart of a method for diagnosing knee osteoarthritis based on multimodal data provided by an embodiment of the present application;
[0046] Figure 2 It is a flowchart of a method for diagnosing knee osteoarthritis based on multimodal data provided by an embodiment of the present application;
[0047] Figure 3 is a schematic diagram of an age image provided by an embodiment of the present application;
[0048] Figure 4 is a schematic structural diagram of a device for diagnosing knee osteoarthritis based on multimodal data provided by an embodiment of the present application;
[0049] Figure 5 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0050] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0051] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0052] In order to solve the problems of the prior art, the embodiments of the present application provide a diagnosis method, device, equipment and computer-readable storage medium for knee osteoarthritis based on multimodal data. The diagnosis method for knee osteoarthritis based on multimodal data provided by the embodiments of the present application is first introduced below.
[0053] Figure 1 and Figure 2 FIG. 1 is a flow chart of a method for diagnosing knee osteoarthritis based on multimodal data provided by an embodiment of the present application. Figure 1 As shown, the diagnosis method of knee osteoarthritis based on multimodal data includes:
[0054] S101, obtaining an anteroposterior X-ray image and a lateral X-ray image of a patient's knee joint;
[0055] S102, converting the patient's age value into an image form to obtain an age image;
[0056] S103, inputting the frontal X-ray image, the lateral X-ray image and the age image into a preset multimodal classification network model of the knee joint, and outputting the diagnosis result of knee osteoarthritis; wherein the diagnosis result of knee osteoarthritis includes: normal, arthritis and degenerative lesions.
[0057] Specifically, the AP and lateral views of the knee are input, the knee position and lateral position of the knee are detected by YOLOV10, and the age in the X-ray is extracted. The age value is converted into an image, that is, the pixel value in the image is replaced with the corresponding age value, and the detected position is cropped. The cropped area is combined according to the left AP image, left lateral image, age image; right AP image, right lateral image, age image, and the image is converted into a 256*256 format and input into a multimodal classification network to output one of the three categories of results: knee arthritis, normal, and degenerative lesions.
[0058] In one embodiment, the patient's age value is converted into an image form to obtain an age image, including:
[0059] The patient's age value is converted into all pixel values in image form to obtain an age image.
[0060] In one embodiment, the patient's age value is converted into all pixel values in an image form to obtain an age image, including:
[0061] Create a blank image with all pixel values initially set to 0;
[0062] Map the patient's age value to the blank image to obtain the age image.
[0063] Specifically, since the input data is all images, the age is converted into an image form, that is, the pixel values in the image are replaced with the age value. For example, if the patient is 52 years old, the pixel values in the generated age map are all 52, and the corresponding age map is as follows Figure 3 shown.
[0064] In one embodiment, the frontal X-ray image, the lateral X-ray image and the age image are input into a preset knee joint multimodal classification network model to output a diagnosis result of knee osteoarthritis, including:
[0065] After superimposing the data of the frontal X-ray image, the lateral X-ray image and the age image, feature extraction is performed to obtain a first feature map;
[0066] Based on the first feature map, a diagnosis result of knee osteoarthritis is output.
[0067] In one embodiment, based on the first feature graph, outputting a diagnosis result of knee osteoarthritis includes:
[0068] In branch one, feature extraction is performed on the first feature map and the frontal X-ray image to obtain a second feature map;
[0069] In branch 2, feature extraction is performed on the first feature map and the lateral X-ray image to obtain a third feature map;
[0070] In branch three, feature extraction is performed on the first feature map and the age image to obtain a fourth feature map;
[0071] The second feature map, the third feature map and the fourth feature map are fused to output the diagnosis result of knee osteoarthritis.
[0072] In one embodiment, the second feature map, the third feature map, and the fourth feature map are fused to output a diagnosis result of knee osteoarthritis, including:
[0073] The second feature map, the third feature map and the fourth feature map are fused by pixel-by-pixel operation to obtain a fused image;
[0074] The fused image is input into the classifier and the diagnosis result of knee osteoarthritis is output.
[0075] Specifically, the knee joint multimodal classification network is composed of a fully convolutional network. Input knee joint frontal image, lateral image and age Figure 3 The images are converted into 256*256 format. The three images are first superimposed together for feature extraction, followed by three branches, each of which extracts features from the frontal image, lateral image and age image respectively.
[0076] The input features of branch one include feature extraction features and frontal image extraction features, the input features of branch two include feature extraction features and side image extraction features, and the input features of branch three include feature extraction features and age image extraction features. The features extracted from the three branches are then fused and output as the classification result.
[0077] In one embodiment, the calculation of the overall loss function includes:
[0078] Determine three loss functions and their corresponding weights respectively; wherein the three loss functions include: frontal image loss function, lateral image loss function and age image loss function;
[0079] Based on the three loss functions and their corresponding weights, the overall loss function is calculated.
[0080] Specifically, the loss function is:
[0081] Design three loss functions: the frontal image loss function Loss_z, the side image loss function Loss_c, and the age image loss function Loss_a.
[0082]
[0083] Among them, M is the category; i is the i-th sample; yic is the label of the i-th sample category c; pic is the probability of the i-th sample category c.
[0084] The comprehensive loss function Loss is as follows:
[0085] Loss=d*Loss_z+b*Loss_c+e*Loss_a
[0086] Among them, d, b, and e represent the weights of the loss function. Here, d=0.3, b=0.3, and e=0.4 are set, which means that the network training focuses more on the patient's age.
[0087] Figure 4: is a schematic diagram of the structure of a diagnosis device for knee osteoarthritis based on multimodal data provided by an embodiment of the present application; the diagnosis device for knee osteoarthritis based on multimodal data comprises:
[0088] An image acquisition module 401 is used to acquire an anteroposterior X-ray image and a lateral X-ray image of a patient's knee joint;
[0089] An image conversion module 402 is used to convert the patient's age value into an image form to obtain an age image;
[0090] The osteoarthritis diagnosis module 403 is used to input the frontal X-ray image, the lateral X-ray image and the age image into a preset knee joint multimodal classification network model, and output the diagnosis result of knee osteoarthritis; wherein the diagnosis result of knee osteoarthritis includes: normal, arthritis and degenerative lesions.
[0091] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown.
[0092] The electronic device may include a processor 501 and a memory 502 storing computer program instructions.
[0093] Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0094] The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 502 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 502 may be inside or outside the electronic device. In a particular embodiment, the memory 502 may be a non-volatile solid-state memory.
[0095] In one embodiment, the memory 502 may be a read-only memory (ROM). In one embodiment, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0096] The processor 501 reads and executes the computer program instructions stored in the memory 502 to implement any one of the diagnosis methods for knee osteoarthritis based on multimodal data in the above embodiments.
[0097] In one example, the electronic device may further include a communication interface 503 and a bus 510. Figure 5 As shown, the processor 501, the memory 502, and the communication interface 503 are connected via a bus 510 and communicate with each other.
[0098] The communication interface 503 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0099] Bus 510 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
[0100] In addition, in combination with the diagnosis method for knee osteoarthritis based on multimodal data in the above embodiments, the present application embodiment can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the diagnosis methods for knee osteoarthritis based on multimodal data in the above embodiments is implemented.
[0101] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0102] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. Programs or code segments can be stored in machine-readable media, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable media" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0103] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0104] The above reference is according to the method of the embodiment of the present application, the flow chart of the device (system) and the computer program product and / or the block diagram described various aspects of the present application.It should be understood that each square box in the flow chart and / or the block diagram and the combination of each square box in the flow chart and / or the block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the realization of the function / action specified in one or more square boxes of the flow chart and / or the block diagram.Such a processor can be but is not limited to a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit.It can also be understood that each square box in the block diagram and / or the flow chart and the combination of the square boxes in the block diagram and / or the flow chart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and computer instructions.
[0105] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A method for diagnosing knee osteoarthritis based on multimodal data, characterized in that: include: Obtain anteroposterior and lateral X-ray images of the patient's knee joint; Convert the patient's age value into an image form to obtain an age image; The frontal X-ray image, lateral X-ray image and age image are input into a preset knee joint multimodal classification network model to output the diagnosis result of knee osteoarthritis; Among them, the diagnosis results of knee osteoarthritis include: normal, arthritis and degenerative lesions.
2. The method for diagnosing knee osteoarthritis based on multimodal data according to claim 1, characterized in that: The patient's age value is converted into an image form to obtain an age image, including: The patient's age value is converted into all pixel values in image form to obtain an age image.
3. The method for diagnosing knee osteoarthritis based on multimodal data according to claim 2, characterized in that: The patient's age value is converted into all pixel values in image form to obtain an age image, including: Create a blank image with all pixel values initially set to 0; Map the patient's age value to the blank image to obtain the age image.
4. The method for diagnosing knee osteoarthritis based on multimodal data according to claim 1, characterized in that: The frontal X-ray image, lateral X-ray image and age image are input into the preset knee joint multimodal classification network model to output the diagnosis results of knee osteoarthritis, including: After superimposing the data of the frontal X-ray image, the lateral X-ray image and the age image, feature extraction is performed to obtain a first feature map; Based on the first feature map, a diagnosis result of knee osteoarthritis is output.
5. The method for diagnosing knee osteoarthritis based on multimodal data according to claim 4, characterized in that: Based on the first feature map, the diagnosis result of knee osteoarthritis is output, including: In branch one, feature extraction is performed on the first feature map and the frontal X-ray image to obtain a second feature map; In branch 2, feature extraction is performed on the first feature map and the lateral X-ray image to obtain a third feature map; In branch three, feature extraction is performed on the first feature map and the age image to obtain a fourth feature map; The second feature map, the third feature map and the fourth feature map are fused to output the diagnosis result of knee osteoarthritis.
6. The method for diagnosing knee osteoarthritis based on multimodal data according to claim 5, characterized in that: The second feature map, the third feature map and the fourth feature map are fused to output the diagnosis results of knee osteoarthritis, including: The second feature map, the third feature map and the fourth feature map are fused by pixel-by-pixel operation to obtain a fused image; The fused image is input into the classifier and the diagnosis result of knee osteoarthritis is output.
7. The method for diagnosing knee osteoarthritis based on multimodal data according to claim 1, characterized in that: The calculation of the overall loss function includes: Determine three loss functions and their corresponding weights respectively; wherein the three loss functions include: frontal image loss function, lateral image loss function and age image loss function; Based on the three loss functions and their corresponding weights, the overall loss function is calculated.
8. A diagnostic device for knee osteoarthritis based on multimodal data, characterized in that: The device comprises: An image acquisition module, used for acquiring an anteroposterior X-ray image and a lateral X-ray image of a patient's knee joint; An image conversion module is used to convert the patient's age value into an image form to obtain an age image; The osteoarthritis diagnosis module is used to input the frontal X-ray image, the lateral X-ray image and the age image into the preset knee joint multimodal classification network model, and output the diagnosis result of knee joint osteoarthritis; Among them, the diagnosis results of knee osteoarthritis include: normal, arthritis and degenerative lesions.
9. An electronic device, characterized in that: The electronic device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for diagnosing knee osteoarthritis based on multimodal data as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for diagnosing knee osteoarthritis based on multimodal data as described in any one of claims 1 to 7 is implemented.