Cartilage injury identification system and method based on multi-modal deep learning

Abstract

The invention relates to a cartilage injury identification system based on multi-modal deep learning, and relates to the field of cartilage injury identification, and the system comprises a deep learning mechanism which is used for executing deep learning operation on a feedforward neural network to obtain an intelligent injury identification model; and the intelligent identification mechanism is used for identifying the patella cartilage injury level of the current patient according to the multi-modal data of the current patient by adopting an intelligent injury identification model. The invention also relates to a cartilage injury identification method based on multi-modal deep learning. According to the invention, for the technical problem that the patella cartilage injury level of each patient lacks a targeted multi-modal intelligent identification model, customized visual information of various nuclear magnetic images and knee front images and multiple patella cartilage associated data of the current patient can be introduced as multi-modal data; an intelligent injury identification model which is customized and designed for a current patient and is subjected to deep learning is introduced as an artificial intelligence model, so that the technical problem is solved.

Description

Technical Field

[0001] The present invention relates to the field of cartilage injury identification, and particularly to a cartilage injury identification system and method based on multi-modal deep learning. Background Art

[0002] Patellar cartilage is a piece of cartilage located in front of the kneecap. Its main function is to reduce friction between bones, while providing cushioning and stability. Knee patellar cartilage injury is a common sports injury, usually caused by intense exercise or trauma.

[0003] Generally, the severity of knee patellar cartilage injury is usually divided into four levels:

[0004] 1. Grade 1 injury: Grade 1 knee patellar cartilage injury is usually referred to as cartilage edema. This type of injury is usually caused by overuse or exercise, resulting in partial damage and edema of the cartilage tissue. Patients may experience knee pain and discomfort, but the symptoms can usually be relieved through rest and physical therapy.

[0005] 2. Grade 2 injury: Grade 2 knee patellar cartilage injury is usually referred to as cartilage laceration. This type of injury is more severe than Grade 1 and is usually caused by intense exercise or trauma. Patients may experience obvious knee pain, swelling, and a sense of instability. Treatment usually includes physical therapy, local cold compress, and drug treatment.

[0006] 3. Grade 3 injury: Grade 3 knee patellar cartilage injury is usually referred to as cartilage tear. This type of injury involves severe damage and tearing of the cartilage tissue, resulting in the formation of fragments or flakes in the joint cavity. Patients may experience severe knee pain, swelling, and limited joint movement. Treatment usually requires surgical intervention to repair the injury.

[0007] 4. Grade 4 injury: Grade 4 knee patellar cartilage injury is usually referred to as cartilage wear. This type of injury involves severe wear of the cartilage tissue, resulting in direct contact and friction between the bones, causing pain, swelling, and severe functional impairment in the joint. Treatment usually requires joint replacement surgery to repair the damaged cartilage tissue.

[0008] Exemplarily, the Chinese invention patent publication text CN113392895A proposes a method and system for detecting knee joint cartilage damage. The method includes: acquiring knee joint MRI image information and preprocessing the image information; layer-by-layer extracting the cartilage damage area from the preprocessed image information to obtain the three-dimensional volume area of the knee joint cartilage; layer-by-layer extracting the region of interest of cartilage damage to obtain the three-dimensional area of cartilage damage; extracting the features of the three-dimensional region of interest, training a classifier network using a neural network method to determine the optimal model; inputting the knee joint MRI image information into the optimal model to perform cartilage damage detection and determine the degree of cartilage damage. It has good robustness, is fast and convenient, and is suitable for assisting doctors in diagnosis.

[0009] Exemplarily, the Chinese invention patent publication text CN109741352A proposes a method for segmenting cartilage damage repair images based on multimodal magnetic resonance. The method includes the following steps: S1: extracting the cartilage detection area of the cartilage damage repair image of multimodal magnetic resonance; S2: acquiring the MRI image in the cartilage detection area of the cartilage damage repair image of multimodal magnetic resonance; S3: performing edge detection on the grayscale image; S4: based on SVM edge classification; S5: based on SVM edge segmentation. In the present invention, after extracting the cartilage detection area of the cartilage damage repair image of multimodal magnetic resonance, SVM edge classification and segmentation are performed, achieving the purpose of segmenting the cartilage damage repair image based on multimodal magnetic resonance, which is of great significance and can enable relevant personnel to judge the cartilage damage repair image more precisely and correctly.

[0010] However, the above-mentioned cartilage damage recognition solutions either do not involve patellar cartilage damage recognition and do not use multimodal data, or only use multimodal data to repair and subdivide nuclear magnetic resonance images without performing patellar cartilage damage recognition, resulting in a lack of multimodal data support in the subfield of intelligent recognition of patellar cartilage damage in the prior art, and at the same time lacking a reliable artificial intelligence model. Furthermore, in the prior art, it is impossible to quickly and effectively provide the degree of patellar cartilage damage of each patient in a non-destructive manner, and naturally it is impossible to provide each patient with a corresponding reliable injury treatment strategy. Summary of the Invention

[0011] In order to solve the technical problems in the prior art, the present invention provides a cartilage injury identification system and method based on multimodal deep learning, by introducing a variety of targeted screened nuclear magnetic resonance images and customized visual information of the front image of the knee and multiple patellar cartilage related data of the current patient as multimodal data for patellar cartilage injury level identification, and at the same time introducing an intelligent injury recognition model customized for the current patient and after deep learning as an artificial intelligence model for patellar cartilage injury level identification, to complete the reliable identification of the patellar cartilage injury level of the current patient, and then adopt corresponding injury treatment strategies, thereby improving the timeliness and effectiveness of patellar cartilage injury treatment.

[0012] According to one aspect of the present invention, a cartilage injury identification system based on multimodal deep learning is provided, the system comprising:

[0013] The first analysis mechanism is used to obtain the T1WI-SAG positioning scan screen, PDW-SAG positioning scan screen, PDW-COR positioning scan screen and T2WI-SPAI R-TRA positioning scan screen of the knee joint of the current patient after the nuclear magnetic resonance scan, and then obtain the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image and T2WI-SPAI R-TRA cartilage image respectively based on the imaging characteristics of the patellar cartilage;

[0014] A second analysis mechanism is used to obtain a knee occupancy image in a frontal imaging picture of the knee of the current patient;

[0015] The third analysis unit is used to obtain the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data and weight data of the current patient as multiple patellar cartilage related data of the current patient;

[0016] A deep learning mechanism, used for performing a deep learning operation on the feedforward neural network exceeding a set number of learning times to obtain a feedforward neural network after the deep learning operation, and outputting the feedforward neural network as an intelligent injury recognition model, wherein the number of learning times in the deep learning operation of the feedforward neural network is monotonically positively correlated with the age data of the current patient;

[0017] The intelligent recognition mechanism is connected to the first analysis mechanism, the second analysis mechanism, the third analysis mechanism and the deep learning mechanism respectively, and is used to adopt an intelligent injury recognition model to intelligently identify the patellar cartilage injury level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, the PDW-SAG cartilage image, the PDW-COR cartilage image, the T2WI-SPAI R-TRA cartilage image and the knee occupancy image, as well as multiple patellar cartilage-related data of the current patient;

[0018] Among them, the customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image, as well as the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image.

[0019] According to another aspect of the present invention, a method for identifying cartilage damage based on multi-modal deep learning is provided, and the method includes:

[0020] Obtain the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAI R-TRA localization scan image obtained after magnetic resonance scanning of the knee joint of the current patient, and then respectively obtain the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, and T2WI-SPAI R-TRA cartilage image based on the patellar cartilage imaging characteristics;

[0021] Obtain the knee occupancy image in the frontal imaging image of the knee of the current patient;

[0022] Obtain the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data, and weight data of the current patient as multiple pieces of patellar cartilage-related data of the current patient;

[0023] Perform deep learning operations on the feedforward neural network for more than a set number of learning times to obtain the feedforward neural network after the deep learning operations, and output it as an intelligent damage recognition model. The number of learning times in the deep learning operations passed by the feedforward neural network is monotonically positively correlated with the age data of the current patient;

[0024] Use the intelligent damage recognition model to intelligently identify the patellar cartilage damage level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image, as well as the multiple pieces of patellar cartilage-related data of the current patient;

[0025] Among them, the customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image, as well as the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image.

[0026] Thus, the present invention at least has the following key inventive points:

[0027] The first inventive point: The artificial intelligence model obtained after designing deep learning uses the multimodal data of the current patient screened specifically to complete the intelligent identification of the patellar cartilage injury level of the current patient. The artificial intelligence model obtained after deep learning is an intelligent injury identification model. The multimodal data includes the customized visual information of the patellar cartilage imaging area in the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAI R-TRA localization scan image obtained after magnetic resonance scanning, the customized visual information of the knee occupancy image in the frontal imaging image of the current patient's knee, and a number of patellar cartilage-related data of the current patient. The introduction of multimodal data enriches the integrity of the basic data for intelligent identification, and the artificial intelligence model obtained through deep learning improves the pertinence of intelligent identification;

[0028] The second inventive point: The intelligent injury identification model for intelligent cartilage injury identification has the following customized structural designs. The intelligent injury identification model is a feedforward neural network after deep learning operations and the number of learning times exceeds the set number of learning times. Particularly crucial is that the number of learning times in the deep learning operations passed by the feedforward neural network is monotonically and positively correlated with the age data of the current patient, thereby improving the pertinence of intelligent identification;

[0029] The third inventive point: The multimodal data for intelligent cartilage injury identification specifically includes the customized visual information corresponding to each of the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image, and a number of patellar cartilage-related data of the current patient. The customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image and the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image. Obtain the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAI R-TRA localization scan image of the current patient's knee joint after magnetic resonance scanning, and then respectively obtain the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, and T2WI-SPAI R-TRA cartilage image based on the patellar cartilage imaging characteristics, and a number of patellar cartilage-related data of the current patient includes the duration of patellar cartilage injury in the current patient, gender data, thigh length data, calf length data, age data, and weight data. The introduction of the above multimodal data enriches the integrity of the basic data for intelligent identification;

[0030] Fourth inventive point: Each pixel point in the image is used as a target pixel point. The gray value sets corresponding to the target pixel point are composed of the gray value of each adjacent pixel point adjacent to the target pixel point and the single gray value of the target pixel point. The mean square error of all gray values in the gray value set corresponding to the target pixel point is used as the gray value gradient data corresponding to the target pixel point, thereby effectively calculating the gray value gradient data of each pixel point;

[0031] Fifth inventive point: In each learning of the feedforward neural network, the known patellar cartilage injury level of a certain patient is used as the single output content of the feedforward neural network. The customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image and knee occupancy image of the certain patient and the multiple patellar cartilage related data of the certain patient are used as the item-by-item input content of the feedforward neural network, and the current learning of the feedforward neural network is completed, thereby ensuring the learning effect of each learning of the feedforward neural network. Brief Description of the Drawings

[0032] The embodiments of the present invention will be described below with reference to the drawings, where:

[0033] Figure 1 It is a schematic diagram of the working principle of a cartilage injury recognition system and method based on multi-modal deep learning according to the present invention.

[0034] Figure 2 It is an internal structure diagram of a cartilage injury recognition system based on multi-modal deep learning shown in the first embodiment of the present invention.

[0035] Figure 3 It is an internal structure diagram of a cartilage injury recognition system based on multi-modal deep learning shown in the second embodiment of the present invention.

[0036] Figure 4 It is an internal structure diagram of a cartilage injury recognition system based on multi-modal deep learning shown in the third embodiment of the present invention.

[0037] Figure 5 It is an internal structure diagram of a cartilage injury recognition system based on multi-modal deep learning shown in the fourth embodiment of the present invention.

[0038] Figure 6 It is an internal structure diagram of a cartilage injury recognition system based on multi-modal deep learning shown in the fifth embodiment of the present invention.

[0039] Figure 7It is a flowchart of the steps of a cartilage injury recognition method based on multimodal deep learning shown in the sixth embodiment of the present invention. Detailed implementation manners

[0040] As Figure 1 shown, a schematic diagram of the working principle of a cartilage injury recognition system and method based on multimodal deep learning shown in the present invention is given.

[0041] The specific technical process of the present invention is as follows:

[0042] Technical process A: To complete the intelligent recognition of the patellar cartilage injury level of the current patient, an artificial intelligence model obtained after deep learning is designed;

[0043] Specifically, the artificial intelligence model obtained after deep learning is an intelligent injury recognition model. The intelligent injury recognition model is a feedforward neural network after deep learning operations and the number of learning times exceeds the set number of learning times, thus completing deep learning;

[0044] And specifically, the number of learning times in the deep learning operations passed by the feedforward neural network is monotonically positively correlated with the age data of the current patient, so as to customize artificial intelligence models with different structures for different patients;

[0045] And specifically, in each learning performed on the feedforward neural network, the known patellar cartilage injury level of a certain patient is used as the single-item output content of the feedforward neural network, and the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image and knee occupancy image of the certain patient respectively, as well as the multiple patellar cartilage-related data of the certain patient are used as the item-by-item input content of the feedforward neural network, completing the current learning performed on the feedforward neural network, thus ensuring the learning effect of each learning of the feedforward neural network;

[0046] In this way, the artificial intelligence model obtained after deep learning improves the pertinence of intelligent recognition;

[0047] Technical process B: To complete the intelligent recognition of the patellar cartilage injury level of the current patient, the multimodal data of the current patient screened specifically;

[0048] Specifically, the multimodal data includes the customized visual information of the patellar cartilage imaging area in the T1WI-SAG positioning scan picture, PDW-SAG positioning scan picture, PDW-COR positioning scan picture and T2WI-SPAI R-TRA positioning scan picture obtained after magnetic resonance scanning, the customized visual information of the knee occupancy image in the front imaging picture of the current patient's knee, and the multiple patellar cartilage-related data of the current patient;

[0049] As Figure 1 shown, the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAIR-TRA localization scan image are illustrated by T1WI-SAG, PDW-SAG, PDW-COR, and T2WI-SPAIR-TRA respectively;

[0050] More specifically, the multi-modal data for intelligent identification of cartilage damage specifically includes respective customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image, as well as multiple patellar cartilage-related data of the current patient;

[0051] More specifically, the customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image, as well as the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image;

[0052] Exemplarily, each pixel point in the image is used as the target pixel point, and the gray value gradient data corresponding to the target pixel point is formed by combining the gray values of each adjacent pixel point adjacent to the target pixel point and the single gray value of the target pixel point. The mean square error of all gray values in the gray value set corresponding to the target pixel point is used as the gray value gradient data corresponding to the target pixel point, thereby effectively calculating the gray value gradient data of each pixel point;

[0053] Exemplarily, the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAIR-TRA localization scan image obtained after magnetic resonance scanning of the knee joint of the current patient are acquired, and then the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, and T2WI-SPAIR-TRA cartilage image are respectively obtained based on the patellar cartilage imaging characteristics;

[0054] More specifically, the multiple patellar cartilage-related data of the current patient includes the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data, and weight data of the current patient. The introduction of the above multi-modal data enriches the integrity of the basic data for intelligent identification;

[0055] In this way, the introduction of the above multi-modal data enriches the integrity of the basic data for intelligent identification;

[0056] Technical process C: The intelligent damage recognition model after deep learning customized for the current patient using technical process A completes the intelligent recognition of the patellar cartilage damage level of the current patient based on the multimodal data of the current patient screened specifically according to technical process B;

[0057] Exemplarily, the severity of knee patellar cartilage damage is usually divided into four levels: grade 1 injury, grade 2 injury, grade 3 injury, and grade 4 injury;

[0058] Technical process D: Based on the patellar cartilage damage level of the current patient intelligently recognized by technical process C, formulate corresponding damage response strategies for the current patient;

[0059] Exemplarily, when the patellar cartilage damage level of the current patient intelligently recognized is grade 2 injury, the corresponding damage response strategies formulated for the current patient include physical therapy, local cold compress, and drug treatment.

[0060] Thus, the present invention completes the reliable recognition of the patellar cartilage damage level of the current patient by introducing customized visual information of a variety of nuclear magnetic images and knee frontal images screened specifically and a number of patellar cartilage-related data of the current patient as multimodal data for patellar cartilage damage level recognition, and at the same time introducing an intelligent damage recognition model customized for the current patient and after deep learning as an artificial intelligence model for patellar cartilage damage level recognition, and then taking corresponding damage treatment strategies, thereby improving the timeliness and effectiveness of patellar cartilage damage treatment.

[0061] The key points of the present invention are: the multimodal data of the current patient screened specifically to complete the intelligent recognition of the patellar cartilage damage level, the intelligent damage recognition model after deep learning customized for the current patient to complete the intelligent recognition of the patellar cartilage damage level, the customized design of each learning mode of the feedforward neural network, and the dynamic formulation of damage treatment strategies based on the intelligent recognition results.

[0062] Next, a cartilage damage recognition system and method based on multimodal deep learning of the present invention will be specifically described by way of examples.

[0063] First Embodiment

[0064] Figure 2 The internal structure diagram of the cartilage damage recognition system based on multimodal deep learning shown according to the first embodiment of the present invention.

[0065] As Figure 2 shown, the cartilage damage recognition system based on multimodal deep learning includes the following components:

[0066] The first analysis mechanism is used to obtain the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAI R-TRA localization scan image of the knee joint of the current patient after magnetic resonance scanning, and then respectively obtain the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, and T2WI-SPA I R-TRA cartilage image based on the patellar cartilage imaging characteristics;

[0067] Exemplarily, the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAI R-TRA localization scan image of the knee joint of the current patient after magnetic resonance scanning can be scan images with the same resolution;

[0068] The second analysis mechanism is used to obtain the knee occupancy image in the frontal imaging image of the knee of the current patient;

[0069] Exemplarily, a movable camera mechanism with a built-in optoelectronic sensor, flexible circuit board, and imaging lens can be used to obtain the frontal imaging image of the knee of the current patient;

[0070] The third analysis mechanism is used to obtain the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data, and weight data of the current patient as multiple pieces of patellar cartilage-related data of the current patient;

[0071] Specifically, multiple different data acquisition components can be used to respectively obtain the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data, and weight data of the current patient as multiple pieces of patellar cartilage-related data of the current patient;

[0072] The deep learning mechanism is used to perform deep learning operations on the feedforward neural network for more than the set number of learning times to obtain the feedforward neural network after the deep learning operation, and output it as an intelligent injury recognition model. The number of learning times in the deep learning operation passed by the feedforward neural network is monotonically positively correlated with the age data of the current patient;

[0073] Exemplarily, the MATLAB toolbox can be selected to complete the test and simulation of the model construction process of performing deep learning operations on the feedforward neural network for more than the set number of learning times to obtain the feedforward neural network after the deep learning operation and output it as an intelligent injury recognition model;

[0074] Specifically, perform deep learning operations on the feedforward neural network for more than a set number of learning times to obtain the feedforward neural network after the deep learning operations, and output it as an intelligent injury recognition model. The number of learning times in the deep learning operations passed by the feedforward neural network has a monotonically positive correlation with the age data of the current patient, including: setting the number of learning times to 400, the age data of the current patient is 10, the number of learning times in the deep learning operations passed by the feedforward neural network is 500, the age data of the current patient is 20, the number of learning times in the deep learning operations passed by the feedforward neural network is 600, the age data of the current patient is 30, the number of learning times in the deep learning operations passed by the feedforward neural network is 700, the age data of the current patient is 40, the number of learning times in the deep learning operations passed by the feedforward neural network is 800, and so on;

[0075] It can be seen that the selected number of learning times above all exceed the set number of learning times, that is, 400 times;

[0076] The intelligent recognition mechanism is respectively connected to the first analysis mechanism, the second analysis mechanism, the third analysis mechanism, and the deep learning mechanism, and is used to intelligently identify the patellar cartilage injury level of the current patient by using the intelligent injury recognition model according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, as well as multiple patellar cartilage-related data of the current patient;

[0077] Specifically, using the intelligent injury recognition model to intelligently identify the patellar cartilage injury level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, as well as multiple patellar cartilage-related data of the current patient, includes: the patellar cartilage injury level of the current patient obtained by intelligent recognition is one of the four injury levels;

[0078] Among them, the customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image, as well as the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image;

[0079] Among them, in each learning process of the feedforward neural network, the known patellar cartilage injury level of a certain patient is used as the single output content of the feedforward neural network, and the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image and knee occupancy image of the certain patient, as well as the multiple pieces of patellar cartilage-related data of the certain patient are used as the item-by-item input content of the feedforward neural network to complete the current learning of the feedforward neural network;

[0080] Among them, based on the patellar cartilage imaging features, the patellar cartilage occupancy areas in the T1WI-SAG positioning scan image, PDW-SAG positioning scan image, PDW-COR positioning scan image and T2WI-SPAI R-TRA positioning scan image are respectively identified and used as the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image and T2WI-SPAI R-TRA cartilage image;

[0081] Among them, the patellar cartilage imaging feature is the upper gray level threshold and the lower gray level threshold corresponding to the human patellar cartilage in the nuclear magnetic resonance scan image. The pixel points with gray level values between the upper gray level threshold and the lower gray level threshold in the nuclear magnetic resonance scan image are used as a patellar cartilage pixel point in the nuclear magnetic resonance scan image. The isolated pixel points of the patellar cartilage pixel points in the nuclear magnetic resonance scan image are removed to obtain multiple patellar cartilage pixel points in the nuclear magnetic resonance scan image, and the multiple patellar cartilage pixel points in the nuclear magnetic resonance scan image are subjected to fitting processing to obtain the patellar cartilage occupancy area in the nuclear magnetic resonance scan image;

[0082] Specifically, the patellar cartilage imaging feature being the upper gray level threshold and the lower gray level threshold corresponding to the human patellar cartilage in the nuclear magnetic resonance scan image includes: the upper gray level threshold corresponding to the human patellar cartilage in the nuclear magnetic resonance scan image is greater than the lower gray level threshold corresponding to the human patellar cartilage in the nuclear magnetic resonance scan image, and the values of the upper gray level threshold corresponding to the human patellar cartilage in the nuclear magnetic resonance scan image and the lower gray level threshold corresponding to the human patellar cartilage in the nuclear magnetic resonance scan image are both within the range of 0 - 255;

[0083] Among them, the nuclear magnetic resonance scan image includes the T1WI-SAG positioning scan image, PDW-SAG positioning scan image, PDW-COR positioning scan image and T2WI-SPAI R-TRA positioning scan image;

[0084] Among them, the patellar cartilage is located behind the patellar cartilage, smoothly covers the back of the patellar cartilage and is connected to the femoral patellar surface;

[0085] Among them, T1WI-SAG is T1-weighted imaging - sagittal plane, PDW-SAG is proton density weighted imaging - sagittal plane, PDW-COR is the coronal scan plane, and T2WI-SPAI R-TRA is the axial scan plane;

[0086] Among them, the frontal imaging picture of the current patient's knee is the imaging picture obtained when the imaging lens of the visible light camera faces the front of the current patient's knee, and the front of the current patient's knee is the side of the current patient's knee away from the current patient's thigh when the current patient's thigh and calf are in a vertical state;

[0087] In this way, by obtaining the relevant visual information of the front of the current patient's knee, data can be provided for the distribution state of the current patient's knee, providing a basic data for the intelligent identification of the injury level of the patellar cartilage in the follow-up;

[0088] And among them, obtaining the knee occupancy image in the frontal imaging picture of the current patient's knee, the frontal imaging picture of the current patient's knee is the imaging picture obtained when the imaging lens of the visible light camera faces the front of the current patient's knee, including: identifying the knee occupancy image in the frontal imaging picture of the current patient's knee based on the external shape reference imaging pattern of the human knee.

[0089] Second Embodiment

[0090] Figure 3 It is the internal structure diagram of the cartilage injury recognition system based on multi-modal deep learning shown in the second embodiment of the present invention.

[0091] As Figure 3 shown, compared with Figure 2 , the cartilage injury recognition system based on multi-modal deep learning further includes:

[0092] A mobile communication mechanism, connected to the intelligent recognition mechanism, for wirelessly transmitting the injury level of the current patient's patellar cartilage to a remote sports injury management server through a mobile communication link;

[0093] Exemplarily, a mobile communication mechanism, connected to the intelligent recognition mechanism, for wirelessly transmitting the injury level of the current patient's patellar cartilage to a remote sports injury management server through a mobile communication link includes: the mobile communication link is based on a frequency division duplex communication link or a time division duplex communication link.

[0094] Third Embodiment

[0095] Figure 4 It is the internal structure diagram of the cartilage injury recognition system based on multi-modal deep learning shown in the third embodiment of the present invention.

[0096] AsFigure 4 As shown, compared with Figure 2 , the cartilage injury recognition system based on multimodal deep learning further includes:

[0097] A strategy selection mechanism, connected to the intelligent recognition mechanism, for receiving the patellar cartilage injury level of the current patient and providing an injury response strategy corresponding to the patellar cartilage injury level of the current patient;

[0098] Specifically, a programmable logic device can be selected to implement the strategy selection mechanism, which is connected to the intelligent recognition mechanism, for receiving the patellar cartilage injury level of the current patient and providing an injury response strategy corresponding to the patellar cartilage injury level of the current patient.

[0099] Fourth Embodiment

[0100] Figure 5 FIG. is an internal structure diagram of a cartilage injury recognition system based on multimodal deep learning according to the fourth embodiment of the present invention.

[0101] As Figure 5 shown, compared with Figure 2 , the cartilage injury recognition system based on multimodal deep learning further includes:

[0102] A data storage mechanism, connected to the deep learning mechanism, for storing the model data of the intelligent injury recognition model to complete the model storage of the intelligent injury recognition model;

[0103] Specifically, a TF storage chip or a FLASH storage chip can be selected to implement the data storage mechanism, which is connected to the deep learning mechanism, for storing the model data of the intelligent injury recognition model to complete the model storage of the intelligent injury recognition model.

[0104] Fifth Embodiment

[0105] Figure 6 FIG. is an internal structure diagram of a cartilage injury recognition system based on multimodal deep learning according to the fifth embodiment of the present invention.

[0106] As Figure 6 shown, compared with Figure 2 , the cartilage injury recognition system based on multimodal deep learning further includes:

[0107] A real-time display mechanism, respectively connected to the intelligent recognition mechanism and the third analysis mechanism, for receiving the patellar cartilage injury level of the current patient and synchronously displaying the patellar cartilage injury level of the current patient and multiple pieces of patellar cartilage-related data of the current patient;

[0108] Exemplarily, the real-time display mechanism is connected to the intelligent recognition mechanism and the third parsing mechanism respectively, and is used to receive the patellar cartilage injury level of the current patient, and synchronously display the patellar cartilage injury level of the current patient and a plurality of patellar cartilage associated data of the current patient, including: the real-time display mechanism is a liquid crystal display screen or an LCD display screen.

[0109] Next, the various embodiments of the present invention will be further described.

[0110] Optionally, in each of the above embodiments, in the cartilage injury recognition system based on multi-modal deep learning:

[0111] Perform deep learning operations on the feedforward neural network for more than a set number of learning times to obtain the feedforward neural network after the deep learning operation, and output it as the intelligent injury recognition model. The number of learning times in the deep learning operation passed by the feedforward neural network is monotonically and positively correlated with the age data of the current patient, including: using a signal conversion function to represent the signal conversion relationship between the number of learning times in the deep learning operation passed by the feedforward neural network and the age data of the current patient as monotonically and positively correlated;

[0112] Exemplarily, using a signal conversion function to represent the signal conversion relationship between the number of learning times in the deep learning operation passed by the feedforward neural network and the age data of the current patient as monotonically and positively correlated includes: selecting to use a numerical simulation mode to complete the test and simulation of using a signal conversion function to represent the signal conversion relationship between the number of learning times in the deep learning operation passed by the feedforward neural network and the age data of the current patient as monotonically and positively correlated;

[0113] Among them, using a signal conversion function to represent the signal conversion relationship between the number of learning times in the deep learning operation passed by the feedforward neural network and the age data of the current patient as monotonically and positively correlated includes: in the signal conversion function, the age data of the current patient is the input signal of the signal conversion function;

[0114] And among them, using a signal conversion function to represent the signal conversion relationship between the number of learning times in the deep learning operation passed by the feedforward neural network and the age data of the current patient as monotonically and positively correlated further includes: the number of learning times in the deep learning operation passed by the feedforward neural network corresponding to the age data of the current patient is the output signal of the signal conversion function.

[0115] Optionally, in each of the above embodiments, in the cartilage injury recognition system based on multi-modal deep learning:

[0116] An intelligent injury recognition model is used to intelligently recognize the patellar cartilage injury level of the current patient based on the respective customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image, as well as multiple pieces of patellar cartilage-related data of the current patient, including: performing numerical normalization processing on the respective customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image, and multiple pieces of patellar cartilage-related data of the current patient respectively, and then parallelly inputting them into the intelligent injury recognition model;

[0117] Specifically, performing numerical normalization processing on the respective customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image, and multiple pieces of patellar cartilage-related data of the current patient respectively, and then parallelly inputting them into the intelligent injury recognition model includes: performing resolution normalization processing on the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and then performing numerical normalization processing respectively;

[0118] Among them, using the intelligent injury recognition model to intelligently recognize the patellar cartilage injury level of the current patient based on the respective customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image, as well as multiple pieces of patellar cartilage-related data of the current patient, also includes: running the intelligent injury recognition model to obtain the patellar cartilage injury level of the current patient output by the intelligent injury recognition model;

[0119] Among them, performing numerical normalization processing on the respective customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image, and multiple pieces of patellar cartilage-related data of the current patient respectively, and then parallelly inputting them into the intelligent injury recognition model includes: using a numerical processing device to complete the numerical normalization processing performed on the respective customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image, and multiple pieces of patellar cartilage-related data of the current patient respectively;

[0120] Among them, after performing numerical normalization processing on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, and multiple patellar cartilage-related data of the current patient, and then inputting them in parallel to the intelligent injury recognition model, it further includes: using a parallel control device to complete the parallel input of each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, and multiple patellar cartilage-related data of the current patient after performing numerical normalization processing to the intelligent injury recognition model;

[0121] Exemplarily, using a parallel control device to complete the parallel input of each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, and multiple patellar cartilage-related data of the current patient after performing numerical normalization processing to the intelligent injury recognition model includes: the parallel control device may include a control chip, a signal receiving end, and a signal control end;

[0122] Among them, using a numerical processing device to complete the numerical normalization processing performed on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, and multiple patellar cartilage-related data of the current patient includes: the numerical processing device is an FPGA chip designed in VHDL language;

[0123] Among them, using a numerical processing device to complete the numerical normalization processing performed on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, and multiple patellar cartilage-related data of the current patient further includes: the numerical normalization processing is an octal numerical conversion processing;

[0124] Among them, using a parallel control device to complete the parallel input of each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, and multiple patellar cartilage-related data of the current patient after performing numerical normalization processing to the intelligent injury recognition model includes: the parallel control device is an ASI C chip and is connected to the numerical processing device;

[0125] And among them, in each learning process performed on the feedforward neural network, the known patellar cartilage injury level of a certain patient is used as the single output content of the feedforward neural network, and each customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image and knee occupancy image of the certain patient, as well as multiple pieces of patellar cartilage-related data of the certain patient, are used as the item-by-item input content of the feedforward neural network. Completing the current learning process performed on the feedforward neural network includes: using the numerical simulation mode to complete the test and simulation of the data processing process of each learning process performed on the feedforward neural network.

[0126] Sixth Embodiment

[0127] Figure 7 It is a flowchart of the steps of a cartilage injury recognition method based on multi-modal deep learning shown in the sixth embodiment of the present invention.

[0128] As Figure 7 shown, the cartilage injury recognition method based on multi-modal deep learning includes the following steps:

[0129] Step 701: Obtain the T1WI-SAG positioning scan image, PDW-SAG positioning scan image, PDW-COR positioning scan image, and T2WI-SPAI R-TRA positioning scan image obtained after magnetic resonance scanning of the knee joint of the current patient, and then respectively obtain the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, and T2WI-SPAI R-TRA cartilage image based on the patellar cartilage imaging characteristics;

[0130] Exemplarily, the T1WI-SAG positioning scan image, PDW-SAG positioning scan image, PDW-COR positioning scan image, and T2WI-SPAI R-TRA positioning scan image obtained after magnetic resonance scanning of the knee joint of the current patient can be scan images with the same resolution;

[0131] Step 702: Obtain the knee occupancy image in the frontal imaging image of the knee of the current patient;

[0132] Exemplarily, the acquisition of the frontal imaging image of the knee of the current patient can be completed by a movable camera mechanism with a built-in optoelectronic sensor, flexible circuit board, and imaging lens;

[0133] Step 703: Obtain the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data, and weight data of the current patient as multiple pieces of patellar cartilage-related data of the current patient;

[0134] Specifically, multiple different data acquisition components can be adopted to respectively obtain the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data, and weight data of the current patient as multiple pieces of patellar cartilage-related data of the current patient;

[0135] Step 704: Perform deep learning operations on the feedforward neural network for more than the set number of learning times to obtain the feedforward neural network after the deep learning operations, and output it as the intelligent injury recognition model. The number of learning times in the deep learning operations passed by the feedforward neural network is monotonically positively correlated with the age data of the current patient;

[0136] Exemplarily, the MATLAB toolbox can be selected to complete the test and simulation of the model construction process of performing deep learning operations on the feedforward neural network for more than the set number of learning times to obtain the feedforward neural network after the deep learning operations, and output it as the intelligent injury recognition model;

[0137] Specifically, performing deep learning operations on the feedforward neural network for more than the set number of learning times to obtain the feedforward neural network after the deep learning operations, and outputting it as the intelligent injury recognition model. The number of learning times in the deep learning operations passed by the feedforward neural network is monotonically positively correlated with the age data of the current patient includes: setting the number of learning times to 400, the age data of the current patient to 10, the number of learning times in the deep learning operations passed by the feedforward neural network to 500, the age data of the current patient to 20, the number of learning times in the deep learning operations passed by the feedforward neural network to 600, the age data of the current patient to 30, the number of learning times in the deep learning operations passed by the feedforward neural network to 700, the age data of the current patient to 40, the number of learning times in the deep learning operations passed by the feedforward neural network to 800, and so on;

[0138] It can be seen that the selected number of learning times above all exceeds the set number of learning times, that is, 400 times;

[0139] Step 705: Use the intelligent injury recognition model to intelligently identify the patellar cartilage injury level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and the multiple pieces of patellar cartilage-related data of the current patient;

[0140] Specifically, an intelligent injury recognition model is used to intelligently recognize the patellar cartilage injury level of the current patient based on the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient, including: the patellar cartilage injury level of the current patient obtained by intelligent recognition is one of the four injury levels;

[0141] Among them, the customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image, as well as the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image;

[0142] Among them, in each learning process of the feedforward neural network, the known patellar cartilage injury level of a certain patient is used as the single output content of the feedforward neural network, and the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAI R-TRA cartilage image, and knee occupancy image of the certain patient, as well as multiple pieces of patellar cartilage-related data of the certain patient, are used as the item-by-item input content of the feedforward neural network to complete the current learning of the feedforward neural network;

[0143] Among them, based on the patellar cartilage imaging characteristics, the patellar cartilage occupancy areas in the T1WI-SAG positioning scan image, PDW-SAG positioning scan image, PDW-COR positioning scan image, and T2WI-SPAI R-TRA positioning scan image are respectively identified and used as the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, and T2WI-SPAI R-TRA cartilage image;

[0144] Among them, the patellar cartilage imaging characteristics are the upper gray threshold and lower gray threshold corresponding to the human patellar cartilage in the nuclear magnetic resonance scan image. The pixel points with gray values in the nuclear magnetic resonance scan image between the upper gray threshold and the lower gray threshold are used as a patellar cartilage pixel point in the nuclear magnetic resonance scan image. The isolated pixel points of each patellar cartilage pixel point in the nuclear magnetic resonance scan image are removed to obtain multiple patellar cartilage pixel points in the nuclear magnetic resonance scan image. The multiple patellar cartilage pixel points in the nuclear magnetic resonance scan image are subjected to fitting processing to obtain the patellar cartilage occupancy area in the nuclear magnetic resonance scan image;

[0145] Specifically, the imaging features of the patellar cartilage are that the upper gray-scale threshold and the lower gray-scale threshold corresponding to the human patellar cartilage in the magnetic resonance imaging (MRI) scan image include: the upper gray-scale threshold corresponding to the human patellar cartilage in the MRI scan image is greater than the lower gray-scale threshold corresponding to the human patellar cartilage in the MRI scan image, and the values of both the upper gray-scale threshold and the lower gray-scale threshold corresponding to the human patellar cartilage in the MRI scan image are between 0 and 255;

[0146] Among them, the MRI scan image includes a T1WI-SAG localization scan image, a PDW-SAG localization scan image, a PDW-COR localization scan image, and a T2WI-SPAIR-TRA localization scan image;

[0147] Among them, the patellar cartilage is located behind the patellar cartilage, smoothly covers the back of the patellar cartilage, and is connected to the femoral patellar surface;

[0148] Among them, T1WI-SAG is T1-weighted imaging - sagittal, PDW-SAG is proton density weighted imaging - sagittal, PDW-COR is a coronal scan position, and T2WI-SPAIR-TRA is an axial scan position;

[0149] Among them, the frontal imaging image of the current patient's knee is the imaging image obtained when the imaging lens of the visible light camera faces the front of the current patient's knee, and the front of the current patient's knee is the side of the current patient's knee away from the current patient's thigh when the current patient's thigh and calf are in a vertical state;

[0150] In this way, by obtaining the relevant visual information of the front of the current patient's knee, data can be provided for the distribution state of the current patient's knee, providing a basic data for the intelligent identification of the damage level of the subsequent patellar cartilage;

[0151] And among them, obtaining the knee occupancy image in the frontal imaging image of the current patient's knee, the frontal imaging image of the current patient's knee is the imaging image obtained when the imaging lens of the visible light camera faces the front of the current patient's knee, including: identifying the knee occupancy image in the frontal imaging image of the current patient's knee based on the external shape reference imaging pattern of the human knee.

[0152] In addition, in the cartilage damage recognition system and method based on multi-modal deep learning according to the present invention:

[0153] The customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image, as well as the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image, including: taking each pixel point in the image as the target pixel point, and forming the gray value set corresponding to the target pixel point by the gray values of each adjacent pixel point adjacent to the target pixel point and the single gray value of the target pixel point, and taking the mean square deviation of all gray values in the gray value set corresponding to the target pixel point as the gray value gradient data corresponding to the target pixel point;

[0154] Among them, taking each pixel point in the image as the target pixel point, and forming the gray value set corresponding to the target pixel point by the gray values of each adjacent pixel point adjacent to the target pixel point and the single gray value of the target pixel point, and taking the mean square deviation of all gray values in the gray value set corresponding to the target pixel point as the gray value gradient data corresponding to the target pixel point includes: forming the gray value set corresponding to the target pixel point by the multiple gray values corresponding to the multiple pixel points covered by the square pixel window determined with the target pixel point as the central pixel point;

[0155] Exemplarily, forming the gray value set corresponding to the target pixel point by the multiple gray values corresponding to the multiple pixel points covered by the square pixel window determined with the target pixel point as the central pixel point includes: the square pixel window is a 3-pixel × 3-pixel window;

[0156] And among them, taking each pixel point in the image as the target pixel point, and forming the gray value set corresponding to the target pixel point by the gray values of each adjacent pixel point adjacent to the target pixel point and the single gray value of the target pixel point, and taking the mean square deviation of all gray values in the gray value set corresponding to the target pixel point as the gray value gradient data corresponding to the target pixel point further includes: the value range of the gray value of each pixel point is between 0 and 255.

[0157] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising said element.

[0158] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device / electronic device / computer-readable storage medium / computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content. The above description is only for the preferred embodiments of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are all included in the protection scope of the present invention.

Claims

1. A cartilage damage recognition system based on multi-modal deep learning, characterized in that, The system includes: A first parsing mechanism for obtaining T1WI-SAG localization scan images, PDW-SAG localization scan images, PDW-COR localization scan images, and T2WI-SPAIR-TRA localization scan images of the knee joint of the current patient after magnetic resonance imaging, and then respectively obtaining T1WI-SAG cartilage images, PDW-SAG cartilage images, PDW-COR cartilage images, and T2WI-SPAIR-TRA cartilage images based on the patellar cartilage imaging characteristics; A second parsing mechanism for obtaining the knee occupancy image in the frontal imaging of the knee of the current patient; A third parsing mechanism for obtaining the duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data, and weight data of the current patient as multiple pieces of patellar cartilage-related data of the current patient; A deep learning mechanism for performing deep learning operations on the feedforward neural network for more than a set number of learning times to obtain the feedforward neural network after the deep learning operation and output it as an intelligent injury recognition model. The number of learning times in the deep learning operation passed by the feedforward neural network is monotonically and positively correlated with the age data of the current patient; An intelligent recognition mechanism, connected to the first parsing mechanism, the second parsing mechanism, the third parsing mechanism, and the deep learning mechanism respectively, for using the intelligent injury recognition model to intelligently recognize the patellar cartilage injury level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and the multiple pieces of patellar cartilage-related data of the current patient; Among them, the customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image and the horizontal coordinate value, vertical coordinate value, and gray value of each edge pixel point in the image.

2. The cartilage injury recognition system based on multi-modal deep learning according to claim 1, characterized in that: In each learning performed on the feedforward neural network, the known patellar cartilage injury level of a certain patient is used as the single output content of the feedforward neural network, and the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image of the certain patient respectively, and the multiple pieces of patellar cartilage-related data of the certain patient are used as the item-by-item input content of the feedforward neural network to complete the current learning performed on the feedforward neural network; Among them, the patellar cartilage occupancy areas in the T1WI-SAG localization scan image, PDW-SAG localization scan image, PDW-COR localization scan image, and T2WI-SPAIR-TRA localization scan image are respectively recognized based on the patellar cartilage imaging characteristics and used as the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, and T2WI-SPAIR-TRA cartilage image respectively; Among them, the imaging features of patellar cartilage are the upper gray threshold and the lower gray threshold corresponding to the patellar cartilage of the human body in the nuclear magnetic resonance (MR) scan image. The pixel points with gray values between the upper gray threshold and the lower gray threshold in the MR scan image are regarded as patellar cartilage pixel points in the MR scan image. Each patellar cartilage pixel point in the MR scan image is processed to remove isolated pixel points to obtain multiple patellar cartilage pixel points in the MR scan image. The multiple patellar cartilage pixel points in the MR scan image are subjected to fitting processing to obtain the occupied area of patellar cartilage in the MR scan image; Among them, the MR scan image includes a T1WI-SAG localization scan image, a PDW-SAG localization scan image, a PDW-COR localization scan image, and a T2WI-SPAIR-TRA localization scan image.

3. The cartilage injury recognition system based on multi-modal deep learning according to claim 2, characterized in that: The patellar cartilage is located behind the patellar cartilage, smoothly covers the back of the patellar cartilage and is connected to the femoral patellar surface; Among them, T1WI-SAG is T1-weighted imaging - sagittal plane, PDW-SAG is proton density weighted imaging - sagittal plane, PDW-COR is a coronal scan plane, and T2WI-SPAIR-TRA is an axial scan plane; Among them, the frontal imaging image of the current patient's knee is the imaging image obtained when the imaging lens of the visible light camera faces the front of the current patient's knee, and the front of the current patient's knee is the side of the current patient's knee away from the current patient's thigh when the current patient's thigh and calf are in a vertical state; Among them, obtaining the knee occupancy image in the frontal imaging image of the current patient's knee, the frontal imaging image of the current patient's knee being the imaging image obtained when the imaging lens of the visible light camera faces the front of the current patient's knee includes: identifying the knee occupancy image in the frontal imaging image of the current patient's knee based on the external reference imaging pattern of the human knee.

4. The cartilage injury recognition system based on multi-modal deep learning according to claim 3, characterized in that, The system further includes: A mobile communication mechanism, connected to the intelligent recognition mechanism, for wirelessly transmitting the patellar cartilage injury level of the current patient to a remote sports injury management server through a mobile communication link.

5. The cartilage injury recognition system based on multi-modal deep learning according to claim 3, characterized in that The system further includes: A strategy selection mechanism, connected to the intelligent recognition mechanism, for receiving the patellar cartilage injury level of the current patient and providing an injury response strategy corresponding to the patellar cartilage injury level of the current patient.

6. The cartilage injury recognition system based on multimodal deep learning according to claim 3, characterized in that The system further includes: A data storage mechanism, connected to the deep learning mechanism, for storing the model data of the intelligent injury recognition model to complete the model storage of the intelligent injury recognition model.

7. The cartilage injury recognition system based on multi-modal deep learning according to claim 3, characterized in that, The system further includes: A real-time display mechanism, connected to the intelligent recognition mechanism and the third analysis mechanism respectively, for receiving the patellar cartilage injury level of the current patient and synchronously displaying the patellar cartilage injury level of the current patient and multiple pieces of patellar cartilage related data of the current patient.

8. The cartilage injury recognition system based on multi-modal deep learning according to any one of claims 3-7, characterized in that: Perform deep learning operations on the feedforward neural network for more than the set number of learning times to obtain the feedforward neural network after the deep learning operations, and output it as an intelligent injury recognition model. The number of learning times in the deep learning operations passed by the feedforward neural network is monotonically and positively correlated with the age data of the current patient, including: representing the signal conversion relationship between the number of learning times in the deep learning operations passed by the feedforward neural network and the age data of the current patient using a signal conversion function; Among them, representing the signal conversion relationship between the number of learning times in the deep learning operations passed by the feedforward neural network and the age data of the current patient using a signal conversion function includes: in the signal conversion function, the age data of the current patient is the input signal of the signal conversion function; Among them, representing the signal conversion relationship between the number of learning times in the deep learning operations passed by the feedforward neural network and the age data of the current patient using a signal conversion function further includes: the number of learning times in the deep learning operations passed by the feedforward neural network corresponding to the age data of the current patient is the output signal of the signal conversion function.

9. The cartilage injury recognition system based on multi-modal deep learning according to any one of claims 3-7, characterized in that: Using the intelligent injury recognition model to intelligently identify the patellar cartilage injury level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image and knee occupancy image respectively, and multiple patellar cartilage related data of the current patient, including: performing numerical normalization processing on the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image and knee occupancy image respectively, and multiple patellar cartilage related data of the current patient, and then inputting them in parallel into the intelligent injury recognition model; Among them, using the intelligent injury recognition model to intelligently identify the patellar cartilage injury level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image and knee occupancy image respectively, and multiple patellar cartilage related data of the current patient further includes: running the intelligent injury recognition model to obtain the patellar cartilage injury level of the current patient output by the intelligent injury recognition model; Among them, after performing numerical normalization processing on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient, and then parallelly inputting them into the intelligent damage recognition model includes: using a numerical processing device to complete the numerical normalization processing performed on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient; Among them, after performing numerical normalization processing on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient, and then parallelly inputting them into the intelligent damage recognition model further includes: using a parallel control device to complete the parallel input of each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient that have respectively undergone numerical normalization processing into the intelligent damage recognition model; Among them, using a numerical processing device to complete the numerical normalization processing performed on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient includes: the numerical processing device is an FPGA chip designed in VHDL language; Among them, using a numerical processing device to complete the numerical normalization processing performed on each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient further includes: the numerical normalization processing is an octal numerical conversion processing; Among them, using a parallel control device to complete the parallel input of each piece of customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image, and knee occupancy image respectively, and multiple pieces of patellar cartilage-related data of the current patient that have respectively undergone numerical normalization processing into the intelligent damage recognition model includes: the parallel control device is an ASIC chip and is connected to the numerical processing device; Among them, in each learning of the feedforward neural network, the known patellar cartilage injury level of a certain patient is used as the single output content of the feedforward neural network, and the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image and knee occupancy image of the certain patient, as well as the multiple patellar cartilage-related data of the certain patient are used as the item-by-item input content of the feedforward neural network. Completing the current learning of the feedforward neural network includes: using the numerical simulation mode to complete the test and simulation of the data processing process of each learning of the feedforward neural network.

10. A method for identifying cartilage damage based on multi-modal deep learning, characterized in that, The method includes: Obtaining the T1WI-SAG localization scan screen, PDW-SAG localization scan screen, PDW-COR localization scan screen and T2WI-SPAIR-TRA localization scan screen obtained after magnetic resonance scanning of the knee joint of the current patient, and then respectively obtaining the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image and T2WI-SPAIR-TRA cartilage image based on the patellar cartilage imaging characteristics; Obtaining the knee occupancy image in the frontal imaging screen of the knee of the current patient; Obtaining the disease duration of patellar cartilage injury, gender data, thigh length data, calf length data, age data and weight data of the current patient as the multiple patellar cartilage-related data of the current patient; Performing deep learning operations on the feedforward neural network for more than the set number of learning times to obtain the feedforward neural network after the deep learning operation, and outputting it as an intelligent injury recognition model. The number of learning times in the deep learning operation passed by the feedforward neural network is monotonically positively correlated with the age data of the current patient; Using the intelligent injury recognition model to intelligently identify the patellar cartilage injury level of the current patient according to the customized visual information corresponding to the T1WI-SAG cartilage image, PDW-SAG cartilage image, PDW-COR cartilage image, T2WI-SPAIR-TRA cartilage image and knee occupancy image, as well as the multiple patellar cartilage-related data of the current patient; Among them, the customized visual information corresponding to each image is the gray value gradient data of each pixel point in the image, as well as the horizontal coordinate value, vertical coordinate value and gray value of each edge pixel point in the image.

Images