Posterior lumbar interbody fusion with titanium cage and anterior cervical discectomy and fusion with titanium cage

Abstract

The application discloses a kind of based on multi-modal medical image's lumbar disc herniation postoperative recurrence prediction system, belong to medical image field.Use lumbar image set and each image corresponding feature true value and each patient corresponding recurrence prediction label to carry out training to based on multi-modal image's lumbar disc herniation postoperative recurrence prediction model, in training process, using random gradient descent algorithm to update model parameter, obtain the trained based on multi-modal image's lumbar disc herniation postoperative recurrence prediction model.Through the application, benefit from the rich image features provided by multi-modal image, the prediction network trained based on multi-modal data shows higher accuracy in lumbar disc herniation postoperative recurrence prediction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical imaging, and particularly relates to a lumbar disc herniation recurrence prediction system based on multi-modal medical imaging. BACKGROUND

[0002] X-ray is a traditional imaging technique that uses ionizing radiation to generate images of internal structures of the body, and is widely used in medical diagnosis. In the diagnosis of lumbar disc herniation, X-ray is mainly used to evaluate the overall structure and alignment of the spine, especially to help detect bone changes in the lumbar spine, disc degeneration, and narrowing of the intervertebral space.

[0003] CT (Computed Tomography) is a medical imaging technique that uses a computer to process data from multiple-angle X-ray images to generate high-resolution three-dimensional images, helping doctors to more accurately diagnose and treat diseases. In the field of lumbar disc herniation diagnosis, CT images are widely used to evaluate the shape of the intervertebral disc, determine the location and degree of intervertebral disc herniation, etc.

[0004] MRI (Magnetic Resonance Imagin) is a medical imaging technique that uses magnetic fields and radio waves to generate images of the body's internal structures, providing high-resolution soft tissue contrast images. In the diagnosis of lumbar disc herniation, MRI is widely used, especially in the evaluation of intervertebral disc herniation, intervertebral space narrowing, and nerve root compression, etc. MRI can clearly show the structural changes of the intervertebral disc, helping doctors to determine whether the intervertebral disc is ruptured, herniated or degenerated, and can effectively evaluate the damage to the spinal cord, nerve roots and surrounding soft tissues.

[0005] Deep convolutional neural network (Convolutional Neural Network, CNN) is one of the commonly used methods for building medical artificial intelligence models in recent years. It extracts high-order feature information of images through multiple layers of convolution processing, and combines pooling processing to reduce the dimension of features. The extracted high-order features are input into subsequent specific networks for specific tasks such as classification, segmentation, registration, detection, noise reduction, etc. The advantage of this method is that it can automatically learn high-order features that are significant for specific tasks through a large number of samples, but it has certain requirements for the amount of data used for training.

[0006] State Space Model (SSM) is a new method gradually applied in the field of medical artificial intelligence in recent years. It is based on state space modeling principle, through the dynamic updating and information transmission mechanism of hidden state, it can capture the global dependence of data, and can effectively model long sequence characteristics. Compared with traditional methods, deep state space model can efficiently process long distance information without explicit attention mechanism, and is suitable for various medical tasks such as image classification, sequence modeling, lesion detection, disease progression prediction and multi-modal fusion. The advantage of this method is that the computational complexity is relatively low, which can maintain efficient reasoning in long time sequence or large scale data set, and the demand for training data is relatively low, which improves the stability and generalization ability of the model, so it has broad application prospect in medical image analysis and time sequence modeling tasks.

[0007] There are still some deficiencies in the model of lumbar disc herniation recurrence prediction field, especially in the use of multi-modal data. Most existing models mainly rely on single modal image data, which lacks sufficient comprehensive information in the diagnosis and prediction process. Therefore, in view of the deficiencies of the existing model, the fusion and comprehensive analysis of multi-modal data become the key direction to improve the accuracy of lumbar disc herniation recurrence prediction. By effectively integrating the advantages of different image modalities such as CT, MRI and X-ray, it can more comprehensively reflect the different levels of lesions, especially in the early detection and intervention of disease recurrence, which has greater potential. SUMMARY

[0008] The purpose of the present application is to provide a lumbar disc herniation recurrence prediction system based on multi-modal medical images to overcome the deficiencies of the prior art.

[0009] The purpose of the present application is achieved by the following technical solutions:

[0010] In a first aspect, the present application provides a lumbar disc herniation recurrence prediction system based on multi-modal medical images, comprising:

[0011] A preprocessing module is used to obtain a set of lumbar images and a set of image feature true values of N patients, and give each patient a corresponding recurrence prediction label;

[0012] A model training module is used to train the lumbar disc herniation recurrence prediction model based on multi-modal images using the set of lumbar images. In the training process, the model parameters are updated using the stochastic gradient descent algorithm to obtain the trained lumbar disc herniation recurrence prediction model based on multi-modal images;

[0013] A recurrence prediction module is used to perform early recurrence prediction based on the trained lumbar disc herniation recurrence prediction model based on multi-modal images.

[0014] Further, the lumbar image set of N patients and the image feature true value set are obtained, and the corresponding recurrence prediction label is given to each patient, specifically:

[0015] The lumbar image set of N patients is obtained, which is composed of lumbar CT images, lumbar MRI images and lumbar X-ray images. The image feature true value set of each patient is obtained, which is composed of CT image feature true value corresponding to the lumbar CT image, MRI image feature true value corresponding to the lumbar MRI image and X-ray image feature true value corresponding to the lumbar X-ray image. Then, the recurrence prediction label 1 is given to the patient who has lumbar disc herniation recurrence after surgery, and the recurrence prediction label 0 is given to the patient who has no lumbar disc herniation recurrence after surgery.

[0016] Further, the lumbar disc herniation recurrence prediction model based on multi-modal image is composed of an index-based recurrence prediction sub-module, an image content processing model and an image index feature generation network. The image index feature generation network is obtained by training the Resnet-18 network based on the lumbar image set. The index-based recurrence prediction sub-module is obtained by training the initial support vector machine based on the image feature true value set and the recurrence prediction label of N patients. The image content processing model adopts the Vim model structure.

[0017] Further, the image index feature generation network is obtained by training the Resnet-18 network based on the lumbar image set, specifically:

[0018] The lumbar X-ray image, lumbar MRI image and lumbar CT image of each patient in the lumbar image set are respectively input into the Resnet-18 network to obtain the X-ray image feature prediction value set, MRI image feature prediction value set and CT image feature prediction value set. Then, the mean square error calculation is performed on the X-ray image feature prediction value set and the X-ray image feature true value of N patients, and the mean square error calculation is performed on the CT image feature prediction value set and the CT image feature true value of N patients to obtain the first loss function and the third loss function. The cross-entropy calculation is performed on the MRI image feature prediction value set and the MRI image feature true value of N patients to obtain the second loss function. The Resnet-18 network is trained based on the first loss function, the second loss function and the third loss function, and the state of the Resnet-18 network at the minimum time is recorded as the first, second and third image index feature extraction networks respectively. The combination of the first, second and third image index feature extraction networks is taken as the image index feature generation network.

[0019] Further, the index-based recurrence prediction sub-module is obtained by training the initial support vector machine based on the image feature true value set and the recurrence prediction label of N patients, specifically:

[0020] inputting the CT, MRI and X-ray image feature true values of each patient in the image feature true value set into the initial support vector machine to obtain a set of recurrence prediction probabilities after lumbar disc herniation surgery, and calculating a fourth loss function with the recurrence prediction labels of N patients using a hinge loss function; and training the initial support vector machine based on the fourth loss function, while recording the state of the initial support vector machine at the minimum time as the recurrence prediction sub-module based on the index.

[0021] Further, in the model training module, the CT, MRI and X-ray images of any patient in the lumbar image set are input into the lumbar disc herniation recurrence prediction model based on multi-modal images to obtain the lumbar disc herniation recurrence prediction result corresponding to the patient, specifically including the following sub-steps:

[0022] (a.1) First, the CT, MRI and X-ray images of any patient in the lumbar image set are input into the lumbar disc herniation recurrence prediction model based on multi-modal images. First, the CT, MRI and X-ray image feature prediction true values are obtained by processing through the image index feature generation network and input into the recurrence prediction sub-module based on the index to obtain the lumbar disc herniation recurrence prediction probability of the patient;

[0023] (a.2) Then, the CT, MRI and X-ray images of any patient are processed through the image content processing model to obtain the image content feature result;

[0024] (a.3) Finally, the lumbar disc herniation recurrence prediction probability and the image content feature result are fused and input into the output layer to obtain the lumbar disc herniation recurrence prediction result of the patient.

[0025] Further, the step (a.2) specifically includes the following sub-steps:

[0026] (a.2.1) The CT, MRI and X-ray images of any patient are segmented into three-dimensional image blocks of the same size through the image content processing model to obtain the CT, MRI and X-ray image block set, and all image blocks in each image block set are projected into a high-dimensional space through a fully connected layer to generate CT, MRI and X-ray feature vectors and splice them with global classification features to obtain a multi-modal feature token splicing vector;

[0027] At the same time, the position information of each image block in the corresponding image is recorded to obtain the CT, MRI and X-ray image block position information set and splice it with the global position embedding vector to obtain the fusion embedding representation vector of the multi-modal medical image input sequence;

[0028] Finally, the multi-modal feature token splicing vector and the fusion embedding representation vector of the multi-modal medical image input sequence are added to obtain the initial input sequence of the patient;

[0029] (a.2.2) After the initial input sequence of the patient is processed by the Vim block structure of L layers and mapped by the standardization and multi-layer perception, the final output sequence of the patient is obtained;

[0030] (a.2.3) Finally, the final output sequence of the patient is standardized, and the image content feature result of the image content processing model is obtained by the multi-layer perception.

[0031] Further, in the model training module, the fifth loss function is obtained by cross-entropy calculation of the recurrence prediction label of N patients and the recurrence prediction result after lumbar disc herniation surgery; and the recurrence prediction model based on multi-modal images after lumbar disc herniation surgery is trained by the fifth loss function.

[0032] In a second aspect, the present application provides an electronic device, comprising a memory and a processor, the memory being coupled with the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the above-mentioned recurrence prediction system for lumbar disc herniation after surgery based on multi-modal medical images.

[0033] In a third aspect, the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the above-mentioned recurrence prediction system for lumbar disc herniation after surgery based on multi-modal medical images.

[0034] The beneficial effects of this invention are as follows: This invention uses a prior knowledge-guided method, generating a first image indicator feature extraction network, a second image indicator feature extraction network, and a third image indicator feature extraction network through a set of lumbar spine images and a set of image feature ground values. These networks extract ground values ​​for image feature prediction guided by prior knowledge, assisting in the training of a multimodal image-based model for predicting recurrence after lumbar disc herniation surgery. This invention aims to integrate multimodal lumbar spine image features and simultaneously extract different image feature ground values ​​to assist in the training of a multimodal image-based model for predicting recurrence after lumbar disc herniation surgery, thereby improving the prediction of recurrence after lumbar disc herniation surgery based on multimodal images. The invention improves the accuracy of predictive classification by using lumbar CT, MRI, and X-ray images to predict and classify postoperative recurrence of lumbar disc herniation. This provides clinicians with more accurate recurrence risk assessments and assists in developing personalized treatment plans. Furthermore, the use of ground truth image features as prior knowledge enhances the interpretability of imaging features, helping radiologists and orthopedic surgeons to conduct deeper analysis and utilization of the pathological changes in lumbar disc herniation. Simultaneously, the use of ground truth image features as prior knowledge further improves the interpretability of imaging features, enabling radiologists and orthopedic surgeons to conduct more in-depth analysis and utilization of the pathological changes in lumbar disc herniation. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of a lumbar disc herniation postoperative recurrence prediction system based on multimodal medical imaging.

[0036] Figure 2 Structure diagram of the network for generating image indicator features;

[0037] Figure 3 This is a structural diagram of a multimodal imaging-based model for predicting recurrence after lumbar disc herniation surgery.

[0038] Figure 4 This is a schematic diagram of an electronic device. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0040] Example 1: As Figure 1 As shown, this invention provides a system for predicting recurrence after lumbar disc herniation surgery based on multimodal medical imaging, comprising:

[0041] Preprocessing module, used to obtain The system uses a set of lumbar spine images and ground truth images of each patient, and assigns a recurrence prediction label to each patient, specifically:

[0042] get Patient CT images of the lumbar spine lumbar MRI images and lumbar spine X-ray images A collection of images of the lumbar spine ,in, And obtain lumbar spine CT images for each patient. Corresponding CT image feature ground truth lumbar MRI images Corresponding MRI image feature ground truth and lumbar spine X-ray images Corresponding X-ray image feature ground truth The set of true values ​​of image features Subsequently, patients who experienced recurrence after lumbar disc herniation surgery were given recurrence prediction label 1: Patients who did not experience postoperative recurrence of lumbar disc herniation were given a recurrence prediction label of 0: .

[0043] The lumbar spine CT images, lumbar spine MRI images, and lumbar spine X-ray images were all acquired from hospital data and all met the size requirements of the subsequent network and contained lesion information. In a specific embodiment, for example, the MRI-lumbar spine T1-weighted images and MRI-lumbar spine T2-weighted images are both 512*512*8 slices of lumbar disc herniation cut from the original lumbar spine MRI images based on the location of the lumbar disc herniation lesion. The two MRI weighted images are superimposed in the input layer with 2 channels. The lumbar spine CT image is a 512*512*16 slice containing lumbar disc herniation information; the lumbar spine X-ray image is a 2048*2048 image containing flexion, extension, and neutral postures. The three X-ray images are superimposed in the input layer with 3 channels. The lumbar spine image set is divided into a training set and a test set according to a predetermined ratio, where the ratio of the training set to the test set is 8:2, and the division method is random allocation. The true values ​​of the image features are obtained by the doctor's diagnosis. In this embodiment, they specifically refer to the lumbar lordosis angle, sagittal range of motion, intervertebral disc height index, facet joint angle, protrusion type, Modic changes, and Pfirrmann classification.

[0044] The model training module is used to train a multimodal image-based recurrence prediction model for lumbar disc herniation surgery using a lumbar spine image set. During the training process, the model parameters are updated using a stochastic gradient descent algorithm to obtain a trained multimodal image-based recurrence prediction model for lumbar disc herniation surgery.

[0045] like Figure 3 As shown, the recurrence prediction model for lumbar disc herniation surgery based on multimodal images consists of an index-based recurrence prediction submodule A1, an image content processing model A2, and an image index feature generation network A3. The image index feature generation network A3 is obtained by training a ResNet-18 network using a set of lumbar spine images. The index-based recurrence prediction submodule A1 uses a ground truth set of image features and... The recurrence prediction labels for each patient were obtained by training an initial support vector machine; the image content processing model A2 adopts the Vim model structure.

[0046] The three-dimensional ResNet-18 structure consists of a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, an average pooling layer, a fully connected layer, and a softmax activation layer.

[0047] The composition of the three-dimensional ResNet-18 structure is shown in Table 1.

[0048] Table 1: Composition of the 3D ResNet-18 structure

[0049]

[0050] The image index feature generation network A3 is obtained by training the ResNet-18 network using a set of lumbar spine images, specifically as follows:

[0051] First, the lumbar spine X-ray images of each patient in the lumbar spine image collection are... The images are input into the ResNet-18 network to obtain lumbar spine X-ray images for each patient. Corresponding X-ray image feature prediction value The set of predicted features for X-ray images is obtained. and with The mean squared error of the true features of the X-ray images of each patient is calculated to obtain the first loss function. : ; and based on the first loss function The ResNet-18 network was trained, and the minimum state of the ResNet-18 network was recorded as the first image indicator feature extraction network B1. The lumbar spine X-ray image... Including X-ray images of lumbar flexion, lumbar extension, and lumbar lateral views, the predicted value of X-ray image features is... This includes the lumbar lordosis angle, sagittal plane range of motion, and intervertebral disc height index.

[0052] Subsequently, the lumbar spine MRI images of each patient in the lumbar spine image collection were analyzed. The images were input into the ResNet-18 network to obtain lumbar spine MRI images for each patient. Corresponding MRI image feature prediction values The set of predicted MRI image features is obtained. and with Cross-entropy is calculated using the ground truth values ​​of MRI image features from each patient to obtain the second loss function. : ; and based on the second loss function A ResNet-18 network was trained, and the minimum state of the ResNet-18 network was recorded as the second image indicator feature extraction network B2. The lumbar spine MRI images were used. Including MRI T1-weighted images of the lumbar spine and MRI T2-weighted images of the lumbar spine, then MRI images Corresponding MRI image feature prediction values This includes prominent types, Modic changes, and Pfirrmann classifications.

[0053] Then, the lumbar spine CT images of each patient in the lumbar spine image set were... The images are input into the ResNet-18 network to obtain lumbar spine CT images for each patient. Corresponding CT image feature prediction values The set of predicted features for CT images is obtained. and with The mean squared error of the true CT image features of each patient is calculated to obtain the third loss function. : ; and based on the third loss function The ResNet-18 network was trained, and the state of the ResNet-18 network at its minimum was recorded as the third image indicator feature extraction network B3. The predicted X-ray image features were then used. This includes the facet joint angle.

[0054] Finally, the combination of the first image indicator feature extraction network B1, the second image indicator feature extraction network B2, and the third image indicator feature extraction network B3 is used as the image indicator feature generation network A3, such as... Figure 2 As shown.

[0055] The index-based recurrence prediction submodule A1 uses the image feature ground value set and The relapse prediction labels for each patient were obtained by training the initial support vector machine, specifically:

[0056] inputting the CT image feature true value, the MRI image feature true value and the X-ray image feature true value of each patient in the image feature true value set into the initial support vector machine to obtain the lumbar disc herniation recurrence prediction probability of each patient after the lumbar disc herniation surgery , obtaining a lumbar disc herniation recurrence prediction probability set , and the recurrence prediction label of the patient The fourth loss function is calculated using the hinge loss function : ; and the initial support vector machine is trained based on the fourth loss function , and the state of the initial support vector machine at the minimum time is recorded as the index-based recurrence prediction submodule A1.

[0057] In the model training module, the lumbar CT image, the lumbar MRI image and the lumbar X-ray image of any patient in the lumbar image set are input into the lumbar disc herniation recurrence prediction model based on multi-modal images to obtain the lumbar disc herniation recurrence prediction result corresponding to the patient, and the specific steps include the following steps:

[0058] (a.1) First, the lumbar CT image , the lumbar MRI image and the lumbar X-ray image of any patient in the lumbar image set are input into the lumbar disc herniation recurrence prediction model based on multi-modal images, and first processed by the image index feature generation network A3 to obtain the CT image feature prediction true value corresponding to the lumbar CT image, the MRI image feature prediction true value corresponding to the lumbar MRI image and the X-ray image feature prediction true value corresponding to the lumbar X-ray image;

[0059] Then, the obtained CT image feature prediction true value , MRI image feature prediction true value and X-ray image feature prediction true value are input into the index-based recurrence prediction submodule A1 to obtain the lumbar disc herniation recurrence prediction probability of the patient.

[0060] (a.2) Then, the lumbar CT image , the lumbar MRI image and the lumbar X-ray image of any patient are processed by the image content processing model A2 to obtain the image content feature result .

[0061] ​The step (a.2) specifically comprises the following sub-steps:

[0062] (a.2.1) processing the CT image of the lumbar vertebrae of any one patient , the MRI image of the lumbar vertebrae and the X-ray image of the lumbar vertebrae through the image content processing model A2: first, the CT image of the lumbar vertebrae , the MRI image of the lumbar vertebrae and the X-ray image of the lumbar vertebrae are respectively segmented into three-dimensional image blocks with the same size of 16*16*1, obtaining the CT image block set of the lumbar vertebrae : , the MRI image block set of the lumbar vertebrae : and the X-ray image block set of the lumbar vertebrae : , wherein represents the number of image blocks after segmentation of the CT image of the lumbar vertebrae , , represents the i-th image block in the CT image of the lumbar vertebrae , represents the number of image blocks after segmentation of the MRI image of the lumbar vertebrae , , represents the i-th image block in the MRI image of the lumbar vertebrae , represents the number of image blocks after segmentation of the X-ray image of the lumbar vertebrae , , represents the i-th image block in the X-ray image of the lumbar vertebrae. At the same time, the position information of each image block in the corresponding image is recorded, obtaining the CT image block position information set of the lumbar vertebrae : , the MRI image block position information set of the lumbar vertebrae : and the X-ray image block position information set of the lumbar vertebrae : , wherein represents the position information of the i-th image block in the MRI image of the lumbar vertebrae , represents the position information of the i-th image block in the MRI image of the lumbar vertebrae , represents the position information of the i-th image block in the X-ray image of the lumbar vertebrae , ​​​​​​position information of each image block.

[0063] Subsequently, each image block in the lumbar CT image block set , the lumbar MRI image block set and the lumbar X-ray image block set is projected into a high-dimensional space through a fully connected layer to generate a lumbar CT feature vector : , a lumbar MRI feature vector : and a lumbar X-ray feature vector : , where denotes a learnable parameter matrix, which serves to map each image block to a unified token vector space.

[0064] The global position embedding vector is then concatenated with the lumbar CT image block position information set , the lumbar MRI image block position information set and the lumbar X-ray image block position information set to obtain a fusion embedding representation vector of the multi-modal medical image input sequence : .

[0065] The global classification feature is then concatenated with the lumbar CT feature vector , the lumbar MRI feature vector and the lumbar X-ray feature vector to obtain a multi-modal feature token concatenation vector : .

[0066] Finally, the multi-modal feature token concatenation vector is added to the fusion embedding representation vector of the multi-modal medical image input sequence to obtain the initial input sequence of the th patient : .

[0067] (a.2.2) The initial input sequence of the patient is then processed through the Vim block structure of the th layer and mapped through a standard multi-layer perceptron to obtain the final output sequence of the patient.

[0068] During the Vim block structure processing of the multi-layer token passing, the output of the Vim block structure processing of the th layer is : ; where, representing Vim block structure processing; .

[0069] The Vim block structure processing specifically includes: performing standardization processing on the input of the previous layer ; generating a main branch vector and a gating vector through two sets of linear transformations respectively; then inputting the main branch vector into a forward convolution layer and a backward convolution layer respectively to extract local context information of the sequence and obtain convolution output ; then projecting the convolution output into state space model parameters , and respectively; discretizing the state space model according to to obtain and ; constructing forward and backward state space models and ; multiplying the outputs of the above two paths by the gating vector and adding a linear projection residual to obtain the output of the current layer: ☉ + ☉ .

[0070] After the Vim block structure processing of layers, the final output is obtained, and the final output is mapped through standardization and a multi-layer perceptron to obtain the final output sequence of the patient for subsequent discrimination.

[0071] (a.2.3) Finally, the final output sequence of the patient is standardized, and the image content feature result of the image content processing model is obtained through a multi-layer perceptron.

[0072] (a.3) Finally, the recurrence prediction probability of the patient after lumbar disc herniation surgery and the image content feature result are fused and input into an output layer to obtain the recurrence prediction result of the patient after lumbar disc herniation surgery .

[0073] In the model training module, the fifth loss function is obtained by cross-entropy calculation of the recurrence prediction labels of N patients and the recurrence prediction results of the patients after lumbar disc herniation surgery; and the post-lumbar disc herniation recurrence prediction model based on multi-modal images is trained through the fifth loss function.

[0074] A recurrence prediction module is configured to perform early recurrence prediction based on the trained multi-modal image-based post-lumbar disc herniation recurrence prediction model.

[0075] The lumbar CT image, the lumbar MRI image and the lumbar X-ray image of the patient to be tested are input into the trained multi-modal image-based post-lumbar disc herniation recurrence prediction model to obtain the post-lumbar disc herniation recurrence prediction result of the patient to be tested.

[0076] The first image index feature extraction network B1, the second image index feature extraction network B2 and the third image index feature extraction network B3 are combined as a part (image index feature generation network A3) of the feature extraction model, and the trained initial support vector is connected in series as a part (index-based recurrence prediction submodule A1) of the prediction model, and then the series result is connected in parallel with another part of the untrained image content processing model A2 to construct a complete multi-modal image-based post-lumbar disc herniation recurrence prediction model A. The multi-modal image-based post-lumbar disc herniation recurrence prediction model A takes the lumbar CT image, the lumbar MRI image and the lumbar X-ray image of the patient as input, and after the lumbar CT image, the lumbar MRI image and the lumbar X-ray image are input into the multi-modal image-based post-lumbar disc herniation recurrence prediction model A, the image index feature generation network A3 and the image content processing model A2 are used to convert the lumbar CT image, the lumbar MRI image and the lumbar X-ray image into simulated index features (image feature prediction true value) and image content features (image content feature result), respectively. The simulated index features are used for prediction, and the index prediction result features (post-lumbar disc herniation recurrence prediction probability) and the image content features are used together to complete the post-lumbar disc herniation recurrence classification of the original multi-modal image, which can improve the recurrence classification accuracy. Moreover, based on the trained multi-modal image-based post-lumbar disc herniation recurrence prediction model, the post-lumbar disc herniation recurrence prediction classification result can be directly output based on the multi-modal image without the participation of the patient index feature data.

[0077] The present application benefits from the prior knowledge brought by the image index features, and the multi-modal image-based post-lumbar disc herniation recurrence prediction deep neural network A has higher accuracy for post-lumbar disc herniation recurrence prediction classification. The image index feature data only acts in the training process of the network and does not need to be provided in clinical application. Therefore, the accuracy of the trained multi-modal image-based post-lumbar disc herniation recurrence prediction network A reaches 0.82, which is higher than the accuracy of 0.70 obtained by simply training the single modality without prior knowledge guidance. The network has higher precision and better stability, and has important clinical application value for post-lumbar disc herniation recurrence prediction.

[0078] Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the lumbar disc herniation recurrence prediction based on multi-modal medical images as described above. As Figure 4 Fig. 1 shows a hardware structure diagram of an arbitrary data processing capable device where the lumbar disc herniation recurrence prediction based on multi-modal medical images provided by the embodiments of the present application is located, in addition to the processor, the memory and the network interface shown in the figure, the arbitrary data processing capable device where the device is located in the embodiments usually further comprises other hardware according to the actual functions of the arbitrary data processing capable device, which will not be described here. Figure 4 Fig. 1 shows a hardware structure diagram of an arbitrary data processing capable device where the lumbar disc herniation recurrence prediction based on multi-modal medical images provided by the embodiments of the present application is located, in addition to the processor, the memory and the network interface shown in the figure, the arbitrary data processing capable device where the device is located in the embodiments usually further comprises other hardware according to the actual functions of the arbitrary data processing capable device, which will not be described here.

[0079] Correspondingly, the present application also provides a computer readable storage medium having computer instructions stored thereon, which are executed by a processor to implement the lumbar disc herniation recurrence prediction based on multi-modal medical images as described above. The computer readable storage medium can be an internal storage unit of the arbitrary data processing capable device, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of the arbitrary data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the arbitrary data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0080] Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present application following the general principles thereof and including any and all of the features of the present application disclosed herein or those that are known to those skilled in the art. The specification and examples are to be regarded as exemplary only.

[0081] The above only shows the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A system for predicting recurrence after lumbar disc herniation surgery based on multi-modal medical images, the system comprising: The application relates to a lumbar disc herniation recurrence prediction method based on multi-modal images. A pre-processing module is configured to obtain N a set of lumbar vertebrae images of patients and a set of image feature true values, and give each patient a corresponding recurrence prediction label; The model training module is used for training a lumbar disc herniation recurrence prediction model based on multi-modal images by using a lumbar image set, wherein, in the training process, a random gradient descent algorithm is used to update model parameters, and a trained lumbar disc herniation recurrence prediction model based on multi-modal images is obtained. The lumbar disc herniation postoperative recurrence prediction model based on multi-modal images is composed of an index-based recurrence prediction submodule, an image content processing model, an image index feature generation network, and an output layer; the image index feature generation network is obtained by training a Resnet-18 network through a lumbar image set; the index-based recurrence prediction submodule is obtained by training an initial support vector machine through a set of image feature true values and N a set of recurrence prediction labels of patients; the image content processing model adopts a Vim model structure; and a recurrence prediction module is configured to perform early recurrence prediction based on the trained lumbar disc herniation postoperative recurrence prediction model based on multi-modal images. 2.The lumbar disc herniation recurrence prediction system based on multi-modal medical images of claim 1, wherein, The obtaining N A lumbar image set and an image feature true value set of each patient are obtained, and a corresponding recurrence prediction label is given to each patient, specifically: Obtaining N a lumbar image set composed of lumbar CT images, lumbar MRI images and lumbar X-ray images of each patient; and obtaining an image feature true value set composed of CT image feature true values corresponding to the lumbar CT images, MRI image feature true values corresponding to the lumbar MRI images and X-ray image feature true values corresponding to the lumbar X-ray images of each patient; Patients who have lumbar disc herniation recurrence are given a recurrence prediction label 1, and patients who do not have lumbar disc herniation recurrence are given a recurrence prediction label 0. 3.The lumbar disc herniation recurrence prediction system based on multi-modal medical images of claim 2, wherein, The image index feature generation network is obtained by training a Resnet-18 network by using a lumbar image set, and specifically comprises the following steps: The lumbar X-ray, lumbar MRI and lumbar CT images of each patient in the lumbar image set are respectively input into the Resnet-18 network, and a set of X-ray, MRI and CT image feature prediction values is obtained. Then the X-ray image feature prediction value set is matched with the X-ray image feature true value of the i-th patient, and the CT image feature prediction value set is matched with the CT image feature true value of the i-th patient, and the MRI image feature prediction value set is matched with the MRI image feature true value of the i-th patient, and the mean square error is calculated to obtain a first loss function and a third loss function; and the cross entropy is calculated to obtain a second loss function; and the Resnet-18 network is trained based on the first loss function, the second loss function and the third loss function, and the state of the Resnet-18 network at the minimum time is recorded as a first, second and third image index feature extraction network respectively; and the combination of the first, second and third image index feature extraction networks is taken as an image index feature generation network. N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N N <000001 4.The lumbar disc herniation recurrence prediction system based on multi-modal medical images of claim 2, wherein, The index-based recurrence prediction submodule is trained by an image feature true value set and N The recurrence prediction label of the patient is obtained by training an initial support vector machine, specifically: The CT, MRI and X-ray image feature true values of each patient in the image feature true value set are input into the initial support vector machine to obtain a recurrence prediction probability set after lumbar disc herniation surgery, and the recurrence prediction probability set after lumbar disc herniation surgery is compared with N The recurrence prediction label of the patient is calculated using a hinge loss function to obtain a fourth loss function; and the initial support vector machine is trained based on the fourth loss function, and the state of the initial support vector machine at the minimum time is recorded as the index-based recurrence prediction submodule. 5.The lumbar disc herniation recurrence prediction system based on multi-modal medical images of claim 1, wherein, In the model training module, the lumbar CT, lumbar MRI and lumbar X-ray images of any patient in the lumbar image set are input into the lumbar disc herniation recurrence prediction model based on multi-modal images, and the corresponding lumbar disc herniation recurrence prediction result of the patient is obtained, and the specific steps comprise the following sub-steps: (a.1) first, the lumbar CT, lumbar MRI and lumbar X-ray images of any patient in the lumbar image set are input into the lumbar disc herniation recurrence prediction model based on multi-modal images, and the CT, MRI and X-ray image feature prediction true values are obtained by the image index feature generation network and input into the index-based recurrence prediction submodule, so as to obtain the lumbar disc herniation recurrence prediction probability of the patient; (a.2) then, the lumbar CT, lumbar MRI and lumbar X-ray images of any patient are processed by the image content processing model, and the image content feature result is obtained; (a.3) finally, the lumbar disc herniation recurrence prediction probability and the image content feature result are fused and input into the output layer, and the lumbar disc herniation recurrence prediction result of the patient is obtained; In the model training module, by N The cross entropy calculation is performed on the recurrence prediction label of the patient and the recurrence prediction result after the lumbar disc herniation surgery to obtain a fifth loss function; and the recurrence prediction model after the lumbar disc herniation surgery based on the multi-modal image is trained through the fifth loss function. 6.The lumbar disc herniation recurrence prediction system based on multi-modal medical images of claim 5, wherein, The step (a.2) specifically comprises the following sub-steps: (a.2.1) the lumbar CT, lumbar MRI and lumbar X-ray images of any patient are segmented into three-dimensional image blocks of the same size by the image content processing model, and a set of lumbar CT, lumbar MRI and lumbar X-ray image blocks is obtained; all image blocks in each image block set are projected into a high-dimensional space through a full connection layer, a lumbar CT, lumbar MRI and lumbar X-ray feature vector is generated, and a global classification feature is spliced, so as to obtain a multi-modal feature token splicing vector; Meanwhile, the position information of each image block in the corresponding image is recorded, a set of lumbar CT, lumbar MRI and lumbar X-ray image block position information is obtained, and a global position embedding vector is spliced, so as to obtain a fusion embedding representation vector of the multi-modal medical image input sequence; Finally, the multi-modal feature token splicing vector and the fusion embedding representation vector of the multi-modal medical image input sequence are added, and the initial input sequence of the patient is obtained. (a.2.2) The patient's initial input sequence was then processed. L After processing the Vim block structure of the layer and normalizing it, and mapping it to the multilayer perceptron, the final output sequence of the patient is obtained. (a.2.3) Finally, the final output sequence of the patient is standardized, and the image content feature result of the image content processing model is obtained through the multi-layer perception.

7. An electronic device comprising a memory and a processor, characterized in that The memory is coupled with the processor; wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the multi-modal medical image-based lumbar disc herniation recurrence prediction system of any one of claims 1-6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the multi-modal medical image-based lumbar disc herniation recurrence prediction system of any one of claims 1-6.

Images