Lumbar intervertebral disc herniation postoperative recurrence prediction system based on multi-modal medical image

Abstract

The invention discloses a lumbar disc herniation postoperative recurrence prediction system based on a multi-modal medical image, and belongs to the field of medical images. The lumbar disc herniation postoperative recurrence prediction model based on the multi-modal image is trained by using the lumbar vertebra image set, the feature true value corresponding to each image and the recurrence prediction label corresponding to each patient, and in the training process, model parameters are updated by using a stochastic gradient descent algorithm; and a trained lumbar disc herniation postoperative recurrence prediction model based on the multi-modal image is obtained. According to the method and the system, due to rich image features provided by the multi-modal image, the prediction network trained based on the multi-modal data shows higher accuracy in postoperative recurrence prediction of the lumbar disc herniation.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging, and in particular relates to a system for predicting recurrence after lumbar disc herniation surgery based on multimodal medical imaging. Background Technology

[0002] X-ray is a traditional imaging technique that uses ionizing radiation to generate images of the body's internal structures and is widely used in medical diagnosis. In the diagnosis of lumbar disc herniation, X-rays are mainly used to assess the overall structure and alignment of the spine, and in particular, can help detect bony changes in the lumbar vertebrae, intervertebral disc degeneration, and narrowing of the intervertebral space.

[0003] CT (Computed Tomography) is a medical imaging technique that uses computers to process data from multi-angle X-ray images to generate high-resolution three-dimensional images, helping doctors to diagnose and treat diseases more accurately. In the diagnosis of lumbar disc herniation, CT imaging is widely used to assess the morphology of the intervertebral disc and determine the location and extent of the herniation.

[0004] MRI (Magnetic Resonance Imaging) is a medical imaging technique that uses magnetic fields and radio waves to generate images of the body's interior, providing high-resolution, contrasting images of soft tissues. MRI is widely used in the diagnosis of lumbar disc herniation, particularly in assessing disc protrusion, intervertebral space narrowing, and nerve root compression. Compared to CT, MRI can more clearly show structural changes in the intervertebral disc, helping doctors determine whether the disc has ruptured, herniated, or degenerated, and effectively assessing damage to the spinal cord, nerve roots, and surrounding soft tissues.

[0005] Deep convolutional neural networks (CNNs) have become a common method for building medical artificial intelligence models in recent years. They extract high-order features from images through multi-layer convolutional processing, while combining pooling to reduce the dimensionality of these features. The extracted high-order features are then input into subsequent specific networks for specific tasks, such as classification, segmentation, registration, detection, and noise reduction. The advantage of this method is that it can automatically learn high-order features that are significant for a specific task from a large number of samples; however, it requires a certain amount of training data.

[0006] State-space models (SSMs) are a novel approach that has been increasingly applied in the field of medical artificial intelligence in recent years. Based on the principles of state-space modeling, they capture global dependencies in data through dynamic updates and information transfer mechanisms of hidden states, and can effectively model long-sequence features. Compared to traditional methods, deep state-space models can efficiently process long-distance information without explicit attention mechanisms and are applicable to various medical tasks, such as image classification, sequence modeling, lesion detection, disease progression prediction, and multimodal fusion. The advantages of this method lie in its lower computational complexity, its ability to maintain efficient inference on long-term series or large-scale datasets, and its relatively low requirement for training data, which improves the model's stability and generalization ability, making it a promising candidate for applications in medical image analysis and time-series modeling tasks.

[0007] Models for predicting recurrence after lumbar disc herniation surgery still have certain limitations, particularly in the utilization of multimodal data. Most existing models rely primarily on single-modal imaging data, lacking sufficient comprehensive information for diagnosis and prediction. Therefore, addressing these shortcomings by introducing the fusion and comprehensive analysis of multimodal data is a key direction for improving the accuracy of predicting recurrence after lumbar disc herniation surgery. Effectively integrating the advantages of different imaging modalities such as CT, MRI, and X-rays can more comprehensively reflect different levels of the lesion, showing great potential, especially in the early detection and intervention of disease recurrence. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a system for predicting postoperative recurrence of lumbar disc herniation based on multimodal medical imaging.

[0009] The objective of this invention is achieved through the following technical solution: In a first aspect, the present invention provides a system for predicting postoperative recurrence of lumbar disc herniation based on multimodal medical imaging, comprising: The preprocessing module is used to obtain a set of lumbar spine images and a set of ground truth images of N patients, and to give each patient a corresponding recurrence prediction label; The model training module is used to train a multimodal image-based prediction model for recurrence after lumbar disc herniation surgery using a lumbar spine image set. During the training process, the model parameters are updated using a stochastic gradient descent algorithm to obtain a trained multimodal image-based prediction model for recurrence after lumbar disc herniation surgery. The recurrence prediction module is used to predict early recurrence based on a trained multimodal imaging-based postoperative recurrence prediction model for lumbar disc herniation.

[0010] Furthermore, the process of obtaining a set of lumbar spine images and a set of ground truth image features for N patients, and assigning a recurrence prediction label to each patient, specifically involves: A set of lumbar spine images, consisting of lumbar CT images, lumbar MRI images, and lumbar X-ray images of N patients, was obtained. A set of image feature ground truths was also obtained for each patient, consisting of the ground truth values ​​of CT image features corresponding to the lumbar CT image, the ground truth values ​​of MRI image features corresponding to the lumbar MRI image, and the ground truth values ​​of X-ray image features corresponding to the lumbar X-ray image. Patients who experienced recurrence after lumbar disc herniation surgery were assigned a recurrence prediction label 1, while patients who did not experience recurrence after lumbar disc herniation surgery were assigned a recurrence prediction label 0.

[0011] Furthermore, the recurrence prediction model for lumbar disc herniation based on multimodal images consists of an index-based recurrence prediction submodule, an image content processing model, and an image index feature generation network. The image index feature generation network is obtained by training a ResNet-18 network using a set of lumbar spine images. The index-based recurrence prediction submodule is obtained by training an initial support vector machine using a set of ground truth image features and recurrence prediction labels for N patients. The image content processing model adopts the Vim model structure.

[0012] Furthermore, the image index feature generation network is obtained by training the ResNet-18 network using a set of lumbar spine images, specifically as follows: The lumbar spine X-ray, lumbar MRI, and lumbar CT images of each patient in the lumbar spine image set are input into the ResNet-18 network to obtain sets of predicted features for X-ray, MRI, and CT images. Then, the mean squared error is calculated between the predicted X-ray image feature set and the ground truth X-ray image features of N patients, and the predicted CT image feature set and the ground truth CT image features of N patients, respectively, to obtain the first and third loss functions. The cross-entropy is calculated between the predicted MRI image feature set and the ground truth MRI image features of N patients to obtain the second loss function. The ResNet-18 network is trained based on the first, second, and third loss functions, and the minimum state of the ResNet-18 network is recorded as the first, second, and third image indicator feature extraction networks, respectively. The combination of the first, second, and third image indicator feature extraction networks is used as the image indicator feature generation network.

[0013] Furthermore, the index-based recurrence prediction submodule is obtained by training the initial support vector machine using the ground truth set of image features and the recurrence prediction labels of N patients, specifically as follows: The true values ​​of CT, MRI, and X-ray image features for each patient in the image feature set are input into the initial support vector machine to obtain the set of predicted recurrence probabilities after lumbar disc herniation surgery. The fourth loss function is calculated using the hinge loss function and the recurrence prediction labels of N patients. The initial support vector machine is then trained based on the fourth loss function, and the state of the initial support vector machine when it is minimized is recorded as the index-based recurrence prediction submodule.

[0014] Furthermore, in the model training module, lumbar CT, lumbar MRI, and lumbar X-ray images of any patient from the lumbar spine image set are input into the lumbar disc herniation postoperative recurrence prediction model based on multimodal imaging to obtain the corresponding lumbar disc herniation postoperative recurrence prediction result for that patient. This specifically includes the following sub-steps: (a.1) First, the lumbar CT, lumbar MRI and lumbar X-ray images of any patient in the lumbar image set are input into the lumbar disc herniation postoperative recurrence prediction model based on multimodal images. The images are first processed by the image index feature generation network to obtain the true values ​​of CT, MRI and X-ray image feature prediction and input into the index-based recurrence prediction submodule to obtain the predicted probability of lumbar disc herniation postoperative recurrence of the patient. (a.2) Subsequently, the lumbar CT, lumbar MRI and lumbar X-ray images of any patient are processed by the image content processing model to obtain the image content feature results; (a.3) Finally, the predicted probability of recurrence after lumbar disc herniation surgery and the results of image content features are fused and input into the output layer to obtain the predicted result of recurrence after lumbar disc herniation surgery for this patient.

[0015] Furthermore, step (a.2) specifically includes the following sub-steps: (a.2.1) The lumbar CT, lumbar MRI and lumbar X-ray images of any patient are segmented into three-dimensional image blocks of the same size by the image content processing model to obtain a set of lumbar CT, lumbar MRI and lumbar X-ray image blocks. All image blocks in each image block set are projected to a high-dimensional space through a fully connected layer to generate lumbar CT, lumbar MRI and lumbar X-ray feature vectors and concatenate them with global classification features to obtain a multimodal feature token concatenation vector. Simultaneously, the position information of each image block in the corresponding image is recorded to obtain the set of position information of lumbar CT, lumbar MRI and lumbar X-ray image blocks, which is then concatenated with the global position embedding vector to obtain the fusion embedding representation vector of the multimodal medical image input sequence; Finally, the concatenated vector of multimodal feature tokens and the fused embedding representation vector of the multimodal medical image input sequence are added together to obtain the patient's initial input sequence. (a.2.2) Subsequently, the patient's initial input sequence was processed by an L-layer Vim block structure and then normalized and mapped to a multilayer perceptron to obtain the patient's final output sequence; (a.2.3) Finally, the patient's final output sequence is standardized, and then the image content feature results of the image content processing model are obtained through a multilayer perceptron.

[0016] Furthermore, in the model training module, the fifth loss function is obtained by calculating the cross-entropy of the recurrence prediction labels of N patients and the recurrence prediction results after lumbar disc herniation surgery; the fifth loss function is used to train the lumbar disc herniation surgery recurrence prediction model based on multimodal images.

[0017] In a second aspect, the present invention provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to realize the above-mentioned recurrence prediction system for lumbar disc herniation based on multimodal medical imaging.

[0018] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described recurrence prediction system for lumbar disc herniation based on multimodal medical imaging.

[0019] The beneficial effects of this invention are as follows: This invention uses a prior knowledge-guided method, generating a first image indicator feature extraction network, a second image indicator feature extraction network, and a third image indicator feature extraction network through a set of lumbar spine images and a set of image feature ground values. These networks extract ground values ​​for image feature prediction guided by prior knowledge, assisting in the training of a multimodal image-based model for predicting recurrence after lumbar disc herniation surgery. This invention aims to integrate multimodal lumbar spine image features and simultaneously extract different image feature ground values ​​to assist in the training of a multimodal image-based model for predicting recurrence after lumbar disc herniation surgery, thereby improving the prediction of recurrence after lumbar disc herniation surgery based on multimodal images. The invention improves the accuracy of predictive classification by using lumbar CT, MRI, and X-ray images to predict and classify postoperative recurrence of lumbar disc herniation. This provides clinicians with more accurate recurrence risk assessments and assists in developing personalized treatment plans. Furthermore, the use of ground truth image features as prior knowledge enhances the interpretability of imaging features, helping radiologists and orthopedic surgeons to conduct deeper analysis and utilization of the pathological changes in lumbar disc herniation. Simultaneously, the use of ground truth image features as prior knowledge further improves the interpretability of imaging features, enabling radiologists and orthopedic surgeons to conduct more in-depth analysis and utilization of the pathological changes in lumbar disc herniation. Attached Figure Description

[0020] Figure 1This is a schematic diagram of a lumbar disc herniation postoperative recurrence prediction system based on multimodal medical imaging. Figure 2 Structure diagram of the network for generating image indicator features; Figure 3 This is a structural diagram of a multimodal imaging-based model for predicting recurrence after lumbar disc herniation surgery. Figure 4 This is a schematic diagram of an electronic device. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0022] Example 1: As Figure 1 As shown, this invention provides a system for predicting recurrence after lumbar disc herniation surgery based on multimodal medical imaging, comprising: Preprocessing module, used to obtain The system uses a set of lumbar spine images and ground truth images of each patient, and assigns a recurrence prediction label to each patient, specifically: get Patient CT images of the lumbar spine MRI images of the lumbar spine and lumbar spine X-ray images A collection of images of the lumbar spine ,in, And obtain lumbar spine CT images for each patient. Corresponding CT image feature ground truth MRI images of the lumbar spine Corresponding MRI image feature ground truth and lumbar spine X-ray images Corresponding X-ray image feature ground truth The set of true values ​​of image features Subsequently, patients who experienced recurrence after lumbar disc herniation surgery were given recurrence prediction label 1: Patients who did not experience postoperative recurrence of lumbar disc herniation were given a recurrence prediction label of 0: .

[0023] The lumbar CT images, lumbar MRI images, and lumbar X-ray images were all acquired from hospital data and all met the size requirements of the subsequent network and contained lesion information. In a specific embodiment, for example, the MRI-lumbar T1-weighted images and MRI-lumbar T2-weighted images are 512*512*8 slices of lumbar disc herniation cut from the original lumbar MRI images based on the location of the lumbar disc herniation lesion. The two MRI weighted images are superimposed in the input layer with 2 channels. The lumbar CT image is a 512*512*16 slice containing lumbar disc herniation information; the lumbar X-ray image is a 2048*2048 image containing flexion, extension, and neutral postures. The three X-ray images are superimposed in the input layer with 3 channels. The lumbar image set is divided into a training set and a test set according to a predetermined ratio, where the ratio of the training set to the test set is 8:2, and the division method is random allocation. The true values ​​of the image features are obtained by the doctor's diagnosis. In this embodiment, they specifically refer to the lumbar lordosis angle, sagittal range of motion, intervertebral disc height index, facet joint angle, protrusion type, Modic changes, and Pfirrmann classification.

[0024] The model training module is used to train a multimodal image-based recurrence prediction model for lumbar disc herniation surgery using a lumbar spine image set. During the training process, the model parameters are updated using a stochastic gradient descent algorithm to obtain a trained multimodal image-based recurrence prediction model for lumbar disc herniation surgery.

[0025] like Figure 3 As shown, the recurrence prediction model for lumbar disc herniation surgery based on multimodal images consists of an index-based recurrence prediction submodule A1, an image content processing model A2, and an image index feature generation network A3. The image index feature generation network A3 is obtained by training a ResNet-18 network using a set of lumbar spine images. The index-based recurrence prediction submodule A1 uses a ground truth set of image features and... The recurrence prediction labels for each patient were obtained by training an initial support vector machine; the image content processing model A2 adopts the Vim model structure.

[0026] The three-dimensional ResNet-18 structure consists of a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer, an average pooling layer, a fully connected layer, and a softmax activation layer.

[0027] The composition of the three-dimensional ResNet-18 structure is shown in Table 1.

[0028] Table 1: Composition of the 3D ResNet-18 structure

[0029] The image index feature generation network A3 is obtained by training the ResNet-18 network using a set of lumbar spine images, specifically as follows: First, the lumbar spine X-ray images of each patient in the lumbar spine image collection are... The images are input into the ResNet-18 network to obtain lumbar spine X-ray images for each patient. Corresponding X-ray image feature prediction value The set of predicted features for X-ray images is obtained. and with The mean squared error of the true features of the X-ray images of each patient is calculated to obtain the first loss function. : ; and based on the first loss function The ResNet-18 network was trained, and the minimum state of the ResNet-18 network was recorded as the first image indicator feature extraction network B1. The lumbar spine X-ray image... Including X-ray images of lumbar flexion, lumbar extension, and lumbar lateral views, the predicted value of X-ray image features is... This includes the lumbar lordosis angle, sagittal plane range of motion, and intervertebral disc height index.

[0030] Subsequently, the lumbar spine MRI images of each patient in the lumbar spine image collection were analyzed. The images were input into the ResNet-18 network to obtain lumbar spine MRI images for each patient. Corresponding MRI image feature prediction values The set of predicted MRI image features is obtained. and with Cross-entropy is calculated using the ground truth values ​​of MRI image features from each patient to obtain the second loss function. : ; and based on the second loss function A ResNet-18 network was trained, and the minimum state of the ResNet-18 network was recorded as the second image indicator feature extraction network B2. The lumbar spine MRI images were used. Including MRI T1-weighted images of the lumbar spine and MRI T2-weighted images of the lumbar spine, then MRI images Corresponding MRI image feature prediction values This includes prominent types, Modic changes, and Pfirrmann classifications.

[0031] Then, the lumbar spine CT images of each patient in the lumbar spine image set were... The images are input into the ResNet-18 network to obtain lumbar spine CT images for each patient. Corresponding CT image feature prediction values The set of predicted features for CT images is obtained. and with The mean squared error of the true CT image features of each patient is calculated to obtain the third loss function. : ; and based on the third loss function The ResNet-18 network was trained, and the state of the ResNet-18 network at its minimum was recorded as the third image indicator feature extraction network B3. The predicted X-ray image features were then used. This includes the facet joint angle.

[0032] Finally, the combination of the first image indicator feature extraction network B1, the second image indicator feature extraction network B2, and the third image indicator feature extraction network B3 is used as the image indicator feature generation network A3, such as... Figure 2 As shown.

[0033] The index-based recurrence prediction submodule A1 uses the image feature ground value set and The relapse prediction labels for each patient were obtained by training the initial support vector machine, specifically: The ground truth values ​​of CT, MRI, and X-ray images for each patient in the ground truth image feature set are input into the initial support vector machine to obtain the predicted probability of recurrence after lumbar disc herniation surgery for each patient. The set of predicted probabilities for recurrence after lumbar disc herniation surgery was obtained. and with Recurrence prediction label for patients The fourth loss function is calculated using the hinge loss function. : ; and based on the fourth loss function The initial support vector machine is trained, and the state when the initial support vector machine is minimized is recorded as the index-based recurrence prediction submodule A1.

[0034] In the model training module, the lumbar spine CT image, lumbar spine MRI image, and lumbar spine X-ray image of any patient from the lumbar spine image set are input into the lumbar disc herniation postoperative recurrence prediction model based on multimodal imaging to obtain the corresponding lumbar disc herniation postoperative recurrence prediction result for that patient. Specifically, this includes the following sub-steps: (a.1) First, take the lumbar spine CT images of any patient in the lumbar spine image set. lumbar MRI images and lumbar spine X-ray images The data is input into a multimodal image-based model for predicting recurrence after lumbar disc herniation surgery. First, it is processed by the A3 image feature generation network to obtain the ground truth prediction values ​​of CT image features corresponding to the lumbar CT images. True value of MRI image feature prediction corresponding to lumbar spine MRI images and the ground truth of X-ray image feature prediction corresponding to lumbar spine X-ray images. ; The obtained CT image feature prediction ground value will then be used. MRI image feature prediction true value True value of X-ray image feature prediction The data is input into the indicator-based recurrence prediction submodule A1 to obtain the predicted probability of recurrence after lumbar disc herniation surgery for this patient. .

[0035] (a.2) Then, take any patient's lumbar spine CT image. MRI images of the lumbar spine and lumbar spine X-ray images Image content feature results are obtained after processing by the image content processing model A2. .

[0036] Step (a.2) specifically includes the following sub-steps: (a.2.1) Take any patient's lumbar spine CT image MRI images of the lumbar spine and lumbar spine X-ray images Image content processing model A2 was used to process the lumbar spine CT images. MRI images of the lumbar spine and lumbar spine X-ray images The image was divided into three-dimensional blocks of the same size (16*16*1) to obtain a set of lumbar spine CT image blocks. : Lumbar spine MRI image block collection : Collection of lumbar spine X-ray images : ,in, Representing lumbar spine CT images The number of image patches after segmentation , Representing lumbar spine CT images The Middle Image blocks, Representing lumbar MRI images The number of image patches after segmentation , Representing lumbar MRI images The Middle Image blocks, Image of lumbar spine X-ray The number of image patches after segmentation , Image of lumbar spine X-ray The Middle Each image block is analyzed. The positional information of each image block within its corresponding image is recorded, resulting in a set of lumbar spine CT image block positional information. : 1. Set of location information of lumbar spine MRI image blocks : Set of location information of lumbar spine X-ray images : ,in, Representing lumbar MRI images The Middle Location information of each image patch Representing lumbar MRI images The Middle Location information of each image patch Image of lumbar spine X-ray The Middle Location information of each image patch.

[0037] Subsequently, the lumbar spine CT image blocks were collected separately. Lumbar spine MRI image block collection Collection of lumbar spine X-ray images Each image patch is projected into a high-dimensional space through a fully connected layer to generate a lumbar spine CT feature vector. : Lumbar spine MRI feature vector : and lumbar spine X-ray feature vector : ,in, This represents the learnable parameter matrix, which maps each image patch to a unified token vector space.

[0038] Then embed the global position vector Set of location information of lumbar spine CT image blocks 1. Set of location information of lumbar spine MRI image blocks Set of location information of lumbar spine X-ray images The images are concatenated to obtain a fused embedding representation vector of the multimodal medical image input sequence. : .

[0039] global classification features Lumbar spine CT feature vector Lumbar spine MRI feature vector and lumbar spine X-ray feature vector The tokens are concatenated to obtain a multimodal feature token concatenation vector. : .

[0040] Finally, the multimodal feature tokens are concatenated into a vector. Fusion embedding representation vector of multimodal medical image input sequence Add them together to get the first one. Initial input sequence for the patient : .

[0041] (a.2.2) The patient's initial input sequence was then processed. After processing the Vim block structure of the layer and normalizing it, and mapping it to the multilayer perceptron, the final output sequence of the patient is obtained.

[0042] When handling Vim block structures during multi-level token passing, the first... The output of the Vim block structure processing layer is : ;in, This indicates the processing of Vim block structures; .

[0043] The Vim block structure processing specifically involves: processing the input from the previous layer... Standardization is performed; the main branch vector is generated through two sets of linear transformations. With gate vector Then the main branch vector By inputting the data into the forward and backward convolutional layers respectively, local contextual information of the sequence is extracted to obtain the convolutional output. Then the convolution output Projected onto the state-space model parameters respectively: , and ;according to Discretizing the state-space model yields: and Construct forward and backward state space models: and Multiply the above two outputs by the gate vector. The results are then summed and the linear projection residual is added to obtain the output of the current layer: ☉ + ☉ .

[0044] go through After processing the Vim block structure of the layer, the final output is obtained. Finally, output After standardization and mapping with a multilayer perceptron, the patient's final output sequence was obtained for subsequent discrimination.

[0045] (a.2.3) Finally, the patient's final output sequence is standardized, and then the image content feature results of the image content processing model are obtained through a multilayer perceptron.

[0046] (a.3) Finally, predict the probability of recurrence after lumbar disc herniation surgery. Fusion with image content feature results The results are then input to the output layer to obtain the prediction results for postoperative recurrence of lumbar disc herniation in this patient. .

[0047] In the model training module, the fifth loss function is obtained by calculating the cross-entropy of the recurrence prediction labels of N patients and the recurrence prediction results after lumbar disc herniation surgery; the fifth loss function is used to train the lumbar disc herniation postoperative recurrence prediction model based on multimodal images.

[0048] The recurrence prediction module is used to predict early recurrence based on a trained multimodal imaging-based postoperative recurrence prediction model for lumbar disc herniation.

[0049] The lumbar CT, lumbar MRI, and lumbar X-ray images of the patient to be tested were input into a trained multimodal imaging-based model for predicting recurrence after lumbar disc herniation surgery, and the prediction results of recurrence after lumbar disc herniation surgery for the patient to be tested were obtained.

[0050] This invention combines the trained first image indicator feature extraction network B1, second image indicator feature extraction network B2, and third image indicator feature extraction network B3 as part of the feature extraction model (image indicator feature generation network A3), and concatenates it with the trained initial support vectors as part of the prediction model (indicator-based recurrence prediction submodule A1). The concatenation result is then concatenated in parallel with another untrained image content processing model A2 to construct a complete multimodal image-based postoperative recurrence prediction model A for lumbar disc herniation. The multimodal image-based lumbar disc herniation postoperative recurrence prediction model A takes the patient's lumbar CT images, lumbar MRI images, and lumbar X-ray images as input. After being input into the multimodal image-based lumbar disc herniation postoperative recurrence prediction model A, the lumbar CT images, lumbar MRI images, and lumbar X-ray images are transformed into simulated indicator features (image feature prediction ground truth) and image content features (image content feature results) through the image indicator feature generation network A3 and the image content processing model A2, respectively. The simulated indicator features participate in the prediction results. With the joint participation of the indicator prediction result features (predicted probability of lumbar disc herniation postoperative recurrence) and image content features, the recurrence classification of lumbar disc herniation postoperative recurrence based on the original multimodal images can be completed, which can improve the accuracy of recurrence classification. Furthermore, the trained multimodal image-based lumbar disc herniation postoperative recurrence prediction model can directly output the recurrence prediction classification results of lumbar disc herniation based on multimodal images without the participation of patient indicator feature data.

[0051] This invention benefits from the prior knowledge derived from imaging index features. The deep neural network A, based on multimodal imaging for predicting postoperative recurrence of lumbar disc herniation, exhibits higher accuracy in predicting and classifying recurrence. The imaging index feature data is only used during network training and is not required in clinical applications. Consequently, the trained multimodal imaging network A for predicting postoperative recurrence of lumbar disc herniation achieves an accuracy of 0.82, compared to 0.70 for training solely with a single modality without prior knowledge. This demonstrates higher accuracy and better stability, making it of significant clinical value for predicting postoperative recurrence of lumbar disc herniation.

[0052] Accordingly, this application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; and, when the one or more programs are executed by the one or more processors, causing the one or more processors to perform the above-described prediction of recurrence after lumbar disc herniation surgery based on multimodal medical imaging. Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities for predicting postoperative recurrence of lumbar disc herniation based on multimodal medical imaging, as provided in an embodiment of the present invention. Except for... Figure 4In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0053] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the above-described prediction of recurrence after lumbar disc herniation surgery based on multimodal medical imaging. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0054] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A system for predicting recurrence after lumbar disc herniation surgery based on multimodal medical imaging, characterized in that, include: Preprocessing module, used to obtain N A set of lumbar spine images and a set of image feature ground truth for each patient, and a corresponding recurrence prediction label for each patient; The model training module is used to train a multimodal image-based prediction model for recurrence after lumbar disc herniation surgery using a lumbar spine image set. During the training process, the model parameters are updated using a stochastic gradient descent algorithm to obtain a trained multimodal image-based prediction model for recurrence after lumbar disc herniation surgery. The recurrence prediction module is used to predict early recurrence based on a trained multimodal imaging-based postoperative recurrence prediction model for lumbar disc herniation.

2. The postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging according to claim 1, characterized in that, The acquisition N The system uses a set of lumbar spine images and ground truth images of each patient, and assigns a recurrence prediction label to each patient, specifically: get N The lumbar spine image set consists of lumbar spine CT images, lumbar spine MRI images, and lumbar spine X-ray images of each patient; and the image feature ground value set is obtained for each patient, consisting of the CT image feature ground value corresponding to the lumbar spine CT image, the MRI image feature ground value corresponding to the lumbar spine MRI image, and the X-ray image feature ground value corresponding to the lumbar spine X-ray image. Patients who experienced recurrence after lumbar disc herniation surgery were assigned a recurrence prediction label 1, while patients who did not experience recurrence after lumbar disc herniation surgery were assigned a recurrence prediction label 0.

3. The postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging according to claim 1, characterized in that, The recurrence prediction model for lumbar disc herniation surgery based on multimodal image processing consists of an index-based recurrence prediction submodule, an image content processing model, and an image index feature generation network. The image index feature generation network is trained on a ResNet-18 network using a set of lumbar spine images. The index-based recurrence prediction submodule uses a ground truth set of image features and... N The recurrence prediction labels for each patient were obtained by training an initial support vector machine; the image content processing model adopted the Vim model structure.

4. The postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging according to claim 3, characterized in that, The image index feature generation network is obtained by training a ResNet-18 network using a set of lumbar spine images, specifically as follows: The lumbar spine X-ray, lumbar spine MRI, and lumbar spine CT images of each patient in the lumbar spine image set are input into the Resnet-18 network to obtain the set of predicted feature values ​​for X-ray, MRI, and CT images. Then the X-ray image feature prediction value set and N The set of true values ​​of X-ray image features and predicted values ​​of CT image features for each patient N The mean squared error was calculated for the true CT image features of each patient to obtain the first and third loss functions; and the predicted MRI image feature sets were then compared with... N Cross-entropy is calculated using the ground truth values ​​of MRI image features from each patient to obtain the second loss function. The ResNet-18 network is trained based on the first, second, and third loss functions, and the state of the ResNet-18 network at its minimum is recorded as the first, second, and third image indicator feature extraction networks, respectively. The combination of the first, second, and third image indicator feature extraction networks is used as the image indicator feature generation network.

5. The postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging according to claim 3, characterized in that, The index-based recurrence prediction submodule uses the image feature ground value set and N The relapse prediction labels for each patient were obtained by training the initial support vector machine, specifically: The ground truth values ​​of CT, MRI, and X-ray images for each patient in the ground truth image feature set are input into the initial support vector machine to obtain the set of predicted probabilities for recurrence after lumbar disc herniation surgery, and then compared with... N The recurrence prediction labels for each patient are used to calculate the fourth loss function using the hinge loss function; and the initial support vector machine is trained based on the fourth loss function. At the same time, the state of the initial support vector machine when it is minimized is recorded as the index-based recurrence prediction submodule.

6. The postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging according to claim 1, characterized in that, In the model training module, lumbar CT, lumbar MRI, and lumbar X-ray images of any patient from the lumbar spine image set are input into the lumbar disc herniation postoperative recurrence prediction model based on multimodal imaging to obtain the corresponding lumbar disc herniation postoperative recurrence prediction result for that patient. This specifically includes the following sub-steps: (a.1) First, the lumbar CT, lumbar MRI and lumbar X-ray images of any patient in the lumbar image set are input into the lumbar disc herniation postoperative recurrence prediction model based on multimodal images. The images are first processed by the image index feature generation network to obtain the true values ​​of CT, MRI and X-ray image feature prediction and input into the index-based recurrence prediction submodule to obtain the predicted probability of lumbar disc herniation postoperative recurrence of the patient. (a.2) Subsequently, the lumbar CT, lumbar MRI and lumbar X-ray images of any patient are processed by the image content processing model to obtain the image content feature results; (a.3) Finally, the predicted probability of recurrence after lumbar disc herniation surgery and the results of image content features are fused and input into the output layer to obtain the predicted result of recurrence after lumbar disc herniation surgery for this patient.

7. The postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging according to claim 6, characterized in that, Step (a.2) specifically includes the following sub-steps: (a.2.1) The lumbar CT, lumbar MRI and lumbar X-ray images of any patient are segmented into three-dimensional image blocks of the same size by the image content processing model to obtain a set of lumbar CT, lumbar MRI and lumbar X-ray image blocks. All image blocks in each image block set are projected to a high-dimensional space through a fully connected layer to generate lumbar CT, lumbar MRI and lumbar X-ray feature vectors and concatenate them with global classification features to obtain a multimodal feature token concatenation vector. Simultaneously, the position information of each image block in the corresponding image is recorded to obtain the set of position information of lumbar CT, lumbar MRI and lumbar X-ray image blocks, which is then concatenated with the global position embedding vector to obtain the fusion embedding representation vector of the multimodal medical image input sequence; Finally, the concatenated vector of multimodal feature tokens and the fused embedding representation vector of the multimodal medical image input sequence are added together to obtain the patient's initial input sequence. (a.2.2) The patient's initial input sequence was then processed. L After processing the Vim block structure of the layer and normalizing it, and mapping it to the multilayer perceptron, the final output sequence of the patient is obtained. (a.2.3) Finally, the patient's final output sequence is standardized, and then the image content feature results of the image content processing model are obtained through a multilayer perceptron.

8. The postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging according to claim 6, characterized in that, In the model training module, through N The recurrence prediction labels and postoperative recurrence prediction results of lumbar disc herniation for each patient were used to calculate the cross-entropy, resulting in the fifth loss function. The fifth loss function was then used to train the postoperative recurrence prediction model of lumbar disc herniation based on multimodal images.

9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the postoperative recurrence prediction system for lumbar disc herniation based on multimodal medical imaging as described in any one of claims 1-8.

Images