Construction method of AIS postoperative image prediction model, equipment and medium

Abstract

The invention provides a construction method, equipment, medium and program product of an AIS (Automatic Identification System) postoperative image prediction model, and a method, equipment, medium and program product for generating an AIS postoperative image based on a preoperative image, and relates to the field of intelligent medical treatment. According to the method, research of generating postoperative X-rays by adopting a conditional generation neural network is adopted, input data is changed and a Transform part in a module is optimized on the basis of a VQ-VAE framework so as to obtain a prediction model for providing an AIS operation value, and the generation network is applied to combination of a preoperative image and an operation fusion segment to generate a postoperative immediate image of a patient, so that the postoperative immediate image of the patient is obtained. By generating paired images rather than simple parameters, immediate surgical outcomes of the AIS can be predicted and evaluated before surgery.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical treatment, and more specifically, to a method, device, medium and program product for constructing an AIS postoperative image prediction model. Background Art

[0002] Adolescent idiopathic scoliosis (AIS) is defined as a spinal curvature of 10° or greater in individuals aged 10 to 18 years of age with unknown etiology. If left untreated, AIS can lead to severe physical deformity and chronic pain, and in severe cases, compromised lung and heart function due to chest compression. Posterior spinal fusion (PSF) surgery is commonly used to correct AIS spinal deformity and prevent progression of the spinal curvature. Predicting the outcome of PSF surgery can help surgeons make informed surgical decisions.

[0003] Several studies have been conducted to predict the changes in radiographic parameters after PSF. Although promising, these studies have significant drawbacks. First, they only predict the changes in parameters without providing a visual representation of the postoperative radiographs, which is crucial for spine surgeons to assess surgical outcomes. Second, the prediction models usually focus only on surgical outcomes within a specific fusion range, limiting their scope of applicability.

[0004] Artificial intelligence (AI) shows great potential in managing spinal diseases, including disease screening, classification, surgical decision-making, intraoperative procedures, and complication prediction. Generative neural networks, such as GANs and VAEs, are advanced AI models that can generate realistic synthetic data. In the medical field, they improve diagnostic accuracy by generating high-resolution medical images and aid in drug discovery by designing new molecules. These models also support personalized medicine by simulating patient responses to treatments and modeling disease progression to enable more effective early intervention. Overall, generative neural networks have the potential to revolutionize healthcare by enhancing diagnostics, accelerating drug development, and enabling customized treatment plans. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a method, device, medium and program product for constructing an AIS postoperative image prediction model, as well as a method, device, medium and program product for generating AIS postoperative images based on preoperative images; the present invention applies a generative network to combine preoperative images with surgical fusion segments to generate immediate postoperative images of patients, and by generating paired images rather than simple parameters, the immediate surgical results of AIS can be predicted and evaluated before surgery.

[0006] Accurate prediction of surgical outcomes after posterior spinal fusion (PSF) in adolescent idiopathic scoliosis (AIS) patients can assist surgeons in making decisions; AIS cases can generally be corrected with PSF posterior spinal fusion.

[0007] The first aspect of the present application discloses a method for constructing an AIS postoperative imaging prediction model, the method comprising: 101, obtaining a first preoperative image of a training set sample, inputting the first preoperative image into a machine learning model to output a second preoperative image; obtaining a first postoperative image of a training set sample, inputting the first postoperative image into a machine learning model to output a second postoperative image; respectively comparing the second preoperative image with the first preoperative image, and the second postoperative image with the first postoperative image, calculating the loss and optimizing to obtain a trained machine learning model; 102, obtaining a first preoperative image, a first postoperative image, and information of a scheduled surgical fusion segment of a training set sample; 103. Input the first preoperative image, the first postoperative image and the predetermined surgical fusion segment information into the trained machine learning model to obtain a predicted image, compare the predicted image with the first postoperative image, calculate the loss and optimize to obtain a postoperative image prediction model.

[0008] In some embodiments, the preoperative imaging includes any one or more of the following: X-ray imaging, computed tomography imaging, magnetic resonance imaging, ultrasound imaging, nuclear medicine imaging; Optionally, the preoperative images and the postoperative images include: coronal images, sagittal images, and transverse images; Optionally, the predetermined surgical fusion segment information includes any one or more of the following: UIV, LIV, any area of ​​the spine; Optionally, the vertebrae include: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and sacrum.

[0009] In some embodiments, the method further includes step 1021 between steps 102 and 103, preprocessing the preoperative image to obtain a preprocessed preoperative image; executing step 103, and replacing the preoperative image with the preprocessed preoperative image; Optionally, the preprocessing includes: marking an ROI region in the spine of the image, cropping the ROI region into a spine ROI image with a fixed aspect ratio; adjusting a pixel block of the cropped spine ROI image, the pixel block needs to include the entire spine; data enhancement adjusting the spine ROI image after the pixel block to enhance the robustness of the model and avoid sample imbalance; Optionally, the data enhancement method includes: random horizontal flipping, rotation, contrast enhancement and horizontal / vertical translation; Optionally, the resolution of the pixel block is 256*1024.

[0010] In some embodiments, the machine learning model is a generative model; Optionally, the machine learning model includes: VQ-VAE, GAN, Diffusion; preferably VQ-VAE; Optionally, the VQ-VAE model includes an encoder, a codebook, and a decoder; Optionally, the trained VQ-VAE model includes encoder, codebook, transformer, and decoder.

[0011] The second aspect of the present application discloses a method for generating AIS postoperative images based on preoperative images, the method comprising: Obtaining the subject's first preoperative image and scheduled surgical fusion segment information; The preoperative images and the predetermined surgical fusion segment information are input into the postoperative image prediction model described in the first aspect of the present application to obtain immediate postoperative images.

[0012] In some embodiments, the preoperative imaging includes any one or more of the following: X-ray imaging, computed tomography imaging, magnetic resonance imaging, ultrasound imaging, nuclear medicine imaging; Optionally, the preoperative images and the postoperative images include: coronal images, sagittal images, and transverse images; Optionally, the predetermined surgical fusion segment information includes any one or more of the following: UIV, LIV, any area of ​​the spine; Optionally, the vertebrae include: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and sacrum.

[0013] In some embodiments, the method further comprises: evaluating the surgical effect based on the immediate postoperative images; Optionally, the surgical effect can be evaluated by clinical parameters in the immediate postoperative imaging; Optionally, the method further comprises: customizing a surgical plan for the subject based on clinical parameters in the immediate postoperative images; Optionally, the clinical parameters include any one or more of the following: Cobb angle, UIV, LIV.

[0014] The third aspect of the present application discloses a computer device, which includes: a memory and a processor; the memory is used to store a computer program; and the processor executes the computer program to implement the steps of the above method.

[0015] A fourth aspect of the present application discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0016] A fifth aspect of the present application discloses a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0017] This application has the following beneficial effects: 1. This application innovatively discloses a method for constructing a postoperative image prediction model for AIS. This method uses a conditional generative network for the first time to generate postoperative images (X-rays). It not only provides spinal surgeons with valuable reference tools for AIS surgery, but also explores the wider application of generative neural networks in medicine. Specifically, the first part of this method first inputs the preoperative images and postoperative images into the machine learning model to learn and train the Encoder, Codebook and Decoder parts (the Transformer part is not involved) to learn the distribution characteristics of bone images; the second part repairs the Encoder, Codebook and Decoder parts, and trains the Transformer part, and obtains the prediction model by inputting preoperative images and fusion segment information training. The construction of the high-level model solves the problem that orthopedic surgeons can only predict parameter changes and cannot obtain a visual representation of postoperative images when predicting and evaluating the immediate surgical results of AIS before surgery; at the same time, the prediction model can not only control the surgical results within a specific fusion range, but also has a wider application. The above SVV-Net prediction model uses three evaluation methods: image generation quality analysis, postoperative imaging parameter evaluation, and clinical expert review to comprehensively evaluate the effectiveness of the generated postoperative images. Six image quality evaluation indicators are selected to compare the real postoperative X-rays with the generated X-rays; the consistency correlation coefficient (CCC) is used to evaluate the consistency between the Cobb angles of the generated X-rays and the real X-rays. Two experienced orthopedic surgeons independently scored the generated X-rays in five aspects: clarity, authenticity, accuracy, rationality, and usability. The evaluation results all showed that SVV-Net has high reliability.

[0018] 2. The preprocessing of images in this application enables the model to focus on the core area and improve the efficiency and effectiveness of the generated results. The preprocessing measures are designed to standardize X-rays from different sources. In the specific process of preprocessing, attempts were made to adjust the images to different sizes, and it was found that 256x1024 resolution was the best resolution. However, this processing method will inevitably lead to some data loss, as factors such as shoulder balance will also affect the results. Since the attention mechanism can adjust the model's focus on different areas, it can be used to improve performance in the spine area.

[0019] 3. This application innovatively generates paired images instead of simple parameters to achieve the immediate prediction and evaluation of AIS surgical results before surgery. At the same time, the predicted images can provide information for surgical decisions, such as determining the upper and lower fused vertebrae (UIV, LIV). In addition, the measurements derived from the generated X-rays can provide the radiological parameters necessary to evaluate surgical results in a standard manner.

[0020] In summary, this application solves new technical problems, uses new data, and has new application scenarios. The research on generating postoperative X-rays using a conditional generative neural network changes the input data and optimizes the Transformer part of the module on the basis of the VQ-VAE architecture to obtain a predictive model that provides the value of AIS surgery, and its effect has been confirmed. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 is a schematic diagram of a method flow chart provided by the first aspect of an embodiment of the present invention; Figure 2 is a schematic diagram of a method flow chart provided by the second aspect of an embodiment of the present invention; Figure 3 Schematic diagram of a system for constructing an AIS postoperative imaging prediction model provided by an embodiment of the present invention; Figure 4 is a schematic diagram of a system for generating AIS postoperative images based on preoperative images provided by an embodiment of the present invention; Figure 5 is a schematic diagram of a computer device provided by an embodiment of the present invention; Figure 6 is a schematic diagram of the architecture of an exemplary computing device provided by an embodiment of the present invention; Figure 7 is a schematic diagram of a storage medium provided by an embodiment of the present invention; Figure 8 The structure, training and reasoning process of the postoperative image prediction model (SVV-net) provided in an embodiment of the present invention; Fig. 9 is the result output after the preoperative image is input into the postoperative image prediction model provided by the embodiment of the present invention; wherein, Fig. 9 a, Fig. 9 b, Fig. 9 c, Fig. 9 d, Fig. 9 e, Fig. 9 f, Fig. 9 g, Fig. 9 The sub-figures of h present a case with a certain severity and type of scoliosis curve. Each sub-figure shows the preoperative X-ray, postoperative X-ray (true), first stage output image, and generated postoperative X-ray from left to right. It can be seen that compared with the real X-ray, the X-ray generated by our model shows a high degree of similarity; Fig.10 is the CCC and Bland-Altman diagram of the Cobb angle between the model-generated X-ray film and the real X-ray film provided by the embodiment of the present invention; wherein, Fig.10 a and Fig.10 b shows the CCC results of the Cobb angle between the generated and true X-rays in the test cohort and prospective test cohort, respectively; Fig.10 c and Fig.10 d shows the Bland-Altman plot of the Cobb angle between the generated and true X-rays in the test cohort and the prospective test cohort, respectively; CCC = Concordance correlation coefficient; Fig.11 The embodiment of the present invention provides a method for generating a corrected Cobb angle between an X-ray film and a real X-ray film using a CCC and Bland-Altman Plot analysis model; Fig.11 a and Fig.11 b shows the CCC results of the corrected Cobb angle between the generated and true X-rays in the test cohort and prospective test cohort, respectively; Fig.11 c and Fig.11 d shows the Bland-Altman plots of the corrected Cobb angles between generated and true X-rays in the test cohort and the prospective test cohort, respectively; Ccc = Concordance correlation coefficient; Fig.12 Schematic diagram of LIV and UIV between a generated X-ray and a real X-ray provided by an embodiment of the present invention; UIV = upper fused vertebra, LIV = lower fused vertebra. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0024] In some of the processes described in the specification and claims of the present invention and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0026] Figure 1 1 is a flow chart of a method for constructing an AIS postoperative imaging prediction model provided by an embodiment of the present invention. Specifically, the method comprises the following steps: 101: Obtain a first preoperative image of a training set sample, input the first preoperative image into a machine learning model to output a second preoperative image; obtain a first postoperative image of a training set sample, input the first postoperative image into a machine learning model to output a second postoperative image; respectively compare the second preoperative image with the first preoperative image, and the second postoperative image with the first postoperative image, calculate the loss, and optimize to obtain a trained machine learning model; 102, obtaining a first preoperative image, a first postoperative image, and information of a scheduled surgical fusion segment of a training set sample; In some embodiments, the predetermined surgical fusion segment information includes any one or more of the following: UIV, LIV, any area of ​​the spine; Optionally, the vertebrae include: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and sacrum.

[0027] In some embodiments, the method further includes step 1021 between steps 102 and 103, preprocessing the preoperative image to obtain a preprocessed preoperative image; executing step 103, and replacing the preoperative image with the preprocessed preoperative image; Optionally, the preprocessing includes: marking an ROI region in the spine of the image, cropping the ROI region into a spine ROI image with a fixed aspect ratio; adjusting a pixel block of the cropped spine ROI image, the pixel block needs to include the entire spine; data enhancement adjusting the spine ROI image after the pixel block to enhance the robustness of the model and avoid sample imbalance; Optionally, the data enhancement method includes: random horizontal flipping, rotation, contrast enhancement and horizontal / vertical translation; Optionally, the resolution of the pixel block is 256*1024.

[0028] 103. Input the first preoperative image, the first postoperative image and the predetermined surgical fusion segment information into the trained machine learning model to obtain a predicted image, compare the predicted image with the first postoperative image, calculate the loss and optimize to obtain a postoperative image prediction model.

[0029] In some embodiments, the preoperative imaging includes any one or more of the following: X-ray imaging, computed tomography imaging, magnetic resonance imaging, ultrasound imaging, nuclear medicine imaging; Optionally, the preoperative images and the postoperative images include: coronal images, sagittal images, and transverse images; In some embodiments, the machine learning model is a generative model; Optionally, the machine learning model includes: VQ-VAE, GAN, Diffusion; preferably VQ-VAE; Optionally, the VQ-VAE model includes an encoder, a codebook, and a decoder; Optionally, the trained VQ-VAE model includes encoder, codebook, transformer, and decoder.

[0030] The second aspect of the present embodiment further discloses a method for generating AIS postoperative images based on preoperative images, the method comprising: 201, obtaining a first preoperative image and scheduled surgical fusion segment information of the subject; In some embodiments, the preoperative imaging includes any one or more of the following: X-ray imaging, computed tomography imaging, magnetic resonance imaging, ultrasound imaging, nuclear medicine imaging; Optionally, the preoperative images and the postoperative images include: coronal images, sagittal images, and transverse images; Optionally, the predetermined surgical fusion segment information includes any one or more of the following: UIV, LIV, any area of ​​the spine; optionally, the spine includes: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and sacrum.

[0031] 202. Input the preoperative image and the predetermined surgical fusion segment information into the postoperative image prediction model described in the first aspect of the present application to obtain an immediate postoperative image.

[0032] In some embodiments, the method further comprises: evaluating the surgical effect based on the immediate postoperative images; Optionally, the surgical effect can be evaluated by clinical parameters in the immediate postoperative imaging; Optionally, the method further comprises: customizing a surgical plan for the subject based on clinical parameters in the immediate postoperative images; Optionally, the clinical parameters include any one or more of the following: Cobb angle, UIV, LIV.

[0033] In some embodiments, the term "subject" or "person to be tested" or "sample to be tested" as used herein refers to any animal (e.g., mammal), including but not limited to humans, non-human primates, rodents, etc., which will be the recipient of a specific treatment. Generally, the terms "subject" and "patient" are used interchangeably herein when referring to a human subject. Preferably, the subject is a human.

[0034] In some embodiments, the sample to be tested is a patient undergoing clinical prognosis assessment.

[0035] Figure 5 is a schematic diagram of a computer device provided by an embodiment of the present invention, such as Figure 5 As shown, the device may include: one or more processors, and one or more memories; wherein the memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the method described above may be executed.

[0036] The processor in this embodiment can be an integrated circuit chip with signal processing capabilities. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture.

[0037] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0038] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Figure 6 The architecture of the computing device 3000 shown in FIG. Figure 6 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as ROM 3030 or hard disk 3070, may store various data or files used for processing and / or communication of the method provided by the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 6 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 6 One or more components of a computing device are shown.

[0039] The embodiment of the present invention also provides a computer-readable storage medium, such as Figure 7As shown, it is a schematic diagram of a storage medium provided by an embodiment of the present invention, and a computer readable instruction 4010 is stored on the computer storage medium 4020. When the computer readable instruction 4010 is executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0040] The embodiments of the present disclosure further provide a computer program product or system, including a computer program, which implements the steps of the above method when executed by a processor.

[0041] In some embodiments, this embodiment also discloses a system for constructing an AIS postoperative imaging prediction model, such as Figure 3 As shown, the system comprises: The image learning training module 301 is used or configured to obtain a first preoperative image of a training set sample, input the first preoperative image into a machine learning model to output a second preoperative image; obtain a first postoperative image of a training set sample, input the first postoperative image into a machine learning model to output a second postoperative image; respectively compare the second preoperative image with the first preoperative image, and the second postoperative image with the first postoperative image, calculate the loss, and optimize to obtain a trained machine learning model; A first acquisition module 302 is used or configured to acquire a first preoperative image, a first postoperative image, and predetermined surgical fusion segment information of a training set sample; A model generation module 303 is used or configured to input the first preoperative image, the first postoperative image and the predetermined surgical fusion segment information into the trained machine learning model to obtain a predicted image, compare the predicted image with the first postoperative image, calculate the loss and optimize to obtain a postoperative image prediction model; In some embodiments, the system further includes a pre-processing module located between the image learning training module 301 and the first acquisition module 302; The preprocessing module is used for or configured to preprocess the preoperative image to obtain a preprocessed preoperative image.

[0042] In some embodiments, the preprocessing includes: marking the ROI area in the spine of the image, cropping the ROI area into a spine ROI image with a fixed aspect ratio; adjusting the pixel blocks of the cropped spine ROI image, which pixel blocks need to include the entire spine; data enhancement of the spine ROI image after adjusting the pixel blocks to enhance the robustness of the model and avoid sample imbalance.

[0043] In some embodiments, this embodiment also discloses a system for generating AIS postoperative images based on preoperative images, such as Figure 4 As shown, the system comprises: The second acquisition module 401 is used or configured to acquire a first preoperative image of the subject and predetermined surgical fusion segment information; An immediate postoperative image output module 402 is used or configured to input the preoperative image and the predetermined surgical fusion segment information into the postoperative image prediction model described in the first aspect of the present application to obtain an immediate postoperative image; In some embodiments, the system further includes a surgery effect evaluation module 4031 and / or a surgery planning customization module 4032 located after the immediate postoperative image output module 402; A surgical effect evaluation module 4031 is used or configured to evaluate the surgical effect based on the immediate postoperative image; optionally, the surgical effect is evaluated by clinical parameters in the immediate postoperative image; A surgical planning customization module 4032, used for or configured to customize a surgical plan for the subject based on clinical parameters in the immediate postoperative image; Optionally, the clinical parameters include any one or more of the following: Cobb angle, UIV, LIV.

[0044] Example

[0045] Materials and methods Data collection and preparation All AIS patients aged 10 to 18 years who underwent PSF surgery at Peking Union Medical College Hospital (PUMCH) from July 1, 2012 to June 1, 2023 were recruited to form the training and internal test data cohorts for model development and testing under the DISCO study framework. Consecutive AIS patients who underwent PSF surgery at Peking Union Medical College Hospital from August 1, 2023 to May 1, 2024 were included in our prospective testing data cohort. Exclusion criteria: (1) Unclear diagnosis of AIS or comorbidity with other bone-related diseases; (2) Age younger than 10 years or older than 18 years; (3) Unclear preoperative and immediate postoperative anteroposterior radiographs; (4) Lack of necessary UIV and LIV surgical information; (5) Main curve Cobb angle ≥ 80°. The retrospectively collected participants will be randomly divided into the training cohort and the internal test cohort in an 8:2 ratio, and the prospectively included participants will serve as the prospective testing cohort. This study was approved by the Ethics Committee of Peking Union Medical College Hospital, and the patients or their guardians provided informed consent.

[0046] Measuring imaging parameters Two orthopedic surgeons measured the collected imaging data, including preoperative, postoperative, and model-generated images. First, they recorded the number and type of curves in the images. Curve types included: upper thoracic curve (apex between T1 and T5), main thoracic curve (apex between T6 and T12), thoracolumbar curve (apex between T12 and L1 or L2), and lumbar curve (apex between L1 and L5). In addition, they also recorded and measured the position of the end vertebra of each curve and the corresponding Cobb angle. The effective Cobb angle was the average of the Cobb angles measured by the two surgeons. If the two surgeons disagreed on the selection of the upper and lower end vertebrae, the final decision was reached through discussion.

[0047] Image preprocessing To minimize the input image variance and improve the quality of the predicted images, we manually annotated the region of interest (ROI) of the spine (C7 ~ L5 vertebrae) in all original images and cropped the region to a fixed aspect ratio of 4:1. Subsequently, the cropped spine ROI was resized to a 256x1024 pixel block, covering the entire spine. Finally, to enhance the robustness of the model and avoid sample imbalance, we applied data augmentation techniques, including random horizontal flipping (p = 0.5), rotation ([-5°,5°]), contrast enhancement ([0.8,1.2]), and horizontal / vertical translation ([-10 pixels, 10 pixels]).

[0048] Model design and structure According to the characteristics of our data, we developed an X-ray generation model based on VQ-VAE (Vector Quantized Variational Autoencoder) and named it SVV-Net (Scoliosis-VQ-VAE). Our model is as follows Figure 8 As shown, it includes four parts. The first part, the encoder, consists of a hierarchically connected convolutional neural network layer for compressing imaging data features. After the original image passes through the Encoder structure, it is compressed from 256x1024 pixels to a 16x32 feature layer. The second part is the structure of the Codebook. The Encoder outputs a latent vector, which is then matched with the closest vector in the Codebook using the nearest neighbor search. This process converts continuous latent representations into discrete latent representations, thereby more effectively capturing the discrete nature of the data. The third part is the decoder, which corresponds to the encoder; the 16x32 feature layer is restored to a 256x1024 image through the Decoder structure. The fourth part is the Transformer structure, which mainly fuses the feature vectors of the preoperative X-ray and the predetermined surgical segment to generate the corresponding postoperative X-ray feature vector.

[0049] Model training and inference process The entire training and inference process is divided into three stages. In the first stage, we trained the Encoder, Codebook, and Decoder parts, while the Transformer part was not involved. Both preoperative and postoperative images are input into the network, the final output image is compared with the input image, the loss is calculated, and the model is optimized. This stage aims to enable the network to learn the distribution characteristics of the image. In the second stage, we fix the Encoder, Codebook, and Decoder parts, and train the Transformer part. In this stage, the model input consists of preoperative X-rays and fusion segments, and the true value is the real postoperative X-ray. The third stage is the reasoning and image generation part of the model. We input the preoperative X-ray into the model, as well as the required UIV and LIV. The trained model can generate the corresponding postoperative X-ray PSF. The detailed training and inference process is as follows. Figure 8As shown. The "frozen" in it usually refers to the "freezing" technology. The following is a detailed description of the freezing technology: Pre-training and freezing: In deep learning, pre-training refers to using a large amount of unlabeled data to train the model so that it can learn common features. Pre-trained models (such as ResNet, Inception, etc.) usually contain millions or even hundreds of millions of parameters. Freezing refers to freezing most of the parameters of the model during the pre-training process and only updating the top layer or part of the layer of the network to adapt to the new task. Doing so can retain the common features learned by the pre-trained model while reducing training time and computing resources. "Frozen" in machine learning refers to the freezing technology, which is an effective transfer learning method. By freezing most of the parameters of the pre-trained model and only fine-tuning it for the new task, it can save training time and computing resources, while using the common features of the pre-trained model to improve the performance of the new task.

[0050] Figure 8 The encoder in the example is a convolutional encoder that compresses the input image into a low-dimensional latent representation; the codebook is a discrete codebook containing context-rich visual parts; discretization is to quantize the encoder output to the codebook encoding position with the closest spatial distance; the decoder is a convolutional decoder that reconstructs the image from the quantized latent representation. The codebook is used to encode the image into a sequence of discrete values.

[0051] Evaluation indicators To comprehensively evaluate the effectiveness of the generated postoperative X-rays, we used three evaluation methods: image generation quality evaluation, postoperative imaging parameter evaluation, and clinical expert evaluation.

[0052] Image Generation Quality Assessment: Six image quality evaluation indicators, including peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), visual information fidelity (VIF), Frechet inception distance (FID), learned perceptual patch similarity (LPIPS) and natural image quality evaluation (NIQE), were selected to compare the real postoperative X-rays and the generated X-rays, and our model was compared with the models based on Diffusion, GAN and VAE.

[0053] Postoperative imaging parameters: We manually identify the fusion level in the generated radiographs and assess its consistency with the intended surgical level. In addition, we calculate the difference in Cobb angle between the generated images and the true postoperative radiographs, as well as the correction of the Cobb angle.

[0054] Clinical expert evaluation: Two experienced orthopedic surgeons were invited to independently rate the generated X-rays in terms of clarity, authenticity, accuracy, rationality, and ease of use. Each aspect required manual evaluation on a scale of 0 to 10, and the total score of the five aspects was calculated as the overall evaluation result.

[0055] Statistical analysis Baseline characteristics of the study participants were described using mean values ​​(standard deviations) for continuous variables and frequencies (percentages) for categorical variables. Kruskal-Wallis and Chi-Square tests were used to compare the differences in variables among the 3 cohorts. Linear regression and Bland-Altman analysis were performed to quantitatively evaluate the validity of the Cobb angle estimated from postoperative radiographs and the corrected Cobb angle. In linear regression, the consistency correlation coefficient (CCC) was used to evaluate the agreement between the Cobb angle of the generated radiographs and the true radiographs. In addition, we performed further subgroup analysis based on the curve type. Finally, the mean scores and standard deviations provided by the two reviewers for different evaluation aspects were calculated. All statistical methods in this study were performed using R, and a two-tailed P value of less than 0.05 was considered statistically significant.

[0056] result Data Queue Description Between July 1, 2012, and June 1, 2023, 720 patients met the criteria and were enrolled in the study; they were randomly assigned to a training cohort of 540 participants and an internal testing cohort of 180 participants. From August 1, 2023, to May 1, 2024, a total of 79 patients were enrolled in the prospective testing cohort. There was no overlap of patients between the model development and prospective testing cohorts. Detailed demographic characteristics of the cohorts are shown in Table 1 , with no significant differences in age, sex, number of curves, curve location, curve Cobb angle, UIV, or LIV. The mean age of the training cohort was 14.4 ± 1.7 years, and 87.8% were female; the mean age of the testing cohort was 14.4 ± 1.7 years, and 85.0% were female; and the prospective cohort, mean age, was 14.5 ± 1.6 years, and 89.9% were female. 61% of the training cohort, 53% of the testing cohort, and 68% of the prospective cohort had 2 curves, with the dominant curve usually being the main thoracic curve. The main curve Cobb angles of the training cohort, internal testing cohort, and prospective testing cohort were 41.53 ± 14.09, 39.29 ± 12.03, and 43.05 ± 14.51, respectively.

[0057] Table 1 Demographic information of the study population

[0058] UIV = upper fused vertebra, LIV = lower fused vertebra Performance of a deep learning framework for X-ray synthesis Patients with different severity and types of spinal curves demonstrate the comprehensive performance of our model in various situations ( Fig. 9 ). From left to right, the preoperative X-ray, postoperative X-ray (real), first stage output image, and generated postoperative X-ray are shown. It can be seen that the X-ray generated by our model is highly similar to the real X-ray. In order to determine the optimal design of the model, the GAN, Diffusion and VAE models were compared and analyzed. Multiple experiments were conducted to determine the optimal Codebook length. Performance evaluation was performed using PSNR, VIF, FID, LPIPS and NIQE indicators. The results show that SVV-net outperforms other generative models in VIF, FID, LPIPS and NIQE indicators, and performs best when the Code slicer length is 256. Reliability of postoperative imaging measurements Cobb angle: The Cobb angle measured from the generated images showed high agreement with the Cobb angle measured from the real images (internal test (CCC: 0.90 (0.88, 0.93)), pretest 0.91 (86, 0.94)). Bland-Altman analysis of the internal test cohort showed an average difference of 0.5° between the predicted and real Cobb angles, with a 95% agreement limit of -2.5° to 3.5°. In the prospective test cohort, the average difference was 0.7°, with a 95% agreement limit of -3.0° to 4.4°. For different scoliosis locations, the main thoracic curve showed the highest agreement in both the internal validation cohort and the prospective validation cohort, with CCCs of 0.91 (0.87, 0.94) and 0.93 (89, 0.96), respectively. The consistency of the thoracolumbar curves was relatively limited, with CCCs of 0.77 (0.68, 0.88) and 0.82 (0.58, 0.96), respectively ( Fig.10 ). Correction of Cobb angle: Similarly, we analyzed the difference in Cobb angle improvement between postoperative X-rays of real images and generated images compared with preoperative X-rays. Both the internal test cohort and the prospective test cohort showed high consistency (internal test cohort (CCC: 0.97 (0.96,0.98)), prospective test cohort 0.96 (0.95,0.98)). The Bland-Altman plot showed that the difference in Cobb angle improvement between the two in the internal test cohort and the prospective test cohort was -0.15 (-5.28,4.99) and -0.58 (-6.59,5.43), respectively. Fig.11 ). UIVs and LIVs: From Fig.12 As can be seen, the generated X-ray UIVs and LIVs have high reliability compared with the real postoperative X-rays, with most cases distributed on the diagonal line. In the test cohort, the difference in UIVs was 0.04 (± 0.58) and the difference in LIVs was - 0.06 (± 1.13). In the prospective test cohort, the difference in UIVs was - 0.05 (± 0.22) and the difference in LIVs was - 0.01 (± 0.20) ( Fig.12 ). Clinical expert evaluation: As shown in Table 2, both reviewers showed positive evaluation results in the five aspects of the image. In the internal test cohort, reviewer 1 gave the highest scores for image clarity and realism, which were 7.61 (±1.07) and 7.54 (±1.29), respectively. Reviewer 2 gave the highest scores for usability and rationality, which were 7.58 (±1.81) and 7.57 (±1.40), respectively. The average total score of the five aspects of the two reviewers was 37.3 (±5.3). In the prospective test cohort, both reviewers gave high average scores, with the highest average score for usability, which was 8.18 (±1.04).

[0059] Table 2 Results of manual evaluation of images by two evaluators from five aspects

[0060] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0061] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0062] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0063] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0064] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0065] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0066] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for constructing an AIS postoperative imaging prediction model, characterized in that: The method comprises: 101, obtaining a first preoperative image of a training set sample, inputting the first preoperative image into a machine learning model to output a second preoperative image; obtaining a first postoperative image of a training set sample, inputting the first postoperative image into a machine learning model to output a second postoperative image; respectively comparing the second preoperative image with the first preoperative image, and the second postoperative image with the first postoperative image, calculating the loss and optimizing to obtain a trained machine learning model; 102, obtaining a first preoperative image, a first postoperative image, and information of a scheduled surgical fusion segment of a training set sample; 103. Input the first preoperative image, the first postoperative image and the predetermined surgical fusion segment information into the trained machine learning model to obtain a predicted image, compare the predicted image with the first postoperative image, calculate the loss and optimize to obtain a postoperative image prediction model.

2. The method for constructing the AIS postoperative imaging prediction model according to claim 1, characterized in that: The preoperative imaging includes any one or more of the following: X-ray imaging, computerized tomography imaging, magnetic resonance imaging, ultrasound imaging, and nuclear medicine imaging; Optionally, the preoperative images and the postoperative images include: coronal images, sagittal images, and transverse images; Optionally, the predetermined surgical fusion segment information includes any one or more of the following: UIV, LIV, any area of ​​the spine; Optionally, the vertebrae include: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and sacrum.

3. The method for constructing the AIS postoperative imaging prediction model according to claim 1, characterized in that: The method further includes step 1021 between steps 102 and 103, preprocessing the preoperative image to obtain a preprocessed preoperative image; executing step 103, and replacing the preoperative image with the preprocessed preoperative image; Optionally, the preprocessing includes: marking the ROI region in the spine of the image, cropping the ROI region into a spine ROI image with a fixed aspect ratio; adjusting the pixel block of the cropped spine ROI image, the pixel block needs to include the entire spine; data enhancing the spine ROI image after adjusting the pixel block; Optionally, the data enhancement method includes: random horizontal flipping, rotation, contrast enhancement and horizontal / vertical translation; Optionally, the resolution of the pixel block is 256*1024.

4. The method for constructing the AIS postoperative imaging prediction model according to claim 1, characterized in that: The machine learning model is a generative model; Optionally, the machine learning model includes: VQ-VAE, GAN, Diffusion; preferably VQ-VAE; Optionally, the VQ-VAE model includes an encoder, a codebook, and a decoder; Optionally, the trained VQ-VAE model includes encoder, codebook, transformer, and decoder.

5. A method for generating AIS postoperative images based on preoperative images, characterized in that: The method comprises: Obtaining the subject's first preoperative image and scheduled surgical fusion segment information; The preoperative image and the predetermined surgical fusion segment information are input into the postoperative image prediction model according to any one of claims 1 to 4 to obtain an immediate postoperative image.

6. The method for generating AIS postoperative images based on preoperative images according to claim 5, characterized in that: The preoperative imaging includes any one or more of the following: X-ray imaging, computerized tomography imaging, magnetic resonance imaging, ultrasound imaging, and nuclear medicine imaging; Optionally, the preoperative images and the postoperative images include: coronal images, sagittal images, and transverse images; Optionally, the predetermined surgical fusion segment information includes any one or more of the following: UIV, LIV, any area of ​​the spine; Optionally, the vertebrae include: cervical vertebrae, thoracic vertebrae, lumbar vertebrae, and sacrum.

7. The method for generating AIS postoperative images based on preoperative images according to claim 5, characterized in that: The method further comprises: evaluating the surgical effect based on the immediate postoperative images; Optionally, the surgical effect can be evaluated by clinical parameters in the immediate postoperative imaging; Optionally, the method further comprises: customizing a surgical plan for the subject based on clinical parameters in the immediate postoperative images; Optionally, the clinical parameters include any one or more of the following: Cobb angle, UIV, LIV.

8. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images