An intraoperative image analysis system, method and use thereof

Abstract

This application provides an intraoperative image analysis system, method, and application. The intraoperative image analysis system includes: a video acquisition module, a preprocessing module, a risk prediction module, a hybrid expert system integration module, and a comprehensive risk prediction module. Through this analysis system, this application can achieve accurate assessment of visceral pleural invasion (VPI) during surgery, providing objective quantitative indicators for surgical planning.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an intraoperative image analysis system, method, and application thereof. Background Technology

[0002] Lung cancer is currently the leading cause of death among malignant tumors, and surgery is the most likely way to cure early-stage lung cancer. To preserve as much normal lung tissue as possible while completely removing the tumor lesion, sublobar resection is recommended for stage IA patients. However, visceral pleural invasion (VPI) is an important staging indicator and an independent adverse prognostic factor for lung cancer, significantly associated with lymph node metastasis, lymphovascular invasion, and local recurrence. According to the latest TNM staging system for lung cancer, even small tumors with a maximum diameter ≤3cm will be upgraded from stage IA to stage IB when VPI occurs, requiring lobectomy. Therefore, accurate prediction of VPI is of significant guiding value for surgical planning. Currently, the gold standard for VPI diagnosis is postoperative pathological examination; however, this method is lagging, increasing the risk of secondary surgery for some patients.

[0003] Artificial intelligence analysis of multimodal non-invasive images such as preoperative CT, MRI, and PET / CT has shown significant value in VPI assessment. However, the information provided by preoperative imaging is limited, resulting in limited sensitivity and specificity, and it is difficult to reflect tumor progression between imaging and surgery. Intraoperative video images during video-assisted thoracoscopic surgery (VATS) can reflect the current state of the lesion and are of great value for VPI diagnosis and surgical planning decisions.

[0004] Current VPI diagnosis based on thoracoscopic video relies heavily on the subjective assessment of thoracic surgeons, making it difficult to extract the large amount of microscopic information contained in the images. The accuracy rate is low, less than 60%, and due to differences in experience, the diagnostic results vary greatly among different assessors. Summary of the Invention

[0005] To address the aforementioned problems, this application proposes an intraoperative image analysis system that can accurately assess VPI during surgery, providing objective quantitative indicators for surgical planning and overcoming the shortcomings and defects mentioned in the background art.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] The present invention provides an intraoperative image analysis system, comprising: a video acquisition module, a preprocessing module, a risk prediction module, a hybrid expert system integration module, and a comprehensive risk prediction module; the video acquisition module contains intraoperative video images; the preprocessing module filters and labels the video images to obtain multiple images containing lesions; the risk prediction module inputs the multiple images containing lesions into a residual neural network model based on spatial region random inactivation to obtain multiple image risk probabilities; then, the multiple image risk probabilities are aggregated to obtain a preliminary risk probability; the hybrid expert system integration module divides the risk prediction task into multiple sub-tasks, trains different models as experts for different inactivation region sizes, and integrates them through a gating model to obtain the image risk probability; the comprehensive risk prediction module calculates and outputs the comprehensive risk probability based on the patient's image risk probability value and clinical information; the calculation method is: inputting the risk probability and clinical information into a comprehensive risk regression model for regression prediction to obtain the comprehensive risk probability.

[0008] In the preprocessing module, the screening includes: decomposing the video image frame by frame to obtain multiple images; after analysis, screening video frames in which the lesion area is not obscured, thereby obtaining multiple images to be analyzed.

[0009] like Figure 1 As shown, in the imaging data, there may be images of lesions, surgical instruments, blood, etc. coexisting with the lesions. This coexistence may be a situation where the surgical instruments, blood, etc. do not obscure each other or obscure each other with the lesions. It is necessary to manually screen out the video frames that are not obscured by surgical instruments, blood, etc. to avoid deviations caused by obscuring and improve accuracy.

[0010] Optionally, the labeling includes: outlining the lesion and surrounding area as a region of interest in multiple images to be analyzed. The region of interest contains all tumors but does not contain surgical instruments, etc., and then the region of interest is extracted to obtain multiple images containing the lesion.

[0011] Specifically, in the image to be analyzed, a rectangular box (other shapes are also acceptable, but a rectangular box is preferred for the convenience of subsequent models) is used to delineate the lesion and the surrounding area as the region of interest. The standard size of the rectangular box is that the region of interest contains all tumors but does not contain surgical instruments, etc. Then, the region of interest is extracted and resampled to 128x128 as the input image for the neural network.

[0012] Optionally, in the risk prediction module, random spatial region deactivation includes: deactivating a region with a side length of block_size in the input image with a side length of L using p dr The probability of random inactivation;

[0013] The specific process is as follows: A Bernoulli distribution with parameter γ is used to generate the center point of the region to be deactivated. Then, this center point is used to expand to a block area of ​​size block_size, generating the final region to be deactivated. The formula for calculating γ is as follows:

[0014]

[0015] Where L is the side length of the input image, p dr Let `block_size` be the deactivation probability, and `p` be the side length of the deactivation region. Using spatial random deactivation can eliminate the information loss caused by overexposure of the endoscope image. This involves deactivating a region of side length `block_size` in the input image of side length `L` using `p`. dr The probability of random inactivation is determined by the side length block_size and the inactivation probability p. dr Set as a hyperparameter, determined through grid search and cross-validation.

[0016] Optionally, the residual neural network model includes: a convolutional layer, an average pooling layer, multiple residual stacked modules, a global average pooling layer, and at least one fully connected layer; then, the image risk probability is obtained through a Soft max classifier function.

[0017] Preferably, the residual module includes, in sequence, a 1x1 convolutional layer, a 3x3 convolutional layer, and a 1x1 convolutional layer, and the input of the residual module is skipped between the second 1x1 convolutional layer.

[0018] Preferably, the residual stacking module includes 5 residual modules; the number of the 5 residual modules is 2, 2, 2, 1 and 1 respectively according to the direction of propagation;

[0019] Preferably, the loss function used is the binary cross-entropy loss.

[0020] Specifically, such as Figure 2 As shown, the image is first mapped from 3D to 32D using 32 large 7x7 convolutional kernels, enabling subsequent residual blocks to extract features more effectively. Next, average pooling with a stride of 2 is used to reduce the image size from 128 to 64. Then, five modules consisting of residual blocks are used, each module using 2, 2, 2, 1, and 1 residual blocks with the same number of output channels, respectively. Each residual block reduces the number of image channels using a 1x1 convolutional kernel, and after a 3x3 convolution, the number of channels is increased again using a 1x1 convolutional kernel.

[0021] The features encoded by the residual neural network are used for risk prediction through global average pooling and soft max. To reduce the impact of data imbalance, focal loss is used as the loss function to obtain the image risk probability p. imgrisk.

[0022] Optionally, in the risk prediction module, the risk probabilities of multiple images are fitted to conform to a Gaussian distribution with mean μ and variance σ; then, the average value of the risk probabilities located in (μ+σ, μ-σ) is taken to obtain the preliminary risk probability, as shown in the following formula.

[0023]

[0024] Where sgn is the sign function, p imgrisk Let be the image risk probability, and i be the image number containing the lesion. Specifically, if the risk probabilities of multiple images of the same patient conform to a Gaussian distribution, the mean of the Gaussian distribution is obtained as μ and the variance as σ using the maximum likelihood estimation method; then, the risk probabilities located in (μ+σ, μ-σ) are averaged to obtain the preliminary risk probability of the patient based on endoscopic video images.

[0025] Considering the different sizes of overexposed areas, it is necessary to address the issue of different sizes of inactive areas. Different models are trained as experts for different sizes of inactive areas, and then integrated through gating models to obtain the image risk probability.

[0026] Optionally, the risk prediction task can be divided into multiple sub-tasks, and different models can be trained as experts for different sizes of inactivation regions. These models can then be integrated through a gating model to obtain the image risk probability.

[0027] Optionally,

[0028]

[0029] Where K is the number of experts, S i Weight S to be trained i ∈(0,1), E i (x) represents the initial risk probability of the model under different hyperparameters.

[0030] For example, when K=3, it is for two hyperparameters (side length block_sizeh and inactivation probability p) dr The model is optimized three times through grid search, and a preliminary prediction probability is obtained based on the optimal parameters of each time.

[0031] That is: the first time, the side length block_sizeh and the inactivation probability p are set. dr Set as hyperparameters, determined through grid search and cross-validation, to obtain the initial block length (block_size) and inactivation probability (p). dr ; The region with side length of block_size in the input image with side length L is divided into p drThe probability of random inactivation is input into a residual neural network model, and the first multi-image risk probability p is obtained after training. imgrisk Then, the risk probabilities of multiple images are fitted to conform to a Gaussian distribution with mean μ and variance σ; then, the risk probabilities within (μ+σ, μ-σ) are averaged to obtain the first preliminary risk probability.

[0032] Then, a second step is performed, using grid search and cross-validation to determine the second side length (block_size) and the inactivation probability (p). dr Then, following the method described above, the risk probabilities p of multiple images for the second time were obtained. imgrisk ; and finally obtain the preliminary risk probability for the second time.

[0033] Then, a third step is performed, using the same method to obtain the third side length (block_size) and the inactivation probability (p). dr The probability p of the third image risk imgrisk And the preliminary risk probability for the third time.

[0034] Since the location of random deactivation is random, each determination yields the block length (block_size) and the deactivation probability (p). dr They are all different or not completely the same.

[0035] K represents the repeated determination value (side length block_size and inactivation probability p). dr The number of iterations (including the corresponding image risk probability and preliminary risk probability) is K, where K ranges from 1 to 20, preferably 1 to 10, and more preferably 1 to 5. In other words, K is the number of iterations of the risk prediction module.

[0036] The training of the model in the risk prediction module includes: 1. Parameter optimization training process; 2. The process of retraining the model based on the optimal parameters, and then obtaining the preliminary risk probability.

[0037] Optionally, in the comprehensive risk prediction module, the probability of imaging risk and clinical information are used as independent variables, and whether VPI invasion occurs is used as the dependent variable. A logistic regression prediction model is constructed, and the comprehensive risk probability of whether VPI occurs is output as the analysis result.

[0038] Optional clinical information includes tumor diameter and tumor type (solid or non-solid nodule).

[0039] Specifically, such as Figure 3As shown, the imaging risk probability (VPI-Net), tumor diameter, and tumor type are used as independent variables, and whether VPI invasion occurs is used as the dependent variable. These are input into the logistic model to train a comprehensive risk regression model. The output of the risk probability of whether VPI occurs is used as the comprehensive risk probability of whether the patient has VPI (VPI-Net-Plus).

[0040] The calculation method is as follows: input the risk probability, diameter, and tumor type into the comprehensive risk regression model for regression prediction to obtain the comprehensive risk probability.

[0041] The comprehensive risk regression model can be implemented using a logistic model. The construction method is as follows: the probability of imaging risk, diameter, and tumor type are used as independent variables, and whether VPI invasion occurs is used as the dependent variable. The comprehensive risk probability of whether VPI occurs is output as the analysis result.

[0042] Optionally, it also includes a prognostic analysis module; wherein the prognostic analysis template includes classifying the overall risk probability into "high-risk" and "low-risk" groups and assessing the prognostic value.

[0043] Optionally, in the prognostic analysis module, the maximum value of the training set's reduced index is used as the cutoff value to divide the comprehensive risk probability into "high-risk" and "low-risk" groups. The Kaplan-Meier curve (KM curve) and log-rank test are used to evaluate the prognostic differences between the two groups under different surgical plans for segmentectomy and lobectomy, respectively, providing objective quantitative indicators for the formulation of surgical plans.

[0044] Another inventive point of this application is to provide an intraoperative image analysis method, comprising: (1) acquiring a video, wherein the video is a video image during surgery; (2) screening and labeling the video image to obtain multiple images containing lesions; (3) inputting the multiple images containing lesions into a residual neural network model based on random inactivation of spatial regions to obtain multiple image risk probabilities; (4) aggregating the multiple image risk probabilities to obtain a preliminary risk probability; and integrating models trained based on different hyperparameters as different experts through a hybrid expert system integration module to obtain the image risk probability; and (5) performing multivariate linear regression on the image risk probability and clinical information to obtain a comprehensive risk probability.

[0045] By using the comprehensive risk probability (VPI risk), it can be used for patient prognosis prediction and surgical procedure planning. Another inventive aspect of this application is to provide an application of the intraoperative image analysis system described above in a surgical navigation device.

[0046] Compared with the prior art, this application has the following advantages:

[0047] (1) The analysis system of this application is based on intraoperative video, accurately extracts images containing lesions, obtains image risk probability based on residual neural network with random area spatial discarding, fits and integrates the obtained image risk probability, and then combines it with other clinical information to make the prediction results more accurate. The detection sensitivity and specificity are both high, and the prediction result AUC (Area under Curve) is as low as 0.85, which is much higher than the intraoperative assessment of thoracic surgeons (AUC = 0.45-0.54) and the preoperative assessment of radiologists based on CT images (AUC = 0.59-0.65).

[0048] (2) The final prediction results of this application can serve as an independent prognostic predictor, providing guidance for patients’ relapse-free survival, overall survival, and time to disease progression.

[0049] (3) The analysis system of this application is easy to use, has a fast prediction speed, and can provide prediction results in real time during the operation, which facilitates surgical decision-making. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating an analysis method provided in an embodiment of this application.

[0051] Figure 2 This is a flowchart illustrating a residual neural network model provided in an embodiment of this application.

[0052] Figure 3 This is a flowchart illustrating a comprehensive risk prediction module provided in an embodiment of this application; wherein, VPI-Net represents the image risk probability; and VPI-Net-Plus represents the comprehensive risk probability.

[0053] Figure 4 This is a flowchart illustrating a video acquisition module provided in one embodiment of this application.

[0054] Figure 5 The following is a comparison of the prediction results of the analysis method provided in an embodiment of this application: (a) shows the probability distribution of scores under different VPI states; (b) shows the ROC curves of the AUC values ​​corresponding to the VPI-Net model, VPI-Net-plus model, two attending thoracic surgeons, and preoperative estimates in the training set, validation set, and test set, respectively; (c) shows the Kaplan-Meier curves of overall survival, recurrence-free survival, and time to disease progression for all patients, respectively; the horizontal axis represents postoperative time in months; the vertical axis represents the probability of overall survival, recurrence-free survival, and time to disease progression, respectively, without units.

[0055] Figure 6The following are comparative images of the prediction results of the analysis method provided in an embodiment of this application: (a) are preoperative CT images, intraoperative video screenshots, VPI-Net gradient-weighted class activation maps (Grad-CAM), and pathological elastic fiber staining images of patients with lung tumors with a maximum diameter >2cm, respectively; (b) are preoperative CT images, intraoperative video screenshots, VPI-Net gradient-weighted class activation maps (Grad-CAM), and pathological elastic fiber staining images of patients with lung tumors with a maximum diameter <2cm, respectively.

[0056] Figure 7 The following are comparative graphs of prediction results of the analysis method provided in an embodiment of this application: (a) ROC curves of the VPI-Net model, VPI-Net-plus model, two attending thoracic surgeons, and preoperative estimates of the corresponding AUC values ​​in the training set, validation set, and test set for patients with small tumors (maximum diameter of lung tumor <2cm); (b) Kaplan-Meier curves of overall survival, recurrence-free survival, and disease progression time for all patients with small tumors (maximum diameter of lung tumor <2cm) in different VPI-Net-Plus groups; the horizontal axis represents postoperative time in months; the vertical axis represents the probability of overall survival, recurrence-free survival, and disease progression time, respectively, without units; (c) The following graphs show the prediction results of patients with small tumors (maximum diameter of lung tumor <2cm) in different VPI-Net-Plus groups. Kaplan-Meier curves of disease progression time under different surgical procedures (sublobar or lobectomy) in group s; the horizontal axis represents postoperative time in months; the vertical axis represents the probability of disease progression time, without units; (d) Kaplan-Meier curves of disease progression time in patients with small tumors (maximum diameter of lung tumor <2cm) undergoing sublobar resection under different VPI-Net-Plus groups and different tumor types (solid or solid lung); the horizontal axis represents postoperative time in months; the vertical axis represents the probability of disease progression time, without units; (e) Schematic diagram of intraoperative surgical procedure auxiliary decision based on model results, lobectomy is recommended for the VPI-Net-Plus high-risk group, and sublobar resection is recommended for the VPI-Net-Plus low-risk group. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, a more detailed description is provided below. However, it should be understood that the description herein is merely for explaining this application and is not intended to limit its scope.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. All reagents and instruments used herein are commercially available, and the characterization methods involved can be found in relevant descriptions in the prior art, and will not be repeated here.

[0059] To further understand this application, the following detailed description is provided in conjunction with the preferred embodiments.

[0060] Example 1

[0061] This embodiment provides an intraoperative image analysis system, including: a video acquisition module, a preprocessing module, an image risk prediction module, and a comprehensive risk prediction module.

[0062] Video acquisition module: The video is video footage during surgery; such as... Figure 4 As shown, the surgeon inserts a small camera through an incision in the chest wall to obtain a view of the inside of the chest cavity and record video of the operation.

[0063] Next is the preprocessing module: the videos obtained above are manually screened to remove video frames that obscure the tumor during surgery. These video frames cannot be used for subsequent training and will cause bias in the analysis results, so they are removed.

[0064] Subsequently, in the image of the lesion that is not obscured, a rectangular bounding box is used to delineate the lesion and the surrounding area as the region of interest. The standard size of the rectangular bounding box is that the region of interest contains all tumors but does not contain surgical instruments, etc. Then, the region of interest is extracted and resampled to 128×128 using bilinear interpolation, which is then used as the input image for the neural network.

[0065] Risk prediction module: Input multiple images containing lesions into a residual neural network model (SResnet) based on spatial region random inactivation to obtain the risk probabilities of multiple images.

[0066] Specifically, the random deactivation of spatial regions involves generating the center point of the region to be deactivated using a Bernoulli distribution with parameter γ. Then, this center point is used to expand a block area of ​​size block_size to generate the final region to be deactivated. The formula for calculating γ is as follows:

[0067]

[0068] Where L is the side length of the input image, p dr p represents the inactivation probability, and block_size represents the side length of the inactivation region. The side length block_size and inactivation probability p are determined through grid search and cross-validation.dr Optimization: in (0, 1) and Within a certain range, pdr (with numerical intervals of 0.05) and block_size (with numerical intervals of 2) were assigned values. For the selected set of hyperparameters, the training samples were divided into 5 subsamples. One subsample was reserved as data for model validation, and the other 4 samples were used for training. Cross-validation was repeated 5 times, validating each subsample once. The average of the 5 results was used as the accuracy for that parameter, and the parameter set with the highest accuracy was selected for subsequent model training.

[0069] The residual neural network model first uses 32 large 7×7 convolutional kernels to map the image from 3D to 32D, enabling subsequent residual blocks to extract features better. Then, average pooling with a stride of 2 is used to reduce the image size from 128 to 64. Next, five modules consisting of residual blocks are used, each module using 2, 2, 2, 1, and 1 residual blocks with the same number of output channels, respectively. Each residual block reduces the number of image channels using a 1×1 convolutional kernel, and after a 3×3 convolution, the number of channels is increased again using a 1×1 convolutional kernel.

[0070] The features encoded by the residual neural network are then used for risk prediction through global average pooling and soft max to obtain the image risk probability p. imgrisk To mitigate the impact of data imbalance, focal loss is used as the loss function during model training. When the focal loss converges and reaches its minimum, the network parameters for that round are saved, resulting in a well-trained network model.

[0071] Then, the risk probabilities of multiple images are fitted using the maximum likelihood estimation method to obtain a Gaussian distribution with mean μ and variance σ; then, the risk probabilities within (μ+σ, μ-σ) are averaged to obtain the image risk probability p. prisk The specific formula is as follows.

[0072]

[0073] Where sgn is the sign function, p imgrisk Let be the image risk probability, and i be the image number containing the lesion. In the hybrid expert system integration module, different models are trained as experts for different sizes of inactivation regions, and integrated through a gating model to obtain the image risk probability;

[0074]

[0075] Where K is the number of experts, 1≤K≤5, and K=3 is preferred; S i For the weights to be trained, 0 i <1; E i ​(x) represents the initial risk probability of the model under different hyperparameters.

[0076] The comprehensive risk prediction module is used to calculate and output the comprehensive risk probability based on the image risk probability, diameter, and tumor type. The calculation method is as follows: input the risk probability, diameter, and tumor type into the comprehensive risk regression model for regression prediction to obtain the comprehensive risk probability.

[0077] The comprehensive risk regression model can be implemented using a logistic model. The construction method is as follows: the probability of imaging risk, diameter, and tumor type are used as independent variables, and whether VPI invasion occurs is used as the dependent variable. These are input into the logistic model, trained to obtain the comprehensive risk regression model, and the comprehensive risk probability of whether VPI occurs is output as the analysis result.

[0078] Example 2

[0079] This embodiment provides an intraoperative image analysis method, including: (1) acquiring a video, wherein the video is a video image during surgery; (2) filtering and labeling the video image to obtain multiple images containing lesions; (3) inputting the multiple images containing lesions into a residual neural network model based on random inactivation of spatial regions to obtain multiple image risk probabilities; (4) then aggregating the multiple image risk probabilities to obtain the image risk probability; (5) performing multivariate logistic regression on the image risk probability and clinical information to obtain the comprehensive risk probability.

[0080] Surgical videos of 346 patients with pleural traction shown on CT images collected from the hospital were divided into training (241 cases), validation (70 cases), and test set (35 cases) in a 7:2:1 ratio. Image analysis was performed according to the above method.

[0081] The results are as follows Figure 5As shown in Figure (a), the probability distributions of the training set, validation set, and test set for the presence of VPI (VPI group) and the absence of VPI (Non-VPI group) are displayed respectively. It can be seen that the training set, validation set, and test set all show a clear distinction between the two groups, and can quickly identify the presence or absence of VPI, indicating that the analysis system and analysis method of this application have good prediction results. (b) The figure shows the ROC curves of AUC (Area Under Curve) for the training set, validation set, and test set. It can be seen from the figure that VPI-Net (image risk probability) and VPI-Net-Plus (comprehensive risk probability) in this application are both above 0.84, which is much higher than other methods, and has high accuracy. (c) Postoperative time curves of overall survival, recurrence-free survival and time to disease progression for VPI-Net-PlusL (SL, low overall risk probability, VPI-Net-Plus < 0.494) and VPI-Net-PlusH (SH, high overall risk probability, VPI-Net-Plus ≥ 0.494) are shown. It can be seen that there is a significant difference between high and low values ​​of this overall risk probability, indicating that this parameter is closely related to surgical prognosis. Figure 6 Four specific cases are presented, representing two scenarios: lung tumors with a maximum diameter > 2 cm and those with a maximum diameter < 2 cm. The data includes preoperative CT images of the patient's lesion, intraoperative video screenshots, VPI-Net gradient-weighted class activation maps (Grad-CAM), and pathological elastic fiber staining images. In the Grad-CAM image, the colored areas represent the feature distribution regions extracted by the model, with features in red areas having higher weights and features in green areas having lower weights. In the pathological elastic fiber staining image (Pathologic), red arrows indicate tumor tissue infiltration of the pleura, blue arrows indicate areas of pleural discontinuity interruption, and green arrows indicate continuous and intact elastic fibers. Figure (a) shows the cases with and without VPI (Pathologic: +). In cases where VPI (Pathologic: +) is present, the Grad-CAM image obtained based on the prediction results of this application shows a comprehensive risk probability of 0.94, consistent with the results of the pathological elastic fiber staining image. However, the CT images (Radiologist: -; i.e., the radiologist's judgment is that there is no VPI) and the intraoperative video screenshots (Surgeon A: +; i.e., thoracic surgeon A believes that VPI is present; Surgeon B: -; i.e., thoracic surgeon B believes that VPI is not present) differ significantly from the actual results and are greatly influenced by the doctor's subjective judgment and experience. Therefore, they cannot provide an objective conclusion and the results are not very reliable.

[0082] In the absence of VPI (Pathologic: -), similar results as described above also occurred. The results predicted in this application, as shown in the Grad-CAM diagram, indicate a comprehensive risk probability of 0.22, which is consistent with the results of the pathological elastic fiber staining diagram, providing a relatively objective and intuitive conclusion.

[0083] Figure (b) similarly illustrates the presence and absence of VPI (Pathologic: +). Since Figure (b) focuses on lung tumors with a maximum diameter <2 cm, the reduced size makes visual observation of the images more stringent, easily leading to erroneous conclusions. CT images and intraoperative video screenshots can completely contradict the actual results. However, the predictions obtained in this application, as shown in the Grad-CAM image, are completely consistent with the pathological elastic fiber staining results, further demonstrating the higher accuracy of the system and method presented in this application.

[0084] Since the above results show that the analysis system and method of this application have better predictive results for small tumors (lung tumors with a maximum diameter <2cm), training, validation, and testing were conducted on patients with small tumors. Surgical videos of 169 patients with CT images showing pleural traction collected from the hospital were divided into training (109 cases), validation (44 cases), and test sets (16 cases). Image analysis was performed according to the above method. Figure 7 As shown in Figure (a), the ROC curves of AUC (Area Under Curve) for the training set, validation set, and test set are displayed. It can be seen from the figure that VPI-Net (image risk probability) and VPI-Net-Plus (comprehensive risk probability) in this application are both above 0.75, and can reach as high as 0.91, which is much higher than other methods, indicating that the analysis method of this application still has high accuracy in risk prediction of small tumors.

[0085] (b) Postoperative time curves of overall survival, recurrence-free survival and time to disease progression are shown for VPI-Net-PlusL (SL, low overall risk probability, VPI-Net-Plus < 0.494) and VPI-Net-PlusH (SH, high overall risk probability, VPI-Net-Plus ≥ 0.494). It can be seen that there is a significant difference between high and low values ​​of this overall risk probability, indicating that this parameter is closely related to surgical prognosis.

[0086] (c) Postoperative time curves of disease progression for patients undergoing lobectomy with VPI-Net-PlusL (SL, low overall risk probability, VPI-Net-Plus < 0.494), VPI-Net-PlusH (SH, high overall risk probability, VPI-Net-Plus ≥ 0.494), VPI-Net-PlusL (SL, low overall risk probability, VPI-Net-Plus < 0.494), and VPI-Net-PlusH (SH, high overall risk probability, VPI-Net-Plus ≥ 0.494) are shown. It can be seen that the high and low values ​​of this overall risk probability have significant differences when different surgical procedures are performed, indicating that this parameter can be used to assist in the selection of surgical procedures.

[0087] (d) Postoperative time curves of disease progression after sublobar resection are shown for patients with non-solid nodules and VPI-Net-PlusL (SL, low overall risk probability, VPI-Net-Plus < 0.494), non-solid nodules and VPI-Net-PlusH (SH, high overall risk probability, VPI-Net-Plus ≥ 0.494), solid nodules and VPI-Net-PlusL (SL, low overall risk probability, VPI-Net-Plus < 0.494), and solid nodules and VPI-Net-PlusH (SH, high overall risk probability, VPI-Net-Plus ≥ 0.494). It can be seen that the high and low values ​​of this overall risk probability, combined with the solidity of the nodule, have a significant difference when undergoing sublobar resection, indicating that this parameter can be used in conjunction with the solidity of the nodule to assist in the selection of surgical procedures. (e) A schematic diagram illustrating the intraoperative surgical procedure assistance decision based on the above conclusions is shown. Lobectomy is recommended for the VPI-Net-Plus high-risk group, and sublobar resection is recommended for the VPI-Net-Plus low-risk group. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and related explanations of the methods described above can be referred to the corresponding processes in the foregoing system embodiments, and will not be repeated here.

[0088] The term "comprising" or any other similar term is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus / device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent in such process, method, article, or apparatus / device.

[0089] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An intraoperative image analysis system, characterized in that, include: The system includes a video acquisition module, a preprocessing module, a risk prediction module, a hybrid expert system integration module, and a comprehensive risk prediction module. The video acquisition module, wherein the video is video footage of the surgery; The preprocessing module filters and labels the video images to obtain multiple images containing lesions; The risk prediction module inputs multiple images containing lesions into a residual neural network model based on random inactivation of spatial regions to obtain the risk probabilities of multiple images. The risk probabilities of multiple images are then aggregated to obtain a preliminary risk probability; The hybrid expert system integration module divides the risk prediction task into multiple sub-tasks, trains different models as experts for different sizes of inactivation areas, and integrates them through gating models to obtain the image risk probability. The comprehensive risk prediction module calculates and outputs the comprehensive risk probability based on the patient's imaging risk probability value and clinical information. The calculation method is as follows: input the risk probability and clinical information into the comprehensive risk regression model for regression prediction to obtain the comprehensive risk probability; In the risk prediction module, the random deactivation of spatial regions includes: deactivating regions with side length block_size in the input image with side length L using p dr The probability of random inactivation; The center point of the region to be deactivated is generated using a Bernoulli distribution with parameter γ. Then, a block region of size block_size is expanded from this center point to generate the final region to be deactivated. The formula for calculating γ is as follows: ； Where L is the side length of the input image, L≥9; p dr The probability of inactivation is 0. <p dr <1; block_size is the side length of the deactivated region, 3≤block_size≤L / 3; The residual neural network model includes: convolutional layers, average pooling layers, multiple residual stacked modules, a global average pooling layer, and at least one fully connected layer; then, the image risk probability is obtained through Softmax; the residual modules sequentially include: 1x1 convolutional layers, 3x3 convolutional layers, and 1x1 convolutional layers, and the input of the residual module is skipped between the second 1x1 convolutional layer; the residual stacked module includes 5 residual modules; according to the propagation direction, the number of the 5 residual modules are 2, 2, 2, 1, and 1 respectively; the loss function used is binary cross-entropy loss; In the risk prediction module, the risk probabilities of multiple images are aggregated to conform to a Gaussian distribution with a mean of μ and a standard deviation of σ. Then, the risk probabilities within (μ-σ, μ+σ) are averaged to obtain the preliminary risk probability, as shown in the following formula: ； Where sgn is the sign function, p imgrisk Let i represent the image risk probability, and i be the image number containing the lesion. In the hybrid expert system integration module, different models are trained as experts for different sizes of inactive regions, and integrated through gating models to obtain the image risk probability. ； Where K is the number of experts; S ik For the weights to be trained, 0 i <1; E ik (x) represents the initial risk probability of the model in each of the K iterations.​ 2. The intraoperative image analysis system according to claim 1, characterized in that, In the preprocessing module, the screening includes: decomposing the video image frame by frame to obtain multiple images; after analysis, screening video frames in which the lesion area is not obscured, thereby obtaining multiple images to be analyzed; The labeling process includes: outlining the lesion and surrounding area as a region of interest in multiple images to be analyzed. This region of interest contains all tumors but excludes surgical instruments. The region of interest is then extracted to obtain multiple images containing the lesion.

3. The intraoperative image analysis system according to claim 1, characterized in that, The comprehensive risk regression model can be implemented using a logistic model. The construction method is as follows: the probability of imaging risk and clinical information are used as independent variables, and whether VPI invasion occurs is used as the dependent variable. These are input into the logistic model to train and obtain the comprehensive risk regression model.

4. The intraoperative image analysis system according to claim 1, characterized in that, It also includes a prognostic analysis module; the prognostic analysis module includes classifying the overall risk probability into "high risk" and "low risk" groups and assessing the prognostic value.

5. An intraoperative image analysis method for an intraoperative image analysis system according to any one of claims 1 to 4, characterized in that, include: (1) Acquire video, wherein the video is video footage of the operation; (2) The video images were filtered and labeled to obtain multiple images containing lesions; (3) Input multiple images containing lesions into a residual neural network model based on random inactivation of spatial regions to obtain multiple image risk probabilities; (4) Then, the risk probabilities of multiple images are aggregated to obtain the preliminary risk probability; the models trained based on different hyperparameters are used as different experts and integrated through the hybrid expert system integration module to obtain the image risk probability; (5) Perform multivariate logistic regression on the imaging risk probability and clinical information to obtain the comprehensive risk probability.

6. The intraoperative image analysis method according to claim 5, characterized in that, Also includes: (6) Divide the overall risk probability into "high risk" and "low risk" groups to assess the prognostic differences under different surgical procedures.

7. The application of the intraoperative image analysis system according to any one of claims 1 to 4 in surgical navigation equipment.

Images