An intelligent auxiliary processing system for abdominal surgery images
By employing techniques such as Gaussian filtering, histogram equalization, and color space conversion, combined with 3D convolutional neural networks and gradient boosting machine models, the problems of inaccurate lesion identification and resource waste in the laparoscopic surgery image-assisted processing system were solved, enabling individualized treatment and resource optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- FIRST HOSPITAL OF SHANXI MEDICAL UNIV
- Filing Date
- 2024-12-04
- Publication Date
- 2026-05-29
AI Technical Summary
Existing image-assisted processing systems for laparoscopic surgery suffer from inaccurate lesion extraction, noise affecting image quality, ineffective extraction of texture and shape features, inaccurate color perception, failure to adjust filtering effects based on image content, and a lack of individualized treatment plans, resulting in poor treatment outcomes and wasted resources.
The data acquisition module acquires laparoscopic 3D optical molecular images and RAL imaging data. Through Gaussian filtering, histogram equalization, region growing, RGB to LAB color space conversion, and texture and morphological feature extraction, a 3D convolutional neural network model is constructed for lesion diagnosis. Combined with a gradient booster model, the severity of the lesion is predicted, and personalized treatment plans are provided.
It improves the accuracy of lesion area identification and image quality, reduces the impact of noise, provides individualized treatment plans, rationally allocates medical resources, and reduces treatment costs.
Smart Images
Figure CN119679525B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically, to an intelligent auxiliary processing system for laparoscopic surgical images. Background Technology
[0002] Patent publication number CN112932663A discloses an intelligent auxiliary method and system for improving the safety of laparoscopic cholecystectomy. The method includes inputting real-time acquired images through a laparoscope into an artificial intelligence computer model for processing to obtain quantitative data on surgical elements; inputting these surgical elements into a safe surgical model for analysis, and outputting real-time anatomical structure markings and real-time CVS scores to guide the reduction of iatrogenic bile duct injury; and the surgeon performing the surgical operation based on the instructions from the real-time anatomical structure markings and CVS scores. This invention, when used in conjunction with currently available laparoscopic machines, provides intelligent assessment and feedback on important intraoperative anatomical information and CVS completion status. In teaching hospitals, it can help resident physicians and specialists establish awareness of safe intraoperative procedures and cultivate standardized surgical skills. In primary hospitals or medically underdeveloped areas, it can also facilitate the transfer of expert techniques, assisting local doctors in performing surgeries safely and efficiently.
[0003] Existing image-assisted processing systems for laparoscopic surgery have the following problems:
[0004] Failure to use region growing may result in inaccurate extraction of lesion areas, affecting lesion identification; the presence of a large amount of random noise in the image may affect the visual quality and subsequent analysis; failure to effectively extract texture and shape features may lead to inaccurate identification and analysis of lesion characteristics; failure to convert the RGB color space to the LAB color space may lead to inaccurate color perception of lesion areas, affecting lesion identification; failure to automatically adjust the filter window radius according to image content and noise level may result in poor filtering effect, either excessive blurring or incomplete noise removal.
[0005] Administering the same treatment plan to all patients without considering individual differences may lead to poor treatment outcomes; low-risk patients may receive overtreatment, increasing unnecessary medical burdens and side effects; treatment plans not tailored to each patient's specific condition may result in low treatment efficiency or even ineffectiveness; irrational use of medical resources may lead to unnecessary expenses and increase overall treatment costs; and the lack of treatment plans for different risk levels may result in insufficient data to drive medical research and the development of new technologies.
[0006] In view of this, the present invention proposes an intelligent auxiliary processing system for laparoscopic surgical images to solve the above problems. Summary of the Invention
[0007] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: an intelligent auxiliary processing system for laparoscopic surgical images, comprising:
[0008] The data acquisition module is used to acquire laparoscopic 3D optical molecular imaging data, RAL imaging data, and equipment operation data;
[0009] The data processing module is used to preprocess the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain 3D image feature dataset, RAL imaging feature dataset and equipment feature dataset; to perform three-dimensional reconstruction on the 3D image feature dataset and equipment feature dataset and to fuse it with the RAL imaging feature dataset to obtain a comprehensive feature dataset.
[0010] The intraoperative lesion diagnosis module is used to construct an intraoperative lesion diagnosis model. The comprehensive feature dataset is input into the intraoperative lesion diagnosis model to predict optical feature indicators.
[0011] The intraoperative lesion analysis module compares the predicted optical feature indicators with the preset optical feature indicator thresholds to determine whether a lesion exists. If no lesion is found, the diagnosis result is displayed as normal; if a lesion is found, the diagnosis result is displayed as abnormal, and information on the abnormal area of the optical feature indicators is output.
[0012] The clinical decision module is used to build a lesion severity prediction model. It inputs the abnormal area information of the acquired optical feature indicators into the lesion severity prediction model to predict the severity of the lesion.
[0013] The treatment plan control module is used to implement corresponding tumor treatment plans based on the severity of the lesions; the modules are connected to each other via wired and / or wireless means.
[0014] Furthermore, the laparoscopic 3D optical molecular imaging data includes optical image data, fluorescence signal data, optical property data, and time series data; the RAL imaging data includes reflected light signal data, two-dimensional image data, RAL optical property data, texture data, and contrast data; and the device operation data includes the movement trajectory, movement speed, and movement time of the surgical robotic arm.
[0015] Furthermore, the method for preprocessing the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data, and device operation data to obtain the 3D image feature dataset, RAL imaging feature dataset, and device feature dataset includes:
[0016] S31. Convert RAL imaging system data into a standard image format;
[0017] S32. Gaussian filtering is used to reduce noise in the laparoscopic 3D optical molecular image data and the RAL imaging system data converted to standard image format; the Gaussian filtering formula is:
[0018] Where I′(x,y) is the pixel value of the output image at position (x,y); I(x,y) is the pixel value of the input image at position (x,y); x is the horizontal coordinate of the image; y is the vertical coordinate of the image; i and j are relative coordinates; k is the radius of the filter window; G(i,j) is the weight value of the filter kernel at the relative position (i,j);
[0019] The filter window radius is adjusted using a window radius adjustment formula, which is: Where U is the width of the image; W is the height of the image; and σ is the standard deviation of the filter.
[0020] S33. Apply histogram equalization to improve image contrast; the transformation function for histogram equalization is: Among them, S k The new gray level mapped to the original gray level k after histogram equalization; k is the original gray level; n j′ is the number of pixels at gray level j′ in the image; j′ is the index of the gray level to be summed; n is the total number of pixels;
[0021] S34. For the histogram-equalized laparoscopic 3D optical molecular image data, the region growing method is used to segment the ROI region, extract the texture and morphological features of the image, and construct a 3D image feature dataset.
[0022] Extracting texture features from an image:
[0023] The values of the red, green and blue channels of each pixel are extracted from the laparoscopic 3D optical molecular image data to form the color values of the RGB color space. The three color channels are mixed in different proportions, and the values of the three color channels are integers between 0 and 255. The values of the three color channels are obtained from the laparoscopic 3D optical molecular image data and combined together to form the RGB color space. The RGB color space is then converted into the LAB color space.
[0024] Let each pixel in the image be designated as the center pixel, and let the brightness values of each pixel in the L, A, and B channels be L, A, and B, respectively. c A c and B c The position coordinates are (x c ,y c );
[0025] Select M equally spaced neighboring pixels around the center pixel. These neighboring pixels lie on a circle with radius r centered at the center pixel. The neighboring pixels are denoted as (x...). d ,y d ), where is the coordinate of the d-th neighboring pixel;
[0026] For each neighboring pixel, compare its brightness value with that of the center pixel; the comparison formula is: Among them, L d C represents the brightness value of a neighboring pixel. d For comparison results;
[0027] The comparison result C of each neighboring pixel d Combine them into a binary number and calculate the LT value;
[0028] The calculation formula is: Where LT is a digital encoding of the texture information of the center pixel and its neighboring pixels; H is the number of points in the neighborhood; 2 g-1 To convert the comparison result of each neighboring pixel into a binary number; g represents each neighboring pixel;
[0029] For all pixels in the image, calculate the texture features of each pixel in the L, A, and B channels, and count the frequency of each feature value to construct a feature histogram. Then, integrate the feature histograms of the L, A, and B channels by concatenation to form the texture features of the image.
[0030] S35. For the RAL imaging data after histogram equalization, morphological and color features are extracted using a region-based method to construct a RAL imaging feature dataset.
[0031] Extracting color features from an image: The process of converting an RGB color space image to a LAB color space image is as follows: Gamma correction is used to eliminate the nonlinear response of the three color channels, making them linear; a standardized transformation matrix is used to convert the corrected RGB values to XYZ values; a nonlinear processing function is applied to the standardized XYZ values to simulate the perception of the visual system; the standardized and nonlinearly processed XYZ values are used to calculate the LAB color space values.
[0032] Processing of morphological feature data: The image is preprocessed by grayscale conversion and noise reduction, and the processed image is binarized to separate the extracted object and background image regions in the image; the skeleton representation of the extracted object is obtained by skeleton extraction algorithm, and the shape features of the extracted object are extracted based on the skeleton image;
[0033] Abnormal data in laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data were detected and removed by box plot method to obtain processed 3D image feature dataset, RAL imaging feature dataset and equipment feature dataset.
[0034] The 3D image feature dataset, RAL imaging feature dataset, and device feature dataset are normalized and converted into a standard normal distribution to obtain the normalized 3D image feature dataset, RAL imaging feature dataset, and device feature dataset.
[0035] Furthermore, the method for performing 3D reconstruction on the 3D image feature dataset and the device feature dataset, and fusing them with the RAL imaging feature dataset to obtain a comprehensive feature dataset includes:
[0036] Rigid transformation is used to align the 3D image feature dataset with the RAL imaging feature dataset. The rigid transformation method is to use a transformation matrix T to register the 3D image feature data to the RAL imaging feature data space. The specific mathematical formula is: P′=T·P; where P is the coordinate of the 3D image data point; P′ is the coordinate of the registered data point.
[0037] The registered 3D image feature dataset and the RAL imaging feature dataset are fused by weighted average to obtain the fused comprehensive feature dataset.
[0038] The fused feature dataset is voxelized to generate unified 3D structural data; a volume rendering algorithm is used to project light onto the data in the entire voxel space to generate a 3D image; through the above steps, a comprehensive feature dataset is obtained.
[0039] Furthermore, the method for fusing the registered 3D image feature dataset with the RAL imaging feature dataset using a weighted average to obtain the fused comprehensive feature dataset includes:
[0040] The registered 3D image feature dataset and the RAL imaging feature dataset are fused using a weighted average fusion formula to obtain a fused comprehensive feature dataset; the weighted average fusion formula is as follows:
[0041] V fu (a,b,c)=α1·L(a,b,c)+α2·V(a,b,c)+β1·M(a,b,c)+β2·C(a,b,c); where α1 and α2 are the weight coefficients of texture features and shape features, respectively; β1 and β2 are the weight coefficients of morphological features and color features, respectively; M(a,b,c) is the morphological feature; C(a,b,c) is the color feature; V fu(a,b,c) are the feature values after weighted average fusion; L(a,b,c) are the texture features; V(a,b,c) are the shape features.
[0042] Furthermore, the method for constructing an intraoperative lesion diagnostic model includes:
[0043] The dataset is divided into a training set, a validation set, and a test set; the dataset includes a historical comprehensive feature dataset and corresponding optical feature indicators; an intraoperative lesion diagnosis model is constructed, which includes an input layer, a convolutional layer, a pooling layer, a batch normalization layer, a fully connected layer, and an output layer; the input layer is used to input the historical comprehensive feature dataset, and the output layer is used to output the optical feature indicators; the intraoperative lesion diagnosis model is a 3D convolutional neural network model;
[0044] The mean absolute error (MAE) is used as the loss function to measure the error between the model's predicted values and the actual values; the MAE loss function is: Where MAE is the mean absolute error; n is the number of data sets; y i This is the true value of the i-th optical feature index; Let be the predicted value of the i-th optical feature index;
[0045] The model is trained using the training set data, and the loss function is minimized using the Adam optimizer. The model's performance is evaluated using the validation set, and the model's hyperparameters are tuned until the model's performance no longer improves significantly or reaches the preset stopping condition.
[0046] The performance of the model in the prediction task is evaluated using a test set. The current comprehensive feature dataset is input into the trained intraoperative lesion diagnosis model to obtain optical feature indices.
[0047] Furthermore, the method for comparing the predicted optical feature indicators with preset optical feature indicator thresholds to determine whether a lesion exists includes:
[0048] If the predicted optical characteristic index is greater than or equal to the preset optical characteristic index threshold, then a lesion is determined to exist.
[0049] If the predicted optical characteristic index is less than the preset optical characteristic index threshold, then it is determined that there is no lesion.
[0050] Furthermore, the abnormal region information of the optical feature index includes the location coordinates of the abnormal region, the area of the abnormal region, the type of the abnormal region, and the density of the abnormal region.
[0051] Furthermore, the method for constructing the lesion severity prediction model includes:
[0052] The dataset is divided into training, validation and test sets. The dataset includes information on abnormal areas of historical optical feature indicators and the corresponding severity of lesions. Each dataset is a sample of the lesion severity prediction model. GBDT is selected as the specific implementation of the gradient booster model to handle the regression task.
[0053] Initialize GBDT parameters and define the objective function; GBDT parameters include the maximum number of leaf nodes in each tree, learning rate, number of trees, and maximum tree depth; use historical optical feature index abnormal area information as input data and the corresponding lesion severity as output label to train the lesion severity prediction model; the lesion severity prediction model is a gradient boosting machine model;
[0054] Mean squared error is used as the loss function to measure the difference between the model's predicted values and the actual values; the mean squared error loss function is: Where n is the number of samples in the dataset; Y b This represents the actual value of the b-th sample in the dataset. Let be the predicted value of the b-th sample in the dataset; b is the index of the sample.
[0055] In each iteration, GBDT fits the negative gradient of the loss function of the previous step by constructing a new decision tree, uses the negative gradient of the objective function as the learning target of the new tree, and adjusts the contribution of the new tree to the final result by the learning rate to minimize the loss function.
[0056] The model's performance is evaluated using a validation set. The model is then tuned, and its parameters are adjusted based on the performance feedback until the model's performance no longer improves significantly or reaches the preset stopping condition. The trained lesion severity prediction model is then used to predict the abnormal area information of the current optical feature indicators to obtain the severity of the lesion.
[0057] Furthermore, the method for implementing corresponding tumor treatment plans based on the severity of the lesion includes:
[0058] A preset lesion severity threshold is defined, which is then further divided into a low-risk severity threshold T. L The threshold for medium-risk severity, T M and the high-risk severity threshold T H The predicted severity of the lesion is denoted as D; the severity of the lesion D is compared with the low-risk severity threshold T. L The threshold for medium-risk severity, T M and the high-risk severity threshold T H Comparisons are made to determine the level of tumor treatment plan;
[0059] If D≤T LIf T L <D<T M If D > T, then the tumor treatment plan level is determined to be a moderate tumor treatment plan level; H If so, the tumor treatment plan level is determined to be a high-level tumor treatment plan level;
[0060] For low-grade tumor treatment regimens: For patients with low-grade lesions, regimens including local surgical resection, local radiotherapy, or local immunotherapy are implemented, without involving systemic treatment;
[0061] For moderate tumor treatment regimens: For patients with moderate lesion severity, a combined treatment regimen including surgical resection, systemic chemotherapy, systemic radiotherapy, and targeted therapy is implemented;
[0062] For high-level tumor treatment regimens: For patients with high lesion severity, a comprehensive treatment regimen including extensive surgical resection, high-dose chemotherapy, total body radiation therapy, and immunotherapy is implemented.
[0063] The technical effects and advantages of the intelligent auxiliary processing system for laparoscopic surgical images of the present invention are as follows:
[0064] This invention reduces random noise in images and improves image clarity and quality through Gaussian filtering; histogram equalization enhances image contrast, making details more clearly visible; region growing (ROI) segmentation allows for more accurate extraction of lesion-related image features; extracting texture and shape features helps in better identifying and analyzing the characteristics of lesion areas; converting the RGB color space to the LAB color space better reflects human color perception, aiding in lesion identification; specific texture feature calculation methods highlight the unique texture patterns of lesion areas; and the filter window radius automatically adjusts based on image content and filter standard deviation, providing a more suitable filter window size for different images and noise levels.
[0065] Providing customized treatment plans based on the specific severity of a patient's condition can better meet the individual needs of the patient; by differentiating treatment plans according to different risk levels, medical resources can be allocated more effectively, ensuring that high-risk patients receive more intensive treatment, while low-risk patients avoid overtreatment; and the rational allocation of treatment plans can reduce unnecessary medical expenses and lower the overall treatment cost. Attached Figure Description
[0066] Figure 1 This is a schematic diagram of the intelligent auxiliary processing system for abdominal surgical images according to the present invention;
[0067] Figure 2 This is a schematic diagram of the intelligent auxiliary processing method for abdominal surgery images according to the present invention. Detailed Implementation
[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0069] Example 1
[0070] Please see Figure 1 As shown in this embodiment, an intelligent auxiliary processing system for laparoscopic surgical images includes:
[0071] The data acquisition module is used to acquire laparoscopic 3D optical molecular imaging data, RAL imaging data, and equipment operation data;
[0072] The data processing module is used to preprocess the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain 3D image feature dataset, RAL imaging feature dataset and equipment feature dataset; to perform three-dimensional reconstruction on the 3D image feature dataset and equipment feature dataset and to fuse it with the RAL imaging feature dataset to obtain a comprehensive feature dataset.
[0073] The intraoperative lesion diagnosis module is used to construct an intraoperative lesion diagnosis model. The comprehensive feature dataset is input into the intraoperative lesion diagnosis model to predict optical feature indicators.
[0074] The intraoperative lesion analysis module compares the predicted optical feature indicators with the preset optical feature indicator thresholds to determine whether a lesion exists. If no lesion is found, the diagnosis result is displayed as normal; if a lesion is found, the diagnosis result is displayed as abnormal, and information on the abnormal area of the optical feature indicators is output.
[0075] The clinical decision module is used to build a lesion severity prediction model. It inputs the abnormal area information of the acquired optical feature indicators into the lesion severity prediction model to predict the severity of the lesion.
[0076] The treatment plan control module is used to implement corresponding tumor treatment plans based on the severity of the lesions; the modules are connected to each other via wired and / or wireless means.
[0077] Laparoscopic 3D optical molecular imaging data includes optical image data, fluorescence signal data, optical property data, and time series data; RAL imaging data includes reflected light signal data, two-dimensional image data, RAL optical property data, texture data, and contrast data; equipment operation data includes the movement trajectory, movement speed, and movement time of the surgical robotic arm.
[0078] Methods for preprocessing acquired laparoscopic 3D optical molecular imaging data, RAL imaging data, and device operation data to obtain 3D image feature datasets, RAL imaging feature datasets, and device feature datasets include:
[0079] S31. Convert RAL imaging system data into a standard image format;
[0080] S32. Gaussian filtering is used to reduce noise in the laparoscopic 3D optical molecular image data and the RAL imaging system data converted to standard image format; the Gaussian filtering formula is:
[0081] Where I′(x,y) is the pixel value of the output image at position (x,y); I(x,y) is the pixel value of the input image at position (x,y); x is the horizontal coordinate of the image; y is the vertical coordinate of the image; i and j are relative coordinates used to represent the relative offset of the convolution kernel around the current center position (x,y); k is the radius of the filter window; and G(i,j) is the weight value of the filter kernel at the relative position (i,j).
[0082] The filter window radius is adjusted using a window radius adjustment formula, which is: Where U is the width of the image; W is the height of the image; and σ is the standard deviation of the filter.
[0083] For example, if the image width is 8, the image height is 20, and the standard deviation of the filter is 5, then the filter window radius is 8.
[0084] S33. Apply histogram equalization to improve image contrast; the transformation function for histogram equalization is: Among them, S k The new gray level mapped to the original gray level k after histogram equalization; k is the original gray level; n j′ is the number of pixels at gray level j′ in the image; j′ is the index of the gray level to be summed; n is the total number of pixels;
[0085] S34. For the histogram-equalized laparoscopic 3D optical molecular image data, the region growing method is used to segment the ROI region, extract the texture and morphological features of the image, and construct a 3D image feature dataset.
[0086] Extracting texture features from an image:
[0087] The values of the red, green and blue channels of each pixel are extracted from the laparoscopic 3D optical molecular image data to form the color values of the RGB color space. The three color channels are mixed in different proportions, and the values of the three color channels are integers between 0 and 255. The values of the three color channels are obtained from the laparoscopic 3D optical molecular image data and combined together to form the RGB color space. The RGB color space is then converted into the LAB color space.
[0088] Let each pixel in the image be designated as the center pixel, and let the brightness values of each pixel in the L, A, and B channels be L, A, and B, respectively. c A c and B c The position coordinates are (x c ,y c );
[0089] Select M equally spaced neighboring pixels around the center pixel. These neighboring pixels lie on a circle with radius r centered at the center pixel. The neighboring pixels are denoted as (x...). d ,y d ), where is the coordinate of the d-th neighboring pixel;
[0090] For each neighboring pixel, compare its brightness value with that of the center pixel; the comparison formula is: Among them, L d C represents the brightness value of a neighboring pixel. d For comparison results;
[0091] For example, if the brightness value of a neighboring pixel is 100, L c If the brightness value is 80, then the comparison result is C. d The value is 1.
[0092] The comparison result C of each neighboring pixel d Combine them into a binary number and calculate the LT value;
[0093] The calculation formula is: Where LT is a digital encoding of the texture information of the center pixel and its neighboring pixels; H is the number of points in the neighborhood; 2 g-1 To convert the comparison result of each neighboring pixel into a binary number; g represents each neighboring pixel;
[0094] For all pixels in the image, calculate the texture features of each pixel in the L, A, and B channels, and count the frequency of each feature value to construct a feature histogram. Then, integrate the feature histograms of the L, A, and B channels by concatenation to form the texture features of the image.
[0095] S35. For the RAL imaging data after histogram equalization, morphological and color features are extracted using a region-based method to construct a RAL imaging feature dataset.
[0096] Extracting color features from an image: The process of converting an RGB color space image to a LAB color space image is as follows: Gamma correction is used to eliminate the nonlinear response of the three color channels, making them linear; a standardized transformation matrix is used to convert the corrected RGB values to XYZ values; a nonlinear processing function is applied to the standardized XYZ values to simulate the perception of the visual system; the standardized and nonlinearly processed XYZ values are used to calculate the LAB color space values.
[0097] Processing of morphological feature data: The image is preprocessed by grayscale conversion and noise reduction, and the processed image is binarized to separate the extracted object and background image regions in the image; the skeleton representation of the extracted object is obtained by skeleton extraction algorithm, and the shape features of the extracted object are extracted based on the skeleton image;
[0098] Abnormal data in laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data were detected and removed by box plot method to obtain processed 3D image feature dataset, RAL imaging feature dataset and equipment feature dataset.
[0099] The 3D image feature dataset, RAL imaging feature dataset, and device feature dataset are normalized and converted into a standard normal distribution to obtain the normalized 3D image feature dataset, RAL imaging feature dataset, and device feature dataset.
[0100] Methods for performing 3D reconstruction on 3D image feature datasets and device feature datasets, and then fusing them with RAL imaging feature datasets to obtain a comprehensive feature dataset include:
[0101] Rigid transformation is used to align the 3D image feature dataset with the RAL imaging feature dataset. The rigid transformation method is to use a transformation matrix T to register the 3D image feature data to the RAL imaging feature data space. The specific mathematical formula is: P′=T·P; where P is the coordinate of the 3D image data point; P′ is the coordinate of the registered data point.
[0102] The registered 3D image feature dataset and the RAL imaging feature dataset are fused by weighted average to obtain the fused comprehensive feature dataset.
[0103] The fused feature dataset is voxelized to generate unified 3D structural data; a volume rendering algorithm is used to project light onto the data in the entire voxel space to generate a 3D image; through the above steps, a comprehensive feature dataset is obtained.
[0104] Methods for fusing the registered 3D image feature dataset with the RAL imaging feature dataset using a weighted average to obtain the fused comprehensive feature dataset include:
[0105] The registered 3D image feature dataset and the RAL imaging feature dataset are fused using a weighted average fusion formula to obtain a fused comprehensive feature dataset; the weighted average fusion formula is as follows:
[0106] V fu (a,b,c)=α1·L(a,b,c)+α2·V(a,b,c)+β1·M(a,b,c)+β2·C(a,b,c); where α1 and α2 are the weight coefficients of texture features and shape features, respectively; β1 and β2 are the weight coefficients of morphological features and color features, respectively; M(a,b,c) is the morphological feature; C(a,b,c) is the color feature; V fu (a,b,c) are the feature values after weighted average fusion; L(a,b,c) are the texture features; V(a,b,c) are the shape features.
[0107] Methods for constructing intraoperative lesion diagnostic models include:
[0108] The dataset is divided into a training set, a validation set, and a test set; the dataset includes a historical comprehensive feature dataset and corresponding optical feature indicators; an intraoperative lesion diagnosis model is constructed, which includes an input layer, a convolutional layer, a pooling layer, a batch normalization layer, a fully connected layer, and an output layer; the input layer is used to input the historical comprehensive feature dataset, and the output layer is used to output the optical feature indicators; the intraoperative lesion diagnosis model is a 3D convolutional neural network model;
[0109] The mean absolute error (MAE) is used as the loss function to measure the error between the model's predicted values and the actual values; the MAE loss function is: Where MAE is the mean absolute error; n is the number of data sets; y i This is the true value of the i-th optical feature index; Let be the predicted value of the i-th optical feature index;
[0110] The model is trained using the training set data, and the loss function is minimized using the Adam optimizer. The model's performance is evaluated using the validation set, and the model's hyperparameters are tuned until the model's performance no longer improves significantly or reaches the preset stopping condition.
[0111] The performance of the model in the prediction task is evaluated using a test set. The current comprehensive feature dataset is input into the trained intraoperative lesion diagnosis model to obtain optical feature indices.
[0112] Methods for determining the presence of lesions by comparing predicted optical feature indicators with preset optical feature indicator thresholds include:
[0113] If the predicted optical characteristic index is greater than or equal to the preset optical characteristic index threshold, then a lesion is determined to exist.
[0114] If the predicted optical characteristic index is less than the preset optical characteristic index threshold, then it is determined that there is no lesion.
[0115] The information on abnormal regions in optical feature indicators includes the location coordinates of the abnormal region, the area of the abnormal region, the type of the abnormal region, and the density of the abnormal region.
[0116] Methods for constructing lesion severity prediction models include:
[0117] The dataset is divided into training, validation and test sets. The dataset includes information on abnormal areas of historical optical feature indicators and the corresponding severity of lesions. Each dataset is a sample of the lesion severity prediction model. GBDT is selected as the specific implementation of the gradient booster model to handle the regression task.
[0118] Initialize GBDT parameters and define the objective function; GBDT parameters include the maximum number of leaf nodes in each tree, learning rate, number of trees, and maximum tree depth; use historical optical feature index abnormal area information as input data and the corresponding lesion severity as output label to train the lesion severity prediction model; the lesion severity prediction model is a gradient boosting machine model;
[0119] Mean squared error is used as the loss function to measure the difference between the model's predicted values and the actual values; the mean squared error loss function is: Where n is the number of samples in the dataset; Y b This represents the actual value of the b-th sample in the dataset. Let be the predicted value of the b-th sample in the dataset; b is the index of the sample.
[0120] In each iteration, GBDT fits the negative gradient of the loss function of the previous step by constructing a new decision tree, uses the negative gradient of the objective function as the learning target of the new tree, and adjusts the contribution of the new tree to the final result by the learning rate to minimize the loss function.
[0121] The model's performance is evaluated using a validation set. The model is then tuned, and its parameters are adjusted based on the performance feedback until the model's performance no longer improves significantly or reaches the preset stopping condition. The trained lesion severity prediction model is then used to predict the abnormal area information of the current optical feature indicators to obtain the severity of the lesion.
[0122] Methods for implementing appropriate tumor treatment plans based on the severity of the lesion include:
[0123] A preset lesion severity threshold is defined, which is then further divided into a low-risk severity threshold T. L The threshold for medium-risk severity, T M and the high-risk severity threshold T H The predicted severity of the lesion is denoted as D; the severity of the lesion D is compared with the low-risk severity threshold T. L The threshold for medium-risk severity, T M and the high-risk severity threshold T H Comparisons are made to determine the level of tumor treatment plan;
[0124] If D≤T L If T L <D<T M If D > T, then the tumor treatment plan level is determined to be a moderate tumor treatment plan level; H If so, the tumor treatment plan level is determined to be a high-level tumor treatment plan level;
[0125] For low-grade tumor treatment regimens: For patients with low-grade lesions, regimens including local surgical resection, local radiotherapy, or local immunotherapy are implemented, without involving systemic treatment;
[0126] For moderate tumor treatment regimens: For patients with moderate lesion severity, a combined treatment regimen including surgical resection, systemic chemotherapy, systemic radiotherapy, and targeted therapy is implemented;
[0127] For high-level tumor treatment regimens: For patients with high lesion severity, a comprehensive treatment regimen including extensive surgical resection, high-dose chemotherapy, total body radiation therapy, and immunotherapy is implemented.
[0128] For example: Suppose the thresholds are set as follows: low-risk severity threshold: 0.2; medium-risk severity threshold: 0.5; high-risk severity threshold: 0.8; the prediction result shows that the lesion severity is 0.4;
[0129] Therefore, the predicted severity of the lesion is 0.4; comparing the severity of the lesion with the threshold:
[0130] Compared to the low-risk severity threshold (0.2): 0.4 > 0.2
[0131] Compared to the medium-risk severity threshold (0.5): 0.4 < 0.5
[0132] Compared to the high-risk severity threshold (0.8): 0.4 < 0.8
[0133] Since 0.4 is greater than the low-risk severity threshold but less than the medium-risk severity threshold, the tumor treatment regimen is classified as moderate.
[0134] The preset optical feature index threshold is set by the staff. Different optical feature indicators are collected through an intelligent auxiliary processing terminal, and the average value of multiple optical feature indicators is taken as the preset optical feature index threshold.
[0135] In this embodiment, Gaussian filtering denoising reduces random noise in the image, improving image clarity and quality; histogram equalization enhances image contrast, making details more clearly visible; region growing (ROI) segmentation allows for more accurate extraction of lesion-related image features; extracting texture and shape features helps in better identifying and analyzing the characteristics of lesion areas; converting the RGB color space to the LAB color space better reflects human color perception, aiding in lesion identification; specific texture feature calculation methods highlight the unique texture patterns of lesion areas; and the filter window radius automatically adjusts based on image content and filter standard deviation, providing a more suitable filter window size for different images and noise levels.
[0136] Providing customized treatment plans based on the specific severity of a patient's condition can better meet the individual needs of the patient; by differentiating treatment plans according to different risk levels, medical resources can be allocated more effectively, ensuring that high-risk patients receive more intensive treatment, while low-risk patients avoid overtreatment; and the rational allocation of treatment plans can reduce unnecessary medical expenses and lower the overall treatment cost.
[0137] Example 2
[0138] Please see Figure 2 As shown, for parts not described in detail in this embodiment, please refer to the description in Embodiment 1. A method for intelligent auxiliary processing of laparoscopic surgical images is provided, including:
[0139] S1. Acquire laparoscopic 3D optical molecular imaging data, RAL imaging data, and equipment operation data;
[0140] S2. Preprocess the acquired laparoscopic 3D optical molecular image data, RAL imaging data, and equipment operation data to obtain a 3D image feature dataset, a RAL imaging feature dataset, and an equipment feature dataset; perform 3D reconstruction on the 3D image feature dataset and the equipment feature dataset, and fuse them with the RAL imaging feature dataset to obtain a comprehensive feature dataset.
[0141] S3. Construct an intraoperative lesion diagnosis model, input the comprehensive feature dataset into the intraoperative lesion diagnosis model, and predict the optical feature indicators;
[0142] S4. Compare the predicted optical feature indicators with the preset optical feature indicator thresholds to determine whether there is a lesion; if no lesion is found, the diagnosis result is displayed as normal; if a lesion is found, the diagnosis result is displayed as abnormal, and the abnormal area information of the optical feature indicators is output.
[0143] S5. Construct a lesion severity prediction model. Input the acquired abnormal area information of optical feature indicators into the lesion severity prediction model to predict the severity of the lesion.
[0144] S6. Implement corresponding tumor treatment plans based on the severity of the lesions.
[0145] Since the electronic device described in this embodiment is the electronic device used in implementing the intelligent auxiliary processing system for laparoscopic surgical images in this application embodiment, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the intelligent auxiliary processing system for laparoscopic surgical images described in this application embodiment. Therefore, how the electronic device implements the method in this application embodiment will not be described in detail here. Any electronic device used by those skilled in the art in implementing the intelligent auxiliary processing system for laparoscopic surgical images in this application embodiment falls within the scope of protection of this application.
[0146] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters and thresholds in the formulas are set by those skilled in the art according to the actual situation.
[0147] The above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for users of ordinary technical skills, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.
Claims
1. An intelligent auxiliary processing system for laparoscopic surgical images, characterized in that, include: The data acquisition module is used to acquire laparoscopic 3D optical molecular imaging data, RAL imaging data, and equipment operation data; The data processing module is used to preprocess the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain 3D image feature dataset, RAL imaging feature dataset and equipment feature dataset; to perform three-dimensional reconstruction on the 3D image feature dataset and equipment feature dataset and to fuse it with the RAL imaging feature dataset to obtain a comprehensive feature dataset. The method for preprocessing the acquired laparoscopic 3D optical molecular image data, RAL imaging data, and equipment operation data to obtain the 3D image feature dataset, RAL imaging feature dataset, and equipment feature dataset includes: S31. Convert RAL imaging system data into a standard image format; S32. Gaussian filtering is used to reduce noise in the laparoscopic 3D optical molecular image data and the RAL imaging system data converted to standard image format; the Gaussian filtering formula is: ;in, To output the image at position Pixel values; For the input image at position Pixel values; The x-coordinate of the image; The vertical coordinate of the image; and Relative coordinates; The radius of the filter window; For the filter kernel in relative position The weight value; The filter window radius is adjusted using a window radius adjustment formula, which is: ;in, The width of the image; The height of the image; The standard deviation of the filter; S33. Apply histogram equalization to improve image contrast; the transformation function for histogram equalization is: ;in, Original grayscale level The new gray level mapped after histogram equalization; Original grayscale level; For the first in the image The number of pixels at each gray level; The index for summing gray levels; Total number of pixels; S34. For the histogram-equalized laparoscopic 3D optical molecular image data, the region growing method is used to segment the ROI region, extract the texture and shape features of the image, and construct a 3D image feature dataset. Extracting texture features from an image: The values of the red, green and blue channels of each pixel are extracted from the laparoscopic 3D optical molecular image data to form the color values of the RGB color space. The three color channels are mixed in different proportions, and the values of the three color channels are integers between 0 and 255. The values of the three color channels are obtained from the laparoscopic 3D optical molecular image data and combined together to form the RGB color space. The RGB color space is then converted into the LAB color space. Let each pixel in the image be designated as the center pixel, and let the brightness values of each pixel in the L, A, and B channels be respectively... , and The position coordinates are ; Select around the center pixel There are 3 equally spaced neighboring pixels, and each neighboring pixel is located in a circle centered at the center pixel with a radius of 1. On the circumference of a circle; neighboring pixels are denoted as , for the first The coordinates of the neighboring pixels; For each neighboring pixel, compare its brightness value with that of the center pixel; the comparison formula is: ;in, The brightness value of the neighboring pixels; For comparison results; The comparison results of each neighboring pixel Combine them into a binary number and calculate. value; The calculation formula is: ;in, A digital encoding of the texture information of the center pixel and its neighboring pixels; The number of points in the neighborhood; To convert the comparison result of each neighboring pixel into a binary number; For each neighboring pixel; For all pixels in the image, calculate the texture features of each pixel in the L, A, and B channels, and count the frequency of each feature value to construct a feature histogram. Then, integrate the feature histograms of the L, A, and B channels by concatenation to form the texture features of the image. S35. For the RAL imaging data after histogram equalization, morphological and color features are extracted using a region-based method to construct a RAL imaging feature dataset. Extracting color features from an image: The process of converting an RGB color space image to a LAB color space image is as follows: Gamma correction is used to eliminate the nonlinear response of the three color channels, making them linear; a standardized transformation matrix is used to convert the corrected RGB values to XYZ values; a nonlinear processing function is applied to the standardized XYZ values to simulate the perception of the visual system; the standardized and nonlinearly processed XYZ values are used to calculate the LAB color space values. Extracting shape and morphological features from the image: The image is preprocessed by grayscale conversion and noise reduction, and the processed image is binarized to separate the extracted object and background image regions in the image; the skeleton representation of the extracted object is obtained through a skeleton extraction algorithm, and the shape and morphological features of the extracted object are extracted based on the skeleton image; Abnormal data in laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data were detected and removed by box plot method to obtain processed 3D image feature dataset, RAL imaging feature dataset and equipment feature dataset. The 3D image feature dataset, RAL imaging feature dataset, and device feature dataset are normalized and converted into a standard normal distribution to obtain the normalized 3D image feature dataset, RAL imaging feature dataset, and device feature dataset. The intraoperative lesion diagnosis module is used to construct an intraoperative lesion diagnosis model. The comprehensive feature dataset is input into the intraoperative lesion diagnosis model to predict optical feature indicators. The intraoperative lesion analysis module compares the predicted optical feature indicators with the preset optical feature indicator thresholds to determine whether a lesion exists. If no lesion is found, the diagnosis result is displayed as normal; if a lesion is found, the diagnosis result is displayed as abnormal, and information on the abnormal area of the optical feature indicators is output. The clinical decision module is used to build a lesion severity prediction model. It inputs the abnormal area information of the acquired optical feature indicators into the lesion severity prediction model to predict the severity of the lesion. The treatment plan control module is used to implement corresponding tumor treatment plans based on the severity of the lesions; the modules are connected to each other via wired and / or wireless means.
2. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 1, characterized in that, The laparoscopic 3D optical molecular imaging data includes optical image data, fluorescence signal data, optical property data, and time series data; the RAL imaging data includes reflected light signal data, two-dimensional image data, RAL optical property data, texture data, and contrast data; and the device operation data includes the movement trajectory, movement speed, and movement time of the surgical robotic arm.
3. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 2, characterized in that, The method for performing 3D reconstruction on 3D image feature datasets and device feature datasets, and fusing them with RAL imaging feature datasets to obtain a comprehensive feature dataset includes: A rigid transformation is used to align the 3D image feature dataset with the RAL imaging feature dataset. The rigid transformation method involves using a transformation matrix T to register the 3D image feature data to the RAL imaging feature data space. The specific mathematical formula is as follows: ;in, The coordinates of the 3D image data points; The coordinates of the registered data points; The registered 3D image feature dataset and the RAL imaging feature dataset are fused by weighted average to obtain the fused comprehensive feature dataset. The fused feature dataset is voxelized to generate unified 3D structural data; a volume rendering algorithm is used to project light onto the data in the entire voxel space to generate a 3D image; through the above steps, a comprehensive feature dataset is obtained.
4. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 3, characterized in that, The method for fusing the registered 3D image feature dataset with the RAL imaging feature dataset by weighted average to obtain the fused comprehensive feature dataset includes: The registered 3D image feature dataset and the RAL imaging feature dataset are fused using a weighted average fusion formula to obtain a fused comprehensive feature dataset; the weighted average fusion formula is as follows: ;in, and These are the weighting coefficients for texture features and shape features, respectively. and These are the weighting coefficients for morphological features and color features, respectively; Morphological characteristics; Color characteristics; These are the eigenvalues after weighted average fusion; Texture features; Shape characteristics.
5. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 4, characterized in that, The method for constructing an intraoperative lesion diagnosis model includes: The dataset is divided into a training set, a validation set, and a test set; the dataset includes a historical comprehensive feature dataset and corresponding optical feature indicators; an intraoperative lesion diagnosis model is constructed, which includes an input layer, a convolutional layer, a pooling layer, a batch normalization layer, a fully connected layer, and an output layer; the input layer is used to input the historical comprehensive feature dataset, and the output layer is used to output the optical feature indicators; the intraoperative lesion diagnosis model is a 3D convolutional neural network model; The mean absolute error (MAE) is used as the loss function to measure the error between the model's predicted values and the actual values; the MAE loss function is: ;in, Mean absolute error; Number of datasets; For the first The true value of each optical characteristic index; For the first Predicted values of several optical characteristic indicators; The model is trained using the training set data, and the loss function is minimized using the Adam optimizer. The model's performance is evaluated using the validation set, and the model's hyperparameters are tuned until the model's performance no longer improves significantly or reaches the preset stopping condition. The performance of the model in the prediction task is evaluated using a test set. The current comprehensive feature dataset is input into the trained intraoperative lesion diagnosis model to obtain optical feature indices.
6. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 5, characterized in that, The method for comparing the predicted optical feature indicators with preset optical feature indicator thresholds to determine whether a lesion exists includes: If the predicted optical characteristic index is greater than or equal to the preset optical characteristic index threshold, then a lesion is determined to exist. If the predicted optical characteristic index is less than the preset optical characteristic index threshold, then it is determined that there is no lesion.
7. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 6, characterized in that, The abnormal region information of the optical feature index includes the location coordinates of the abnormal region, the area of the abnormal region, the type of the abnormal region, and the density of the abnormal region.
8. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 7, characterized in that, The method for constructing the lesion severity prediction model includes: The dataset is divided into training, validation and test sets. The dataset includes information on abnormal areas of historical optical feature indicators and the corresponding severity of lesions. Each dataset is a sample of the lesion severity prediction model. GBDT is selected as the specific implementation of the gradient booster model to handle the regression task. Initialize GBDT parameters and define the objective function; GBDT parameters include the maximum number of leaf nodes in each tree, learning rate, number of trees, and maximum tree depth; use historical optical feature index abnormal area information as input data and the corresponding lesion severity as output label to train the lesion severity prediction model; the lesion severity prediction model is a gradient boosting machine model; Mean squared error is used as the loss function to measure the difference between the model's predicted values and the actual values; the mean squared error loss function is: ;in, This represents the number of samples in the dataset. For the first in the dataset The actual value of each sample; For the first in the dataset Predicted values for each sample; For the index of the sample; In each iteration, GBDT fits the negative gradient of the loss function of the previous step by constructing a new decision tree, uses the negative gradient of the objective function as the learning target of the new tree, and adjusts the contribution of the new tree to the final result by the learning rate to minimize the loss function. The model's performance is evaluated using a validation set. The model is then tuned, and its parameters are adjusted based on the performance feedback until the model's performance no longer improves significantly or reaches the preset stopping condition. The trained lesion severity prediction model is then used to predict the abnormal area information of the current optical feature indicators to obtain the severity of the lesion.
9. The intelligent auxiliary processing system for laparoscopic surgical images according to claim 8, characterized in that, The method for implementing corresponding tumor treatment plans based on the severity of the lesion includes: A preset lesion severity threshold is defined, which is then further subdivided into low-risk severity thresholds. Threshold for medium-risk severity and high-risk severity threshold The predicted severity of the lesion is recorded as... ; to determine the severity of the lesion Respectively compared with the low-risk severity threshold Threshold for medium-risk severity and high-risk severity threshold Comparisons are made to determine the level of tumor treatment plan; like If so, the tumor treatment plan level is determined to be a low-grade tumor treatment plan level; if Then the tumor treatment plan level is determined to be a moderate tumor treatment plan level; if If so, the tumor treatment plan level is determined to be a high-level tumor treatment plan level; For low-grade tumor treatment regimens: For patients with low-grade lesions, regimens including local surgical resection, local radiotherapy, or local immunotherapy are implemented, without involving systemic treatment; For moderate tumor treatment regimens: For patients with moderate lesion severity, a combined treatment regimen including surgical resection, systemic chemotherapy, systemic radiotherapy, and targeted therapy is implemented; For high-level tumor treatment regimens: For patients with high lesion severity, a comprehensive treatment regimen including extensive surgical resection, high-dose chemotherapy, total body radiation therapy, and immunotherapy is implemented.