Intelligent auxiliary processing system for abdominal surgery images

Abstract

The invention belongs to the technical field of image processing, and discloses an intelligent auxiliary processing system for an abdominal surgery image, and the system comprises a data collection module which is used for collecting laparoscope 3D optical molecular image data, RAL imaging data and equipment operation data; the data processing module is used for obtaining a comprehensive feature data set according to the obtained laparoscope 3D optical molecular image data, RAL imaging data and equipment operation data; the intraoperative lesion diagnosis module is used for constructing an intraoperative lesion diagnosis model and predicting to obtain an optical characteristic index; the intraoperative lesion analysis module is used for comparing the predicted optical characteristic index with a preset optical characteristic index threshold value and judging whether a lesion exists or not; the clinical decision module is used for constructing a lesion degree prediction model and predicting the severity of the lesion; and the scheme regulation and control module is used for implementing a corresponding tumor treatment scheme according to the severity of the lesion.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to an intelligent auxiliary processing system for abdominal surgery images. Background Art

[0002] The patent with the patent publication number CN112932663A discloses an intelligent auxiliary method and system for improving the safety of laparoscopic cholecystectomy, including inputting the real-time collected images into the artificial intelligence computer model for processing through the laparoscope to obtain the quantitative data of the surgical elements; inputting the surgical elements into the safe surgical model for analysis, and outputting the real-time marking of the anatomical structure and the real-time scoring of CVS, which are the guiding elements for reducing iatrogenic bile duct injury; the surgeon performs the surgical operation according to the instructions of the real-time marking of the anatomical structure and the real-time scoring of CVS. The present invention is used in combination with the laparoscopic machine currently used on the market to provide intelligent evaluation and feedback of important intraoperative anatomical information and CVS completion status. In teaching hospitals, it can help residents and specialists establish awareness of safe intraoperative operations and cultivate standardized surgical skills; in grassroots hospitals or underdeveloped medical areas, it can also realize the output of expert technology to assist local doctors in completing the operation safely and efficiently.

[0003] The existing laparoscopic surgery image-assisted processing system has the following problems:

[0004] Failure to use the region growing method may lead to inaccurate extraction of the lesion area, affecting the identification of the lesion; there may be a large amount of random noise in the image, affecting the visual effect and subsequent analysis of the image; texture and shape features are not effectively extracted, which may lead to inaccurate identification and analysis of lesion characteristics; failure to convert the RGB color space to the LAB color space may lead to inaccurate color perception of the lesion area, affecting the identification of the lesion; the filter window radius fails to automatically adjust according to the image content and noise level, which may lead to poor filtering effect, either excessive blurring or incomplete noise removal;

[0005] All patients receive similar treatment plans without considering individual differences, which may lead to poor treatment effects; low-risk patients may receive overtreatment, increasing unnecessary medical burdens and side effects; treatment plans that are not tailored to the patient's specific situation may lead to inefficient or even ineffective treatment; irrational use of medical resources may lead to unnecessary expenses and increase overall treatment costs; the lack of treatment plans for different risk levels may lead to a lack of sufficient data to promote medical research and the development of new technologies.

[0006] In view of this, the present invention proposes an intelligent auxiliary processing system for abdominal surgery images to solve the above problems. Summary of the invention

[0007] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: an intelligent auxiliary processing system for abdominal surgery images, comprising:

[0008] Data acquisition module, used to collect laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data;

[0009] A data processing module is used to pre-process the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain a 3D imaging feature data set, a RAL imaging feature data set and an equipment feature data set; perform three-dimensional reconstruction on the 3D imaging feature data set and the equipment feature data set, and fuse them with the RAL imaging feature data set to obtain a comprehensive feature data set;

[0010] An intraoperative lesion diagnosis module is used to construct an intraoperative lesion diagnosis model, input the comprehensive feature data set into the intraoperative lesion diagnosis model, and predict the optical feature indicators;

[0011] The intraoperative lesion analysis module compares the predicted optical characteristic index with the preset optical characteristic index threshold to determine whether there is a lesion; if it is determined that there is no lesion, the diagnosis result is displayed as normal; if it is determined that there is a lesion, the diagnosis result is displayed as abnormal, and the optical characteristic index abnormal area information is output;

[0012] The clinical decision-making module is used to construct a lesion severity prediction model, input the acquired optical characteristic index abnormal area information into the lesion severity prediction model, and predict the severity of the lesion;

[0013] The scheme control module is used to implement the corresponding tumor treatment scheme according to the severity of the lesion; each module is connected by wired and / or wireless means.

[0014] Furthermore, the laparoscopic 3D optical molecular imaging data includes optical imaging data, fluorescence signal data, optical property data and time series data; the RAL imaging data includes reflected light signal data, two-dimensional image data, RAL optical property data, texture data and contrast data; the equipment operation data includes the movement trajectory, movement speed and movement time of the surgical robot arm.

[0015] Furthermore, the method of preprocessing the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain a 3D imaging feature data set, a RAL imaging feature data set and an equipment feature data set includes:

[0016] S31, converting RAL imaging system data into a standard image format;

[0017] S32, using Gaussian filtering to perform noise reduction processing on the laparoscopic 3D optical molecular imaging data and the RAL imaging system data converted into a standard image format; the Gaussian filtering formula is:

[0018] Where I′(x,y) is the pixel value of the output image at position (x,y); I(x,y) is the pixel value of the input image at position (x,y); x is the horizontal coordinate of the image; y is the vertical coordinate of the image; i and j are relative coordinates; k is the radius of the filter window; G(i,j) is the weight value of the filter kernel at the relative position (i,j);

[0019] The filter window radius is adjusted by the window radius adjustment formula, and the window radius adjustment formula is: Among them, U is the width of the image; W is the height of the image; σ is the standard deviation of the filter;

[0020] S33, applying histogram equalization to improve image contrast; the transformation function of the histogram equalization is: Among them, S k is the new gray level mapped to the original gray level k after histogram equalization; k is the original gray level; n j′ is the number of pixels of the j′th grayscale in the image; j′ is the index of the summed grayscale level; n is the total number of pixels;

[0021] S34, segmenting the laparoscopic 3D optical molecular imaging data after histogram equalization using a region growing method, extracting texture features and morphological features of the image, and constructing a 3D imaging feature data set;

[0022] Extract texture features from an image:

[0023] Extracting the values ​​of the red, green and blue channels of each pixel from the laparoscopic 3D optical molecular imaging data to form the color value of the RGB color space, wherein the three color channels are mixed in different proportions, and the values ​​in the three color channels are integer values ​​between 0 and 255; obtaining the values ​​of the three color channels from the laparoscopic 3D optical molecular imaging data and combining them together to form the RGB color space, and converting the RGB color space into the LAB color space;

[0024] Each pixel in the image is recorded as the central pixel, and the brightness values ​​of each pixel in the image in the L, A, and B channels are L c , A c and B c , the position coordinate is (x c ,y c );

[0025] Select M equally spaced neighboring pixels around the central pixel. The neighboring pixels are located on a circle with the central pixel as the center and a radius of r. The neighboring pixels are denoted as (x d ,y d ), is the coordinate of the dth neighborhood pixel;

[0026] For each neighborhood pixel, compare the brightness value with the central pixel; the comparison formula is: Among them, L d is the brightness value of the neighborhood pixel; C d To compare the results;

[0027] The comparison result C of each neighborhood pixel is d Combine into a binary number and calculate the LT value;

[0028] The calculation formula is: Among them, LT is a digital code for the texture information of the central pixel and the neighboring pixels; H is the number of points in the neighborhood; 2 g-1 To convert the comparison result of each neighborhood pixel into a binary number; g is each neighborhood pixel;

[0029] For all pixels in the image, the texture features of each pixel in the L, A, and B channels are calculated, and the frequency of occurrence of each feature value is counted to construct a feature histogram. The feature histograms of the L, A, and B channels are connected in series to form the texture features of the image.

[0030] S35, extracting morphological features and color features from the RAL imaging data after histogram equalization using a region-based method to construct a RAL imaging feature data set;

[0031] Extracting color features of the image: The process of converting the RGB color space image to the LAB color space image is as follows: eliminating the nonlinear response of the three color channels by gamma correction to make them linear; using a standardized conversion matrix to convert the corrected RGB values ​​into XYZ values; applying a nonlinear processing function to the standardized XYZ values ​​to simulate the perception of the visual system; and using the standardized and nonlinearly processed XYZ values ​​to calculate the values ​​of the LAB color space;

[0032] Processing of morphological feature data: grayscale and denoise the image before processing, binarize the processed image to separate the extracted object and the background image area in the image; obtain the skeleton representation of the extracted object in the image through the skeleton extraction algorithm, and extract the shape features of the extracted object based on the skeleton image;

[0033] The abnormal data in the laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data were detected and eliminated by the box plot method to obtain the processed 3D imaging feature data set, RAL imaging feature data set and equipment feature data set;

[0034] The 3D image feature dataset, the RAL imaging feature dataset and the device feature dataset are normalized and converted into a standard normal distribution to obtain the normalized 3D image feature dataset, the RAL imaging feature dataset and the device feature dataset.

[0035] Furthermore, the method of performing three-dimensional reconstruction on the 3D image feature dataset and the device feature dataset and fusing them with the RAL imaging feature dataset to obtain a comprehensive feature dataset includes:

[0036] The 3D image feature data set is aligned with the RAL imaging feature data set using a rigid transformation. The rigid transformation method is to use a transformation matrix T to register the 3D image feature data to the RAL imaging feature data space. The specific mathematical formula is: P′=T·P; where P is the coordinate of the 3D image data point; P′ is the coordinate of the registered data point.

[0037] The registered 3D image feature dataset and the RAL imaging feature dataset are weighted averaged to obtain a fused comprehensive feature dataset;

[0038] The fused feature data set is voxelized to generate unified three-dimensional structure data; a volume rendering algorithm is used to perform ray projection on the data of the entire voxel space to generate a 3D image; and a comprehensive feature data set is obtained through the processing of the above steps.

[0039] Furthermore, the method of performing weighted averaging fusion of the registered 3D image feature dataset and the RAL imaging feature dataset to obtain a fused comprehensive feature dataset includes:

[0040] The registered 3D image feature data set and the RAL imaging feature data set are weighted average fused by a weighted average fusion formula to obtain a fused comprehensive feature data set; the weighted average fusion formula is:

[0041] V fu (a,b,c)=α1·L(a,b,c)+α2·V(a,b,c)+β1·M(a,b,c)+β2·C(a,b,c); where α1 and α2 are weight coefficients of texture feature and shape feature respectively; β1 and β2 are weight coefficients of morphological feature and color feature respectively; M(a,b,c) is morphological feature; C(a,b,c) is color feature; V fu(a, b, c) is the feature value after weighted average fusion; L(a, b, c) is the texture feature; V(a, b, c) is the shape feature.

[0042] Furthermore, the method for constructing an intraoperative lesion diagnosis model includes:

[0043] The data set is divided into a training set, a validation set and a test set; the data set includes a historical comprehensive feature data set and corresponding optical feature indicators; an intraoperative lesion diagnosis model is constructed, and the intraoperative lesion diagnosis model includes an input layer, a convolution layer, a pooling layer, a batch normalization layer, a fully connected layer and an output layer; the input layer is used to input the historical comprehensive feature data set, and the output layer is used to output the optical feature indicators; the intraoperative lesion diagnosis model is a 3D convolutional neural network model;

[0044] The mean absolute error is used as the loss function to measure the error between the model's predicted value and the actual value; the mean absolute error loss function is: Among them, MAE is the mean absolute error; n is the number of data sets; y i is the true value of the i-th optical characteristic index; is the predicted value of the i-th optical characteristic index;

[0045] Use the training set data to train the model and minimize the loss function using the Adam optimizer. Use the validation set to evaluate the performance of the model and tune the model's hyperparameters until the model performance no longer improves significantly or reaches the preset stopping condition.

[0046] The test set is used to evaluate the performance of the model in the prediction task, and the current comprehensive feature dataset is input into the trained intraoperative lesion diagnosis model to obtain the optical feature indicators.

[0047] Furthermore, the method of comparing the predicted optical characteristic index with a preset optical characteristic index threshold to determine whether a lesion exists includes:

[0048] If the predicted optical characteristic index is greater than or equal to the preset optical characteristic index threshold, it is determined that a lesion exists;

[0049] If the predicted optical characteristic index is less than a preset optical characteristic index threshold, it is determined that no lesion exists.

[0050] Furthermore, the optical characteristic index abnormal region information includes abnormal region position coordinates, abnormal region area, abnormal region type and abnormal region density.

[0051] Furthermore, the method for constructing the lesion severity prediction model includes:

[0052] The dataset is divided into training set, validation set and test set; the dataset includes the abnormal area information of historical optical feature indicators and the corresponding lesion severity. Each dataset is a sample of the lesion severity prediction model. GBDT is selected as the specific implementation of the gradient boosting machine model to handle the regression task;

[0053] Initialize GBDT parameters and define the objective function; GBDT parameters include the maximum number of leaf nodes in each tree, learning rate, number of trees and maximum depth of trees; use the abnormal area information of historical optical feature indicators as input data and the corresponding lesion severity as output label to train the lesion severity prediction model; the lesion severity prediction model is a gradient boosting machine model;

[0054] The mean square error is used as the loss function to measure the difference between the model's predicted value and the actual value; the mean square error loss function is: Where n is the number of samples in the data set; Y b is the actual value of the bth sample in the data set; is the predicted value of the bth sample in the data set; b is the index of the sample;

[0055] In each iteration, GBDT builds a new decision tree to fit the negative gradient of the loss function of the previous step, uses the negative gradient of the objective function as the learning target of the new tree, and adjusts the contribution of the new tree to the final result through the learning rate to minimize the loss function.

[0056] Use the validation set to evaluate the performance of the model, tune the model, and adjust the model parameters based on its performance feedback until the model performance no longer improves significantly or reaches the preset stopping condition; use the trained lesion severity prediction model to predict the abnormal area information of the current optical feature indicators to obtain the severity of the lesion.

[0057] Furthermore, the method of implementing a corresponding tumor treatment plan according to the severity of the lesion includes:

[0058] The preset lesion severity threshold is further divided into a low-risk severity threshold T L , medium risk severity threshold T M and high risk severity threshold T H ; The predicted lesion severity is recorded as D; The lesion severity D is respectively compared with the low risk severity threshold T L , medium risk severity threshold T M and high risk severity threshold T H Make comparisons and determine the level of tumor treatment options;

[0059] If D≤T L, then the tumor treatment plan level is determined to be a low-grade tumor treatment plan level; if T L <D<T M , then the tumor treatment plan level is determined to be moderate tumor treatment plan level; if D>T H , then the tumor treatment plan level is determined to be a high tumor treatment plan level;

[0060] Level of treatment regimen for low-grade tumors: For patients with low-grade lesions, a regimen consisting of local surgical resection, local radiation therapy, or local immunotherapy is implemented without involving systemic therapy;

[0061] For moderate tumors, treatment options are: For patients with moderate lesion severity, a combination of surgical resection, systemic chemotherapy, systemic radiotherapy, and targeted therapy is implemented;

[0062] For patients with high-grade tumors, a comprehensive treatment plan including wide surgical resection, intensive-dose chemotherapy, systemic radiotherapy and immunotherapy is implemented.

[0063] The technical effects and advantages of the intelligent auxiliary processing system for abdominal surgery images of the present invention are as follows:

[0064] The present invention reduces random noise in images and improves image clarity and quality through Gaussian filtering noise reduction; histogram equalization can enhance image contrast and make details in images more clearly visible; segmenting ROI regions through region growing method can more accurately extract image features related to lesions; extracting texture features and shape features helps to better identify and analyze the characteristics of lesion regions; converting RGB color space to LAB color space can better reflect the human eye's perception of color and help identify lesions; through a specific texture feature calculation method, the unique texture pattern of the lesion region can be highlighted; the filter window radius can be automatically adjusted according to the image content and the filter standard deviation, which provides a more suitable filter window size for different images and different noise levels;

[0065] Providing customized treatment plans based on the specific severity of the patient's lesion can better meet the individual needs of the patient; by distinguishing treatment plans for different risk levels, medical resources can be allocated more effectively, ensuring that high-risk patients receive more intensive treatment and low-risk patients avoid overtreatment; reasonable allocation of treatment plans can reduce unnecessary medical expenses and reduce overall treatment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 It is a schematic diagram of the structure of an intelligent auxiliary processing system for abdominal surgery images of the present invention;

[0067] Figure 2 The figure is a flow chart of an intelligent auxiliary processing method for abdominal surgery images of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] Embodiment 1

[0070] See also Figure 1 As shown, the intelligent auxiliary processing system for abdominal surgery images described in this embodiment includes:

[0071] Data acquisition module, used to collect laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data;

[0072] A data processing module is used to pre-process the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain a 3D imaging feature data set, a RAL imaging feature data set and an equipment feature data set; perform three-dimensional reconstruction on the 3D imaging feature data set and the equipment feature data set, and fuse them with the RAL imaging feature data set to obtain a comprehensive feature data set;

[0073] An intraoperative lesion diagnosis module is used to construct an intraoperative lesion diagnosis model, input the comprehensive feature data set into the intraoperative lesion diagnosis model, and predict the optical feature indicators;

[0074] The intraoperative lesion analysis module compares the predicted optical characteristic index with the preset optical characteristic index threshold to determine whether there is a lesion; if it is determined that there is no lesion, the diagnosis result is displayed as normal; if it is determined that there is a lesion, the diagnosis result is displayed as abnormal, and the optical characteristic index abnormal area information is output;

[0075] The clinical decision-making module is used to construct a lesion severity prediction model, input the acquired optical characteristic index abnormal area information into the lesion severity prediction model, and predict the severity of the lesion;

[0076] The scheme control module is used to implement the corresponding tumor treatment scheme according to the severity of the lesion; each module is connected by wired and / or wireless means.

[0077] Laparoscopic 3D optical molecular imaging data includes optical imaging data, fluorescence signal data, optical property data and time series data; RAL imaging data includes reflected light signal data, two-dimensional image data, RAL optical property data, texture data and contrast data; equipment operation data includes the movement trajectory, movement speed and movement time of the surgical robot arm.

[0078] The method of preprocessing the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain a 3D imaging feature data set, a RAL imaging feature data set and an equipment feature data set includes:

[0079] S31, converting RAL imaging system data into a standard image format;

[0080] S32, using Gaussian filtering to perform noise reduction processing on the laparoscopic 3D optical molecular imaging data and the RAL imaging system data converted into a standard image format; the Gaussian filtering formula is:

[0081] Among them, I′(x,y) is the pixel value of the output image at position (x,y); I(x,y) is the pixel value of the input image at position (x,y); x is the horizontal coordinate of the image; y is the vertical coordinate of the image; i and j are relative coordinates, used to indicate the relative offset of the convolution kernel around the current center position (x,y); k is the filter window radius; G(i,j) is the weight value of the filter kernel at the relative position (i,j);

[0082] The filter window radius is adjusted by the window radius adjustment formula, and the window radius adjustment formula is: Among them, U is the width of the image; W is the height of the image; σ is the standard deviation of the filter;

[0083] For example, if the image width is 8, the image height is 20, and the filter standard deviation is 5, then the filter window radius is 8.

[0084] S33, applying histogram equalization to improve image contrast; the transformation function of the histogram equalization is: Among them, S k is the new gray level mapped to the original gray level k after histogram equalization; k is the original gray level; n j′ is the number of pixels of the j′th grayscale in the image; j′ is the index of the summed grayscale level; n is the total number of pixels;

[0085] S34, segmenting the laparoscopic 3D optical molecular imaging data after histogram equalization using a region growing method, extracting texture features and morphological features of the image, and constructing a 3D imaging feature data set;

[0086] Extract texture features from an image:

[0087] Extracting the values ​​of the red, green and blue channels of each pixel from the laparoscopic 3D optical molecular imaging data to form the color value of the RGB color space, wherein the three color channels are mixed in different proportions, and the values ​​in the three color channels are integer values ​​between 0 and 255; obtaining the values ​​of the three color channels from the laparoscopic 3D optical molecular imaging data and combining them together to form the RGB color space, and converting the RGB color space into the LAB color space;

[0088] Each pixel in the image is recorded as the central pixel, and the brightness values ​​of each pixel in the image in the L, A, and B channels are L c , A c and B c , the position coordinate is (x c ,y c );

[0089] Select M equally spaced neighboring pixels around the central pixel. The neighboring pixels are located on a circle with the central pixel as the center and a radius of r. The neighboring pixels are denoted as (x d ,y d ), is the coordinate of the dth neighborhood pixel;

[0090] For each neighborhood pixel, compare the brightness value with the central pixel; the comparison formula is: Among them, L d is the brightness value of the neighborhood pixel; C d To compare the results;

[0091] For example, the brightness value of the neighborhood pixel is 100, L c The brightness value is 80, then the comparison result is C d is 1.

[0092] The comparison result C of each neighborhood pixel is d Combine into a binary number and calculate the LT value;

[0093] The calculation formula is: Among them, LT is a digital code for the texture information of the central pixel and the neighboring pixels; H is the number of points in the neighborhood; 2 g-1 To convert the comparison result of each neighborhood pixel into a binary number; g is each neighborhood pixel;

[0094] For all pixels in the image, the texture features of each pixel in the L, A, and B channels are calculated, and the frequency of occurrence of each feature value is counted to construct a feature histogram. The feature histograms of the L, A, and B channels are connected in series to form the texture features of the image.

[0095] S35, extracting morphological features and color features from the RAL imaging data after histogram equalization using a region-based method to construct a RAL imaging feature data set;

[0096] Extracting color features of the image: The process of converting the RGB color space image to the LAB color space image is as follows: eliminating the nonlinear response of the three color channels by gamma correction to make them linear; using a standardized conversion matrix to convert the corrected RGB values ​​into XYZ values; applying a nonlinear processing function to the standardized XYZ values ​​to simulate the perception of the visual system; and using the standardized and nonlinearly processed XYZ values ​​to calculate the values ​​of the LAB color space;

[0097] Processing of morphological feature data: grayscale and denoise the image before processing, binarize the processed image to separate the extracted object and the background image area in the image; obtain the skeleton representation of the extracted object in the image through the skeleton extraction algorithm, and extract the shape features of the extracted object based on the skeleton image;

[0098] The abnormal data in the laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data were detected and eliminated by the box plot method to obtain the processed 3D imaging feature data set, RAL imaging feature data set and equipment feature data set;

[0099] The 3D image feature dataset, the RAL imaging feature dataset and the device feature dataset are normalized and converted into a standard normal distribution to obtain the normalized 3D image feature dataset, the RAL imaging feature dataset and the device feature dataset.

[0100] The method of performing three-dimensional reconstruction on the 3D image feature dataset and the device feature dataset and fusing them with the RAL imaging feature dataset to obtain a comprehensive feature dataset includes:

[0101] The 3D image feature data set is aligned with the RAL imaging feature data set using a rigid transformation. The rigid transformation method is to use a transformation matrix T to register the 3D image feature data to the RAL imaging feature data space. The specific mathematical formula is: P′=T·P; where P is the coordinate of the 3D image data point; P′ is the coordinate of the registered data point.

[0102] The registered 3D image feature dataset and the RAL imaging feature dataset are weighted averaged to obtain a fused comprehensive feature dataset;

[0103] The fused feature data set is voxelized to generate unified three-dimensional structure data; a volume rendering algorithm is used to perform ray projection on the data of the entire voxel space to generate a 3D image; and a comprehensive feature data set is obtained through the processing of the above steps.

[0104] The method of performing weighted average fusion of the registered 3D image feature dataset and the RAL imaging feature dataset to obtain a fused comprehensive feature dataset includes:

[0105] The registered 3D image feature data set and the RAL imaging feature data set are weighted average fused by a weighted average fusion formula to obtain a fused comprehensive feature data set; the weighted average fusion formula is:

[0106] V fu (a,b,c)=α1·L(a,b,c)+α2·V(a,b,c)+β1·M(a,b,c)+β2·C(a,b,c); where α1 and α2 are weight coefficients of texture feature and shape feature respectively; β1 and β2 are weight coefficients of morphological feature and color feature respectively; M(a,b,c) is morphological feature; C(a,b,c) is color feature; V fu (a, b, c) is the feature value after weighted average fusion; L(a, b, c) is the texture feature; V(a, b, c) is the shape feature.

[0107] Methods for constructing intraoperative lesion diagnosis models include:

[0108] The data set is divided into a training set, a validation set and a test set; the data set includes a historical comprehensive feature data set and corresponding optical feature indicators; an intraoperative lesion diagnosis model is constructed, and the intraoperative lesion diagnosis model includes an input layer, a convolution layer, a pooling layer, a batch normalization layer, a fully connected layer and an output layer; the input layer is used to input the historical comprehensive feature data set, and the output layer is used to output the optical feature indicators; the intraoperative lesion diagnosis model is a 3D convolutional neural network model;

[0109] The mean absolute error is used as the loss function to measure the error between the model's predicted value and the actual value; the mean absolute error loss function is: Among them, MAE is the mean absolute error; n is the number of data sets; y i is the true value of the i-th optical characteristic index; is the predicted value of the i-th optical characteristic index;

[0110] Use the training set data to train the model and minimize the loss function using the Adam optimizer. Use the validation set to evaluate the performance of the model and tune the model's hyperparameters until the model performance no longer improves significantly or reaches the preset stopping condition.

[0111] The test set is used to evaluate the performance of the model in the prediction task, and the current comprehensive feature dataset is input into the trained intraoperative lesion diagnosis model to obtain the optical feature indicators.

[0112] The method of comparing the predicted optical characteristic index with the preset optical characteristic index threshold to determine whether there is a lesion includes:

[0113] If the predicted optical characteristic index is greater than or equal to the preset optical characteristic index threshold, it is determined that a lesion exists;

[0114] If the predicted optical characteristic index is less than a preset optical characteristic index threshold, it is determined that no lesion exists.

[0115] The optical characteristic index abnormal region information includes the abnormal region position coordinates, the abnormal region area, the abnormal region type and the abnormal region density.

[0116] The method for constructing the lesion severity prediction model includes:

[0117] The dataset is divided into training set, validation set and test set; the dataset includes the abnormal area information of historical optical feature indicators and the corresponding lesion severity. Each dataset is a sample of the lesion severity prediction model. GBDT is selected as the specific implementation of the gradient boosting machine model to handle the regression task;

[0118] Initialize GBDT parameters and define the objective function; GBDT parameters include the maximum number of leaf nodes in each tree, learning rate, number of trees and maximum depth of trees; use the abnormal area information of historical optical feature indicators as input data and the corresponding lesion severity as output label to train the lesion severity prediction model; the lesion severity prediction model is a gradient boosting machine model;

[0119] The mean square error is used as the loss function to measure the difference between the model's predicted value and the actual value; the mean square error loss function is: Where n is the number of samples in the data set; Y b is the actual value of the bth sample in the data set; is the predicted value of the bth sample in the data set; b is the index of the sample;

[0120] In each iteration, GBDT builds a new decision tree to fit the negative gradient of the loss function of the previous step, uses the negative gradient of the objective function as the learning target of the new tree, and adjusts the contribution of the new tree to the final result through the learning rate to minimize the loss function.

[0121] Use the validation set to evaluate the performance of the model, tune the model, and adjust the model parameters based on its performance feedback until the model performance no longer improves significantly or reaches the preset stopping condition; use the trained lesion severity prediction model to predict the abnormal area information of the current optical feature indicators to obtain the severity of the lesion.

[0122] Methods for implementing tumor treatment plans according to the severity of the lesion include:

[0123] The preset lesion severity threshold is further divided into a low-risk severity threshold T L , medium risk severity threshold T M and high risk severity threshold T H ; The predicted lesion severity is recorded as D; The lesion severity D is respectively compared with the low risk severity threshold T L , medium risk severity threshold T M and high risk severity threshold T H Make comparisons and determine the level of tumor treatment options;

[0124] If D≤T L , then the tumor treatment plan level is determined to be a low-grade tumor treatment plan level; if T L <D<T M , then the tumor treatment plan level is determined to be moderate tumor treatment plan level; if D>T H , then the tumor treatment plan level is determined to be a high tumor treatment plan level;

[0125] Level of treatment regimen for low-grade tumors: For patients with low-grade lesions, a regimen consisting of local surgical resection, local radiation therapy, or local immunotherapy is implemented without involving systemic therapy;

[0126] For moderate tumors, treatment options are: For patients with moderate lesion severity, a combination of surgical resection, systemic chemotherapy, systemic radiotherapy, and targeted therapy is implemented;

[0127] For patients with high-grade tumors, a comprehensive treatment plan including wide surgical resection, intensive-dose chemotherapy, systemic radiotherapy and immunotherapy is implemented.

[0128] For example, suppose the thresholds are set as follows: low risk severity threshold: 0.2; medium risk severity threshold: 0.5; high risk severity threshold: 0.8; the prediction result shows that the severity of the lesion is 0.4;

[0129] The predicted lesion severity is 0.4; compare the lesion severity with the threshold:

[0130] Compared with the low risk severity threshold (0.2): 0.4>0.2

[0131] Compared with the medium risk severity threshold (0.5): 0.4 < 0.5

[0132] Compared with the high risk severity threshold (0.8): 0.4 < 0.8

[0133] Because 0.4 is greater than the low-risk severity threshold but less than the moderate-risk severity threshold, the tumor treatment option grade is moderate.

[0134] The preset optical characteristic index threshold is set by a staff member, and different optical characteristic indexes are collected through an intelligent auxiliary processing terminal, and an average value of multiple optical characteristic indexes is taken as the preset optical characteristic index threshold.

[0135] In this embodiment, the random noise in the image can be reduced by Gaussian filtering and noise reduction, and the clarity and quality of the image can be improved; the histogram equalization can enhance the contrast of the image, making the details in the image more clearly visible; the ROI area can be segmented by the region growing method, and the image features related to the lesion can be extracted more accurately; the extraction of texture features and shape features is helpful to better identify and analyze the characteristics of the lesion area; the conversion of the RGB color space to the LAB color space can better reflect the human eye's perception of color and help identify the lesion; the unique texture pattern of the lesion area can be highlighted by a specific texture feature calculation method; the window radius of the filter can be automatically adjusted according to the image content and the filter standard deviation, which provides a more suitable filter window size for different images and different noise levels;

[0136] Providing customized treatment plans based on the specific severity of the patient's lesion can better meet the individual needs of the patient; by distinguishing treatment plans for different risk levels, medical resources can be allocated more effectively, ensuring that high-risk patients receive more intensive treatment and low-risk patients avoid overtreatment; reasonable allocation of treatment plans can reduce unnecessary medical expenses and reduce overall treatment costs.

[0137] Embodiment 2

[0138] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, which provides an intelligent auxiliary processing method for abdominal surgery images, including:

[0139] S1, collect laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data;

[0140] S2. Preprocess the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain a 3D imaging feature data set, a RAL imaging feature data set and an equipment feature data set; perform three-dimensional reconstruction on the 3D imaging feature data set and the equipment feature data set, and fuse them with the RAL imaging feature data set to obtain a comprehensive feature data set;

[0141] S3, constructing an intraoperative lesion diagnosis model, inputting the comprehensive feature data set into the intraoperative lesion diagnosis model, and predicting the optical feature index;

[0142] S4, comparing the predicted optical characteristic index with the preset optical characteristic index threshold to determine whether there is a lesion; if it is determined that there is no lesion, the diagnosis result is displayed as normal; if it is determined that there is a lesion, the diagnosis result is displayed as abnormal, and the optical characteristic index abnormal area information is output;

[0143] S5, constructing a lesion severity prediction model, inputting the acquired optical characteristic index abnormal area information into the lesion severity prediction model, and predicting the severity of the lesion;

[0144] S6. Implement corresponding tumor treatment plan according to the severity of the lesion.

[0145] Since the electronic device introduced in this embodiment is an electronic device used to implement an intelligent auxiliary processing system for abdominal surgery images in the embodiment of this application, based on the intelligent auxiliary processing system for abdominal surgery images introduced in the embodiment of this application, a person skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not described in detail here. As long as a person skilled in the art implements the electronic device used in the intelligent auxiliary processing system for abdominal surgery images in the embodiment of this application, it belongs to the scope of protection of this application.

[0146] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0147] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An intelligent auxiliary processing system for abdominal surgery images, characterized in that: include: Data acquisition module, used to collect laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data; A data processing module is used to pre-process the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain a 3D imaging feature data set, a RAL imaging feature data set and an equipment feature data set; perform three-dimensional reconstruction on the 3D imaging feature data set and the equipment feature data set, and fuse them with the RAL imaging feature data set to obtain a comprehensive feature data set; An intraoperative lesion diagnosis module is used to construct an intraoperative lesion diagnosis model, input the comprehensive feature data set into the intraoperative lesion diagnosis model, and predict the optical feature indicators; The intraoperative lesion analysis module compares the predicted optical characteristic index with the preset optical characteristic index threshold to determine whether there is a lesion; if it is determined that there is no lesion, the diagnosis result is displayed as normal; If it is determined that there is a lesion, the diagnosis result is displayed as abnormal, and the optical characteristic index abnormal area information is output; The clinical decision-making module is used to construct a lesion severity prediction model, input the acquired optical characteristic index abnormal area information into the lesion severity prediction model, and predict the severity of the lesion; The scheme control module is used to implement the corresponding tumor treatment scheme according to the severity of the lesion; each module is connected by wired and / or wireless means.

2. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 1 is characterized in that: The laparoscopic 3D optical molecular imaging data includes optical imaging data, fluorescence signal data, optical property data and time series data; the RAL imaging data includes reflected light signal data, two-dimensional image data, RAL optical property data, texture data and contrast data; the equipment operation data includes the movement trajectory, movement speed and movement time of the surgical robot arm.

3. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 2 is characterized in that: The method of preprocessing the acquired laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data to obtain a 3D imaging feature data set, a RAL imaging feature data set and an equipment feature data set comprises: S31, converting RAL imaging system data into a standard image format; S32, using Gaussian filtering to perform noise reduction processing on the laparoscopic 3D optical molecular imaging data and the RAL imaging system data converted into a standard image format; the Gaussian filtering formula is: Where I′(x,y) is the pixel value of the output image at position (x,y); I(x,y) is the pixel value of the input image at position (x,y); x is the horizontal coordinate of the image; y is the vertical coordinate of the image; i and j are relative coordinates; k is the radius of the filter window; G(i,j) is the weight value of the filter kernel at the relative position (i,j); The filter window radius is adjusted by the window radius adjustment formula, and the window radius adjustment formula is: Among them, U is the width of the image; W is the height of the image; σ is the standard deviation of the filter; S33, applying histogram equalization to improve image contrast; the transformation function of the histogram equalization is: Among them, S k is the new gray level mapped to the original gray level k after histogram equalization; k is the original gray level; n j′ is the number of pixels of the j′th grayscale in the image; j′ is the index of the summed grayscale level; n is the total number of pixels; S34, segmenting the laparoscopic 3D optical molecular imaging data after histogram equalization using a region growing method, extracting texture features and shape features of the image, and constructing a 3D imaging feature data set; Extract texture features from an image: Extracting the values ​​of the red, green and blue channels of each pixel from the laparoscopic 3D optical molecular imaging data to form the color value of the RGB color space, wherein the three color channels are mixed in different proportions, and the values ​​in the three color channels are integer values ​​between 0 and 255; obtaining the values ​​of the three color channels from the laparoscopic 3D optical molecular imaging data and combining them together to form the RGB color space, and converting the RGB color space into the LAB color space; Each pixel in the image is recorded as the central pixel, and the brightness values ​​of each pixel in the image in the L, A, and B channels are L c , A c and B c , the position coordinate is (x c ,y c ); Select M equally spaced neighboring pixels around the central pixel. The neighboring pixels are located on a circle with the central pixel as the center and a radius of r. The neighboring pixels are denoted as (x d ,y d ), is the coordinate of the dth neighborhood pixel; For each neighborhood pixel, compare the brightness value with the central pixel; the comparison formula is: Among them, L d is the brightness value of the neighborhood pixel; C d To compare the results; The comparison result C of each neighborhood pixel is d Combine into a binary number and calculate the LT value; The calculation formula is: Among them, LT is a digital code for the texture information of the central pixel and the neighboring pixels; H is the number of points in the neighborhood; 2 g-1 To convert the comparison result of each neighborhood pixel point into a binary number; g is each neighborhood pixel point; For all pixels in the image, the texture features of each pixel in the L, A, and B channels are calculated, and the frequency of occurrence of each feature value is counted to construct a feature histogram. The feature histograms of the L, A, and B channels are connected in series to form the texture features of the image. S35, extracting morphological features and color features from the RAL imaging data after histogram equalization using a region-based method to construct a RAL imaging feature data set; Extracting color features of the image: The process of converting the RGB color space image to the LAB color space image is as follows: eliminating the nonlinear response of the three color channels by gamma correction to make them linear; using a standardized conversion matrix to convert the corrected RGB values ​​into XYZ values; applying a nonlinear processing function to the standardized XYZ values ​​to simulate the perception of the visual system; and using the standardized and nonlinearly processed XYZ values ​​to calculate the values ​​of the LAB color space; Extract the shape and morphological features of the image: grayscale and denoise the image before preprocessing, and binarize the processed image to separate the extracted object and the background image area in the image; obtain the skeleton representation of the extracted object in the image through the skeleton extraction algorithm, and extract the shape and morphological features of the extracted object based on the skeleton image; The abnormal data in the laparoscopic 3D optical molecular imaging data, RAL imaging data and equipment operation data were detected and eliminated by the box plot method to obtain the processed 3D imaging feature data set, RAL imaging feature data set and equipment feature data set; The 3D image feature dataset, the RAL imaging feature dataset and the device feature dataset are normalized and converted into a standard normal distribution to obtain the normalized 3D image feature dataset, the RAL imaging feature dataset and the device feature dataset.

4. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 3 is characterized in that: The method of performing three-dimensional reconstruction on the 3D image feature dataset and the device feature dataset and fusing them with the RAL imaging feature dataset to obtain a comprehensive feature dataset comprises: The 3D image feature data set is aligned with the RAL imaging feature data set using a rigid transformation. The rigid transformation method is to use a transformation matrix T to register the 3D image feature data to the RAL imaging feature data space. The specific mathematical formula is: P′=T·P; where P is the coordinate of the 3D image data point; P′ is the coordinate of the registered data point. The registered 3D image feature dataset and the RAL imaging feature dataset are weighted averaged to obtain a fused comprehensive feature dataset; The fused feature data set is voxelized to generate unified three-dimensional structure data; a volume rendering algorithm is used to perform ray projection on the data of the entire voxel space to generate a 3D image; and a comprehensive feature data set is obtained through the processing of the above steps.

5. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 4 is characterized in that: The method of performing weighted average fusion of the registered 3D image feature dataset and the RAL imaging feature dataset to obtain a fused comprehensive feature dataset includes: The registered 3D image feature data set and the RAL imaging feature data set are weighted average fused by a weighted average fusion formula to obtain a fused comprehensive feature data set; the weighted average fusion formula is: V fu (a,b,c)=α1·L(a,b,c)+α2·V(a,b,c)+β1·M(a,b,c)+β2·C(a,b,c); where α1 and α2 are weight coefficients of texture feature and shape feature respectively; β1 and β2 are weight coefficients of morphological feature and color feature respectively; M(a,b,c) is morphological feature; C(a,b,c) is color feature; V fu (a, b, c) is the feature value after weighted average fusion; L(a, b, c) is the texture feature; V(a, b, c) is the shape feature.

6. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 5, characterized in that: The method for constructing an intraoperative lesion diagnosis model comprises: The data set is divided into a training set, a validation set and a test set; the data set includes a historical comprehensive feature data set and corresponding optical feature indicators; an intraoperative lesion diagnosis model is constructed, and the intraoperative lesion diagnosis model includes an input layer, a convolution layer, a pooling layer, a batch normalization layer, a fully connected layer and an output layer; the input layer is used to input the historical comprehensive feature data set, and the output layer is used to output the optical feature indicators; the intraoperative lesion diagnosis model is a 3D convolutional neural network model; The mean absolute error is used as the loss function to measure the error between the model's predicted value and the actual value; the mean absolute error loss function is: Among them, MAE is the mean absolute error; n is the number of data sets; y i is the true value of the i-th optical characteristic index; is the predicted value of the i-th optical characteristic index; Use the training set data to train the model and minimize the loss function using the Adam optimizer. Use the validation set to evaluate the performance of the model and tune the model's hyperparameters until the model performance no longer improves significantly or reaches the preset stopping condition. The test set is used to evaluate the performance of the model in the prediction task, and the current comprehensive feature dataset is input into the trained intraoperative lesion diagnosis model to obtain the optical feature indicators.

7. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 6 is characterized in that: The method of comparing the predicted optical characteristic index with the preset optical characteristic index threshold to determine whether there is a lesion includes: If the predicted optical characteristic index is greater than or equal to the preset optical characteristic index threshold, it is determined that a lesion exists; If the predicted optical characteristic index is less than a preset optical characteristic index threshold, it is determined that no lesion exists.

8. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 7 is characterized in that: The optical characteristic index abnormal region information includes abnormal region position coordinates, abnormal region area, abnormal region type and abnormal region density.

9. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 8, characterized in that: The method for constructing the lesion severity prediction model comprises: The dataset is divided into training set, validation set and test set; the dataset includes the abnormal area information of historical optical feature indicators and the corresponding lesion severity. Each dataset is a sample of the lesion severity prediction model. GBDT is selected as the specific implementation of the gradient boosting machine model to handle the regression task; Initialize GBDT parameters and define the objective function; GBDT parameters include the maximum number of leaf nodes in each tree, learning rate, number of trees and maximum depth of trees; use the abnormal area information of historical optical feature indicators as input data and the corresponding lesion severity as output label to train the lesion severity prediction model; the lesion severity prediction model is a gradient boosting machine model; The mean square error is used as the loss function to measure the difference between the model's predicted value and the actual value; the mean square error loss function is: Where n is the number of samples in the data set; Y b is the actual value of the bth sample in the data set; is the predicted value of the bth sample in the data set; b is the index of the sample; In each iteration, GBDT builds a new decision tree to fit the negative gradient of the loss function of the previous step, uses the negative gradient of the objective function as the learning target of the new tree, and adjusts the contribution of the new tree to the final result through the learning rate to minimize the loss function. Use the validation set to evaluate the performance of the model, tune the model, and adjust the model parameters based on its performance feedback until the model performance no longer improves significantly or reaches the preset stopping condition; use the trained lesion severity prediction model to predict the abnormal area information of the current optical feature indicators to obtain the severity of the lesion.

10. The intelligent auxiliary processing system for laparoscopic surgery images according to claim 9, characterized in that: The method of implementing a corresponding tumor treatment plan according to the severity of the lesion includes: The preset lesion severity threshold is further divided into a low-risk severity threshold T L , medium risk severity threshold T M and high risk severity threshold T H ; The predicted lesion severity is recorded as D; The lesion severity D is respectively compared with the low risk severity threshold T L , medium risk severity threshold T M and high risk severity threshold T H Make comparisons and determine the level of tumor treatment options; If D≤T L , then the tumor treatment plan level is determined to be a low-grade tumor treatment plan level; if T L <D<T M , then the tumor treatment plan level is determined to be moderate tumor treatment plan level; if D>T H , then the tumor treatment plan level is determined to be a high tumor treatment plan level; Level of treatment regimen for low-grade tumors: For patients with low-grade lesions, a regimen consisting of local surgical resection, local radiation therapy, or local immunotherapy is implemented without involving systemic therapy; For moderate tumors, treatment options are: For patients with moderate lesion severity, a combination of surgical resection, systemic chemotherapy, systemic radiotherapy, and targeted therapy is implemented; For patients with high-grade tumors, a comprehensive treatment plan including wide surgical resection, intensive-dose chemotherapy, systemic radiotherapy and immunotherapy is implemented.

Images