Real-time operation feedback classification method and system

Abstract

The invention is suitable for the technical field of medical informatization, and provides a real-time operation feedback classification method and system.The method comprises the following steps that multi-dimensional information is collected, and the multi-dimensional information comprises microendoscope data, mechanical sensor data and physiological monitoring data; preprocessing the multi-dimensional information, and performing feature extraction and fusion on the preprocessed multi-dimensional information to obtain operation feedback feature information; constructing a surgical feedback data set based on the historical surgical feedback feature information, and expanding the surgical feedback data set by adopting a generative adversarial network; training and optimizing the SVM classification model through the operation feedback data set to obtain a feedback classification model; and inputting the real-time operation feedback feature information into the feedback classification model, and performing real-time classification. According to the method, the limitation of a single data source can be avoided, the potential correlation between the surgical operation and the physiological reaction of the patient can be mined, and the accuracy and reliability of feedback classification are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical informatization, and particularly to a real-time surgical feedback classification method and system. Background Art

[0002] In the field of modern medicine, surgery, as a key means of treating various diseases, its success or failure is directly related to the life and health of patients and their quality of life after surgery. With the rapid development of medical technology, advanced technologies such as minimally invasive surgery and robot-assisted surgery have emerged continuously, greatly improving the accuracy and safety of surgery. However, the complex and changeable physiological environment, differences in tissue characteristics, and subtle differences in operating techniques during the surgery may all have a significant impact on the surgical effect. Therefore, how to achieve real-time and accurate feedback and evaluation during the surgery so that doctors can adjust the surgical strategy in time has become an important research topic in current medicine.

[0003] Traditional surgical feedback mechanisms mainly rely on doctors' experience judgment and postoperative pathological analysis. This method has obvious lag and cannot provide immediate guidance during the surgery. In recent years, with the progress of sensor technology, image processing technology, and artificial intelligence algorithms, multi-dimensional information during the surgery can be obtained, providing a rich data basis for surgical feedback.

[0004] However, how to effectively process these massive and heterogeneous data, extract clinically significant features, and build an efficient and accurate classification model to achieve real-time classification and early warning of surgical feedback is still the main technical challenge faced currently. Therefore, a real-time surgical feedback classification method and system are needed to solve the above problems. Summary of the Invention

[0005] Aiming at the deficiencies of the existing technology, the purpose of the present invention is to provide a real-time surgical feedback classification method and system to solve the problems in the above background art.

[0006] The present invention is implemented as follows. A real-time surgical feedback classification method, the method includes the following steps: Collect multi-dimensional information, where the multi-dimensional information includes microscopic endoscope data, mechanical sensor data, and physiological monitoring data; Preprocess the multi-dimensional information, and perform feature extraction and fusion on the preprocessed multi-dimensional information to obtain surgical feedback feature information; Construct a surgical feedback data set based on historical surgical feedback feature information, and use a generative adversarial network to expand the surgical feedback data set; Train and optimize an SVM classification model through the surgical feedback data set to obtain a feedback classification model; Input the real-time surgical feedback feature information into the feedback classification model for real-time classification.

[0007] As a further solution of the present invention: The step of collecting multi-dimensional information specifically includes: Collecting microscopic endoscope data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image, and the microscopic endoscope supports RGB and near-infrared dual-modal imaging; Collecting mechanical sensor data through a six-axis force sensor or a six-axis torque sensor, and the sampling frequency ≥ 1 kHz; Monitoring heart rate data, blood pressure data, blood oxygen saturation, and electroencephalogram signals, and integrating them to obtain physiological monitoring data.

[0008] As a further solution of the present invention: The step of extracting features and fusing the preprocessed multi-dimensional information to obtain surgical feedback feature information specifically includes: Extracting endoscopic features, using a pre-trained ResNet-50 or EfficientNet as the backbone network to extract features from microscopic images to obtain deep feature vectors; Extracting mechanical features, calculating the mean and variance of the contact force between the surgical instrument and the tissue, performing a fast Fourier transform on the contact force signal, and extracting the spectral energy distribution; Extracting physiological features, calculating the mean and standard deviation of heart rate, heart rate variability, and the energy of the power spectral density of the electroencephalogram signal in a set frequency band; Fusing the endoscopic features, mechanical features, and physiological features into a high-dimensional vector to obtain surgical feedback feature information.

[0009] As a further solution of the present invention: The step of expanding the surgical feedback data set using a generative adversarial network specifically includes: Determining the basic architecture of the GAN model, selecting DCGAN or StyleGAN for endoscopic features, selecting LSTM-GAN or 1D-CNN-GAN for mechanical features, and selecting coupled GAN or cross-modal GAN for physiological features; Determining the generator structure, discriminator structure, and loss function, and expanding the surgical feedback data set through a generative adversarial network.

[0010] Another object of the present invention is to provide a real-time surgical feedback classification system, and the system includes: A multi-dimensional information collection module for collecting multi-dimensional information, and the multi-dimensional information includes microscopic endoscope data, mechanical sensor data, and physiological monitoring data; An information processing and extraction module for preprocessing the multi-dimensional information, and extracting and fusing the preprocessed multi-dimensional information to obtain surgical feedback feature information; A feedback dataset augmentation module, which is used to construct a surgical feedback dataset based on historical surgical feedback feature information and augment the surgical feedback dataset using a generative adversarial network; A classification model construction module, which is used to train and optimize an SVM classification model through a surgical feedback dataset to obtain a feedback classification model; A classification result output module, which is used to input real-time surgical feedback feature information into the feedback classification model for real-time classification.

[0011] As a further solution of the present invention: the multi-dimensional information acquisition module includes: An endoscope data acquisition unit, which is used to acquire microscopic endoscope data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image, and the microscopic endoscope supports RGB and near-infrared dual-modal imaging; A mechanical data acquisition unit, which is used to acquire mechanical sensor data through a six-axis force sensor or a six-axis torque sensor, and the sampling frequency ≥ 1 kHz; A physiological monitoring acquisition unit, which is used to monitor heart rate data, blood pressure data, blood oxygen saturation, and electroencephalogram signals, and integrate them to obtain physiological monitoring data.

[0012] As a further solution of the present invention: the information processing and extraction module further includes: An endoscope feature extraction unit, which is used to extract endoscope features, uses a pre-trained ResNet-50 or EfficientNet as the backbone network to extract features from microscopic images, and obtains a deep feature vector; A mechanical feature extraction unit, which is used to extract mechanical features, calculates the mean and variance of the contact force between the surgical instrument and the tissue, performs a fast Fourier transform on the contact force signal, and extracts the spectral energy distribution; A physiological feature extraction unit, which is used to extract physiological features, calculates the mean and standard deviation of heart rate, heart rate variability, and the energy of the power spectral density of the electroencephalogram signal in a set frequency band; A feature splicing and fusion unit, which is used to fuse endoscope features, mechanical features, and physiological features into a high-dimensional vector to obtain surgical feedback feature information.

[0013] As a further solution of the present invention: the feedback dataset augmentation module includes: A GAN architecture determination unit, which is used to determine the basic architecture of the GAN model. For endoscope features, DCGAN or StyleGAN is selected; for mechanical features, LSTM-GAN or 1D-CNN-GAN is selected; for physiological features, coupled GAN or cross-modal GAN is selected; A dataset unit, which is used to determine the generator structure, discriminator structure, and loss function, and augment the surgical feedback dataset through a generative adversarial network.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention achieves all-round perception of the surgical scene by collecting multi-dimensional information, avoiding the limitations of a single data source. The surgical feedback feature information can explore the potential correlation between surgical operations and patient physiological responses, significantly improving the accuracy and reliability of feedback classification. Using GAN to expand the surgical feedback data set can effectively solve the problem of medical data scarcity, generate diversified training samples, improve the model's adaptability to complex surgical scenarios, reduce the risk of overfitting, avoid overfitting due to insufficient training data, and improve the generalization performance in actual surgery. By inputting real-time surgical feedback feature information into the feedback classification model, real-time classification will be automatically performed, which can realize instant analysis of key information during the operation and provide real-time decision support for surgeons. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flowchart of a real-time surgical feedback classification method.

[0016] Figure 2 Flowchart for collecting multi-dimensional information in a real-time surgical feedback classification method.

[0017] Figure 3 Flowchart for preprocessing multi-dimensional information in a real-time surgical feedback classification method.

[0018] Figure 4 A flowchart for obtaining surgical feedback feature information in a real-time surgical feedback classification method.

[0019] Figure 5 Flowchart for determining the GAN model infrastructure for a real-time surgical feedback classification approach.

[0020] Figure 6 Schematic diagram of the structure of a real-time surgical feedback classification system. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0022] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0023] like Figure 1 As shown, an embodiment of the present invention provides a real-time surgical feedback classification method, the method comprising the following steps: S100, collecting multi-dimensional information, where the multi-dimensional information includes microendoscopic data, mechanical sensor data, and physiological monitoring data; S200, preprocessing the multi-dimensional information, extracting and fusing features of the preprocessed multi-dimensional information to obtain surgical feedback feature information; S300, constructing a surgical feedback dataset based on historical surgical feedback feature information, and using a generative adversarial network to expand the surgical feedback dataset; S400, training and optimizing the SVM classification model using the surgical feedback data set to obtain a feedback classification model; S500, inputting the real-time surgical feedback feature information into a feedback classification model for real-time classification.

[0024] It should be noted that the existing surgical feedback mechanism mainly relies on the doctor's experience and judgment and postoperative pathological analysis. This method has obvious lag and cannot provide immediate guidance during the operation. In recent years, with the advancement of sensor technology, image processing technology and artificial intelligence algorithms, multi-dimensional information during the operation can be obtained, providing a rich data basis for surgical feedback. However, how to effectively process these massive and heterogeneous data, extract clinically significant features, and build an efficient and accurate classification model to achieve real-time classification and early warning of surgical feedback is still the main technical challenge currently faced. The embodiments of the present invention are intended to solve the above problems.

[0025] In the embodiment of the present invention, relevant information is collected in multiple dimensions during the operation, including microendoscopic data, mechanical sensor data, and physiological monitoring data, to achieve all-round perception of the surgical scene and avoid the limitations of a single data source. The multi-dimensional information is then automatically pre-processed, and feature extraction and fusion of the pre-processed multi-dimensional information are performed to obtain surgical feedback feature information. The surgical feedback feature information can explore the potential correlation between surgical operations and patient physiological responses, significantly improving the accuracy and reliability of feedback classification.

[0026] Next, a surgical feedback dataset will be constructed based on the historical surgical feedback feature information. The surgical feedback dataset is obtained by defining surgical classification labels based on expert experience. Then, GAN is used to expand the surgical feedback dataset, which can effectively solve the problem of medical data scarcity, generate diverse training samples, and improve the model's adaptability to complex surgical scenarios and reduce the risk of overfitting. Through data enhancement, the model can learn more universal features, avoid overfitting due to insufficient training data, and improve the generalization performance in actual surgery.

[0027] Then, the SVM classification model is trained and optimized with the surgical feedback dataset to obtain the feedback classification model. Support Vector Machine (SVM) has the advantages of high computational efficiency and clear classification boundaries when dealing with high-dimensional feature data, making it suitable for the fast classification requirements of real-time surgical feedback. The kernel function of the SVM classification model is the radial basis kernel. Bayesian optimization is used to search for C (regularization parameter) and γ. To address the class imbalance problem, a loss function is defined. Finally, the feedback classification model is optimized based on accuracy, F1 score, and ROC-AUC. Finally, when the real-time surgical feedback feature information is input into the feedback classification model, real-time classification will be automatically performed, enabling immediate analysis of key information during the surgical process and providing real-time decision support for surgeons.

[0028] As Figure 2 shown, as a preferred embodiment of the present invention, the step of collecting multi-dimensional information specifically includes: S101, collecting microscopic endoscope data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image, where the microscopic endoscope supports RGB and near-infrared dual-modal imaging; S102, collecting mechanical sensor data through a six-axis force sensor or a six-axis torque sensor, with a sampling frequency ≥ 1 kHz; S103, monitoring heart rate data, blood pressure data, blood oxygen saturation, and electroencephalogram signals, and integrating them to obtain physiological monitoring data.

[0029] Specifically, the steps of collecting microscopic endoscope data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image are as follows: Obtaining the near-infrared channel original image from the endoscope data, and determining the blood vessel reflectivity threshold based on the tissue details obtained from the endoscope data; Measuring the near-infrared light reflection intensity values of all pixel points in the near-infrared channel original image, and calculating the difference between the near-infrared light reflection intensity value and the blood vessel reflectivity threshold to obtain the reflection intensity difference; Converting the reflection intensity difference into a dynamic adjustment coefficient using an S-shaped curve, and generating a dynamic adjustment coefficient matrix for the near-infrared channel through the dynamic adjustment coefficient; Obtaining the RGB channel consecutive frame images from the endoscope data, and calculating the pixel-by-pixel grayscale value comparison between the current frame image of the RGB channel and the previous frame image of the RGB channel to obtain the brightness change rate of the pixel points in the time dimension; Normalizing the brightness change rate of the pixel points in the time dimension to generate a time motion sensitivity matrix for the RGB channel; Determining the basic fusion weight parameter from the endoscope data, and linearly superimposing the dynamic adjustment coefficient matrix of the infrared channel and the basic fusion weight parameter to obtain the dual-modal fusion weight distribution map; Determine the reference illumination intensity based on the distance between the endoscope and the tissue, and calculate the relative ratio of the brightness of each pixel in the current frame image of the RGB channels to the reference illumination intensity to generate an RGB enhanced image; Perform non-linear brightness compensation on the RGB enhanced image to generate an RGB enhanced image after illumination correction; Multiply the time motion sensitivity matrix of the RGB channels pixel by pixel with the RGB enhanced image after illumination correction to generate an RGB feature image with motion enhancement characteristics; Combine the original image of the near-infrared channel and the RGB feature image with motion enhancement characteristics, and perform regional division through a bimodal fusion weight distribution map to generate a surgical scene fusion image.

[0030] Furthermore, collect mechanical sensor data through a six-axis force sensor or a six-axis torque sensor. The specific steps are as follows: Obtain a three-dimensional force component vector and a three-dimensional torque component vector from the mechanical sensor data; Determine the weight of the three-dimensional force component vector through the three-dimensional force component vector, and weight the three-dimensional force component vector using the weight of the three-dimensional force component vector to obtain a weighted three-dimensional force component vector; Perform a square summation operation and a square root operation on the weighted three-dimensional force component vector in sequence to obtain a resultant force intensity scalar value; Determine the maximum safe force threshold based on medical research data on the force-bearing capacity of human tissues, and divide the resultant force intensity scalar value by the maximum safe force threshold to obtain a normalized force intensity index value; Determine the weight of the three-dimensional torque component vector through the three-dimensional torque component vector, and weight the three-dimensional torque component vector using the weight of the three-dimensional torque component vector to obtain a weighted three-dimensional torque component vector; Perform a square summation operation and a square root operation on the weighted three-dimensional torque component vector in sequence to obtain a resultant torque intensity scalar value; Determine the scene adaptation coefficient through the surgical stage, and multiply the resultant torque intensity scalar value by the scene adaptation coefficient to obtain a torque intensity index value optimized by the scene; Calculate the sum of the products of the components of the three-dimensional force component vector and the three-dimensional torque component vector in each direction to obtain a direction vector product; Multiply the resultant force intensity scalar value by the resultant torque intensity scalar value to obtain an intensity product, and divide the direction vector product by the intensity product to obtain an instantaneous force-torque direction similarity; Perform a time differentiation operation on the instantaneous force-torque direction similarity to obtain a force-torque co-variation rate index value; Determine the co-variation adaptive coefficient based on the characteristic information of historical surgery feedback, multiply the force-moment co-variation rate index value by the co-variation adaptive coefficient to obtain the optimized co-variation characteristic value; Perform linear superposition on the normalized force intensity index value, the scene-optimized moment intensity index value, and the optimized co-variation characteristic value to obtain the comprehensive real-time mechanical state score; Determine the mechanical sensor data based on the comprehensive real-time mechanical state score.

[0031] In the embodiments of the present invention, when collecting microscopic endoscope data, a 4K or 8K microscopic endoscope is selected, and RGB and near-infrared dual-modal imaging are supported. The 4K / 8K resolution provides ultra-high-definition visual feedback, enabling doctors to clearly observe tissue details (such as microvessels and nerve endings), reducing misjudgment and operation errors. Near-infrared imaging can penetrate the tissue surface and display deep structures (such as tumor boundaries and blood flow distributions), assisting in precise positioning and boundary recognition. The RGB mode is used to observe conventional anatomical structures and provide real color restoration. The near-infrared mode enhances the contrast of diseased tissues (such as cancer cells) through fluorescence labeling or autofluorescence technology to achieve real-time intraoperative navigation. When collecting mechanical sensor data, a six-axis force sensor or a six-axis torque sensor is selected. The six-axis force sensor can measure the force and torque (Fx, Fy, Fz, Mx, My, Mz) of the surgical instrument in three-dimensional space in real time, enabling doctors to perceive the interaction force between the instrument and the tissue.

[0032] As Figure 3 shown, as a preferred embodiment of the present invention, the step of preprocessing multi-dimensional information specifically includes: S201, extract multi-view images from the microscopic endoscope data, and register the multi-view images through the affine transformation matrix T. The relational formula in the corresponding process is: ; Among them, is the original image, is the aligned image, is the affine transformation matrix; S202, calculate the contact force F between the surgical instrument and the tissue based on the mechanical sensor data. The relational formula in the corresponding process is: ; Among them, is the mass matrix, is the damping matrix, is the stiffness matrix, is the displacement, is the time, is the processing, is the small change amount; S203. Calculate the heart rate variability index based on a sliding window and physiological monitoring data, with a window length of 30 seconds.

[0033] In the embodiments of the present invention, after obtaining the multi-view images of the endoscope, the multi-view images will be registered through the affine transformation matrix T. Affine transformation can handle scaling, rotation, and translation caused by view changes, is applicable to scenarios where the camera moves slightly or tilts, and has a certain tolerance for noise and local distortion. T needs to be calculated and determined in advance, and the determination depends on feature point matching and parameter solving. Then, the contact force F between the surgical instrument and the tissue needs to be calculated based on the mechanical sensor data. represents the first derivative of displacement with respect to time. represents the second derivative of displacement with respect to time. The mass matrix, damping matrix, and stiffness matrix all need to be determined in advance. Finally, the heart rate variability index also needs to be calculated based on the sliding window and physiological monitoring data. Here, RMSSD is calculated. RMSSD represents the root mean square of the differences between adjacent RR intervals and is a commonly used time-domain index for measuring heart rate variability. The RR interval is the time interval between two heartbeats.

[0034] Such as Figure 4 shown, as a preferred embodiment of the present invention, the step of extracting and fusing features from the preprocessed multi-dimensional information to obtain the surgical feedback feature information specifically includes: S204. Extract endoscope features. Use the pre-trained ResNet-50 or EfficientNet as the backbone network to extract features from the microscopic images and obtain the deep feature vectors. S205. Extract mechanical features. Calculate the mean and variance of the contact force between the surgical instrument and the tissue, perform a fast Fourier transform on the contact force signal, and extract the spectral energy distribution. S206. Extract physiological features. Calculate the mean and standard deviation of the heart rate, heart rate variability, and the energy of the power spectral density of the electroencephalogram signal in a set frequency band. S207. Fuse the endoscope features, mechanical features, and physiological features into a high-dimensional vector to obtain the surgical feedback feature information.

[0035] Specifically, to extract mechanical features, calculate the mean and variance of the contact force between the surgical instrument and the tissue, perform a fast Fourier transform on the contact force signal, and extract the spectral energy distribution, the specific steps are as follows: Calculate the square root of the variance of the contact force to obtain the standard deviation of the contact force, and divide the mean of the contact force by the standard deviation of the contact force to obtain the time-domain stability index value. Perform a fast Fourier transform on the contact force signal to obtain the contact force spectral distribution, divide the contact force spectral distribution into a preset number of equal-width frequency bands, and calculate the energy value of each frequency band. Dynamically weight the energy values of each frequency band using an S-curve function to obtain the weighted energy values of the frequency bands, and sum up the weighted energy values of all frequency bands to obtain the comprehensive frequency-domain energy index; Multiply the time-domain stability index value by the comprehensive frequency-domain energy index to obtain the final operating stability evaluation value; Determine the safety threshold based on the surgical stage, and compare the operating stability evaluation value with the safety threshold in real time to generate a safety warning signal; Process the contact force spectrum distribution by combining the operating stability evaluation value and the safety warning signal to generate the spectrum energy distribution.

[0036] Furthermore, fuse the endoscopic features, mechanical features, and physiological features into a high-dimensional vector. The specific steps are as follows: Use a convolutional neural network to extract the endoscopic features to obtain the independent intensity characterization values of each channel, and perform a square summation operation on the independent intensity characterization values of all channels to obtain the original energy accumulation value; Perform a square root operation on the original energy accumulation value to obtain the overall intensity value of the visual feature vector; Obtain the instantaneous change rate of the contact force through the mechanical sensor data, and after anti-zero processing of the instantaneous change rate of the contact force, add it to the overall intensity value of the visual feature vector to obtain a combined quantity; Perform a non-linear mapping on the combined quantity to generate the dynamic weight coefficient of the visual feature; Determine the maximum contact force based on the contact force spectrum energy distribution, extract the cumulative energy value within a certain frequency band according to the energy distribution to obtain the contact force cumulative energy value; Use the hyperbolic tangent function to perform a non-linear transformation on the ratio of the contact force cumulative energy value to the maximum contact force to obtain the normalized mechanical sensitivity coefficient; Perform a weighted average on the mechanical features through the sliding window integration method to obtain a standardized time-domain mechanical vector; Perform a fast Fourier transform on the contact force signal to obtain the transformed mechanical features; Obtain the resonance frequency of the surgical instrument, and divide the transformed mechanical features by the resonance frequency of the surgical instrument to obtain the characteristic frequency band; Calculate the KL divergence between the energy of each characteristic frequency band and the limit energy spectrum based on the characteristic frequency band to obtain the frequency-domain anomaly index vector; Perform a Gaussian kernel mapping on the standardized time-domain mechanical vector to obtain the time-domain mechanical component, and perform a Hadamard product operation on the time-domain mechanical component and the frequency-domain anomaly index to obtain the multi-dimensional mechanical feature vector; Obtain the brain wave power value through the brain wave signal, and multiply the heart rate variability index by the brain wave power value to obtain the physiological regulation coefficient; Normalize the mean heart rate and the standard deviation of heart rate to obtain the normalized mean heart rate and the standard deviation of heart rate, and construct a representation through the normalized mean heart rate and the standard deviation of heart rate to obtain a multi-dimensional physiological feature vector; Perform dimensionality expansion processing on the visual feature vector to obtain an expanded visual feature vector, and use the visual feature dynamic weight coefficient to adjust and scale the expanded visual feature vector to obtain a weighted visual feature vector; Use the normalized mechanical sensitivity coefficient to perform element-wise weighting on the multi-dimensional mechanical feature vector to obtain a weighted mechanical feature; Perform an element-wise product operation on the multi-dimensional physiological feature vector and the physiological regulation coefficient to obtain a weighted physiological feature vector; Connect the weighted visual feature vector, the weighted mechanical feature, and the weighted physiological feature vector to obtain a high-dimensional vector.

[0037] In the embodiment of the present invention, when performing multi-source feature extraction, first use the pre-trained ResNet-50 or EfficientNet as the backbone network to extract features from the microscopic image to obtain a deep feature vector; input: RGB image (resolution: 224×224), output: deep feature vector, and freeze the weights of the first 10 convolutional layers, and only train the last 10 layers to adapt to the surgical scenario. Then calculate the mean and variance of the contact force F between the surgical instrument and the tissue, perform a fast Fourier transform (FFT) on the contact force signal, and extract the spectral energy distribution E(f). Next, extract physiological features, including the mean and standard deviation of heart rate, heart rate variability, and the energy of the power spectral density of the electroencephalogram signal in the set frequency bands (such as α wave, β wave).

[0038] As Figure 5 shown, as a preferred embodiment of the present invention, the steps of expanding the surgical feedback data set by using a generative adversarial network specifically include: S301, determine the basic architecture of the GAN model, select DCGAN or StyleGAN for endoscopic features, select LSTM-GAN or 1D-CNN-GAN for mechanical features, and select coupled GAN or cross-modal GAN for physiological features; S302, determine the generator structure, discriminator structure, and loss function, and expand the surgical feedback data set through the generative adversarial network.

[0039] In the embodiments of the present invention, when using a generative adversarial network (GAN) for augmentation, it is necessary to select a GAN variant according to the data type. Specifically, for endoscopic features, DCGAN or StyleGAN is selected to preserve spatial details; for mechanical features, LSTM-GAN or 1D-CNN-GAN is selected to capture temporal dependencies; for physiological features, coupled GAN or cross-modal GAN is selected to jointly generate data of different modalities. Then, the generator structure, discriminator structure, and loss function are determined, and the surgical feedback dataset is augmented through the generative adversarial network. The generator structure is: fully connected layer (FC) → batch normalization (BatchNorm) → ReLU activation, and reconstructed into a vector with the same dimension as the original feature; the discriminator structure is: FC → LeakyReLU activation → Dropout (0.3).

[0040] As Figure 6 shown, the embodiments of the present invention also provide a real-time surgical feedback classification system, which includes: A multi-dimensional information acquisition module 100, configured to acquire multi-dimensional information, where the multi-dimensional information includes microscopic endoscopic data, mechanical sensor data, and physiological monitoring data; An information processing and extraction module 200, configured to preprocess the multi-dimensional information, and perform feature extraction and fusion on the preprocessed multi-dimensional information to obtain surgical feedback feature information; A feedback dataset augmentation module 300, configured to construct a surgical feedback dataset based on historical surgical feedback feature information, and augment the surgical feedback dataset using a generative adversarial network; A classification model construction module 400, configured to train and optimize an SVM classification model through the surgical feedback dataset to obtain a feedback classification model; A classification result output module 500, configured to input the real-time surgical feedback feature information into the feedback classification model for real-time classification.

[0041] As a preferred embodiment of the present invention, the multi-dimensional information acquisition module 100 includes: An endoscopic data acquisition unit, configured to acquire microscopic endoscopic data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image, and the microscopic endoscope supports RGB and near-infrared dual-modal imaging; A mechanical data acquisition unit, configured to acquire mechanical sensor data through a six-axis force sensor or a six-axis torque sensor, and the sampling frequency ≥ 1 kHz; A physiological monitoring acquisition unit, configured to monitor heart rate data, blood pressure data, blood oxygen saturation, and electroencephalogram signals, and integrate them to obtain physiological monitoring data.

[0042] As a preferred embodiment of the present invention, the information processing and extraction module 200 includes: An endoscopic image registration unit, which is used to extract multi-view images from microscopic endoscopic data and register the multi-view images through an affine transformation matrix T. The relational formula existing in the corresponding process is: ; Among them, is the original image, is the aligned image, is the affine transformation matrix; A contact force calculation unit, which is used to calculate the contact force F between the surgical instrument and the tissue based on the mechanical sensor data. The relational formula existing in the corresponding process is: ; Among them, is the mass matrix, is the damping matrix, is the stiffness matrix, is the displacement, is the time, is and processing, is the small change amount; A heart rate variability unit, which is used to calculate heart rate variability indexes based on a sliding window and physiological monitoring data, and the window length is 30 seconds.

[0043] As a preferred embodiment of the present invention, the information processing and extraction module 200 further includes: An endoscopic feature extraction unit, which is used to extract endoscopic features, uses a pre-trained ResNet-50 or EfficientNet as the backbone network to extract features from microscopic images, and obtains a deep feature vector; A mechanical feature extraction unit, which is used to extract mechanical features, calculate the mean and variance of the contact force between the surgical instrument and the tissue, perform a fast Fourier transform on the contact force signal, and extract the spectral energy distribution; A physiological feature extraction unit, which is used to extract physiological features, calculate the mean and standard deviation of heart rate, heart rate variability, and the energy of the power spectral density of the electroencephalogram signal in a set frequency band; A feature splicing and fusion unit, which is used to fuse endoscopic features, mechanical features, and physiological features into a high-dimensional vector to obtain surgical feedback feature information.

[0044] As a preferred embodiment of the present invention, the feedback data set expansion module 300 includes: A GAN architecture determination unit, which is used to determine the basic architecture of the GAN model. For endoscopic features, DCGAN or StyleGAN is selected. For mechanical features, LSTM-GAN or 1D-CNN-GAN is selected. For physiological features, coupled GAN or cross-modal GAN is selected; A dataset unit is used to determine the generator structure, discriminator structure, and loss function, and expand the surgical feedback dataset through a generative adversarial network.

[0045] The above only describes the preferred embodiments of the present invention in detail, and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention should be included within the protection scope of the present invention.

[0046] It should be understood that although the steps in the flowcharts of the embodiments of the present invention are shown in sequence according to the arrows, these steps do not necessarily have to be executed in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, at least a part of the steps in each embodiment may include multiple sub-steps or multiple stages. These sub-steps or stages do not necessarily have to be executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages does not necessarily have to be sequential, but can be executed alternately or alternately with at least a part of other steps or sub-steps or stages of other steps.

[0047] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0048] Other embodiments of the present disclosure will be readily apparent to those skilled in the art after considering the disclosure in the specification and the embodiments. This application is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

Claims

1. A real-time surgical feedback classification method, characterized in that, The method includes the following steps: Collect multi-dimensional information, where the multi-dimensional information includes microscopic endoscope data, mechanical sensor data, and physiological monitoring data; Preprocess the multi-dimensional information, and perform feature extraction and fusion on the preprocessed multi-dimensional information to obtain surgical feedback feature information; Construct a surgical feedback data set based on historical surgical feedback feature information, and use a generative adversarial network to expand the surgical feedback data set; Train and optimize an SVM classification model through the surgical feedback data set to obtain a feedback classification model; Input the real-time surgical feedback feature information into the feedback classification model for real-time classification.

2. The real-time surgical feedback classification method according to claim 1, wherein The step of collecting multi-dimensional information specifically includes: Collect microscopic endoscope data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image, where the microscopic endoscope supports RGB and near-infrared dual-modal imaging; Collect mechanical sensor data through a six-axis force sensor or a six-axis torque sensor, with a sampling frequency ≥ 1 kHz; Monitor heart rate data, blood pressure data, blood oxygen saturation, and electroencephalogram signals, and integrate them to obtain physiological monitoring data.

3. The real-time surgical feedback classification method according to claim 2, wherein Collect microscopic endoscope data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image. The specific steps are as follows: Obtain the original near-infrared channel image from the endoscope data, and determine the blood vessel reflectivity threshold according to the tissue details obtained from the endoscope data; Measure the near-infrared light reflection intensity values of all pixel points in the original near-infrared channel image, and calculate the difference between the near-infrared light reflection intensity value and the blood vessel reflectivity threshold to obtain the reflection intensity difference; Convert the reflection intensity difference into a dynamic adjustment coefficient using an S-shaped curve, and generate a dynamic adjustment coefficient matrix for the near-infrared channel through the dynamic adjustment coefficient; Obtain the continuous frame images of the RGB channel from the endoscope data, and perform pixel-by-pixel grayscale value comparison calculation on the current frame image of the RGB channel and the previous frame image of the RGB channel to obtain the brightness change rate of the pixel points in the time dimension; Normalize the brightness change rate of the pixel points in the time dimension to generate a time motion sensitivity matrix for the RGB channel; Determine the basic fusion weight parameters from the endoscope data, and linearly superimpose the dynamic adjustment coefficient matrix of the infrared channel and the basic fusion weight parameters to obtain a dual-modal fusion weight distribution map; Determine the reference illumination intensity based on the distance between the endoscope and the tissue, and calculate the relative ratio of the brightness of each pixel in the current frame image of the RGB channel to the reference illumination intensity to generate an RGB enhanced image; Perform non-linear brightness compensation on the RGB enhanced image to generate an RGB enhanced image after illumination correction; Multiply the time motion sensitivity matrix of the RGB channel and the RGB enhanced image after illumination correction pixel by pixel to generate an RGB feature image with motion enhancement characteristics; Combine the original near-infrared channel image and the RGB feature image with motion enhancement characteristics, and perform regional division through the dual-modal fusion weight distribution map to generate a surgical scene fusion image.

4. The real-time surgical feedback classification method according to claim 3, wherein, Collect mechanical sensor data through a six-axis force sensor or a six-axis torque sensor. The specific steps are as follows: Obtain the three-dimensional force component vector and the three-dimensional moment component vector from the mechanical sensor data; Determine the weight of the three-dimensional force component vector based on the three-dimensional force component vector, and weight the three-dimensional force component vector using the weight of the three-dimensional force component vector to obtain the weighted three-dimensional force component vector; Perform the square summation operation and the square root operation on the weighted three-dimensional force component vector in sequence to obtain the resultant force intensity scalar value; Determine the maximum safe force threshold based on the medical research data of human tissue bearing force, and divide the resultant force intensity scalar value by the maximum safe force threshold to obtain the normalized force intensity index value; Determine the weight of the three-dimensional moment component vector based on the three-dimensional moment component vector, and weight the three-dimensional moment component vector using the weight of the three-dimensional moment component vector to obtain the weighted three-dimensional moment component vector; Perform the square summation operation and the square root operation on the weighted three-dimensional moment component vector in sequence to obtain the resultant moment intensity scalar value; Determine the scene adaptation coefficient based on the surgical stage, and multiply the resultant moment intensity scalar value by the scene adaptation coefficient to obtain the scene-optimized moment intensity index value; Calculate the sum of the products of the components of the three-dimensional force component vector and the three-dimensional moment component vector in each direction to obtain the direction vector product; Multiply the resultant force intensity scalar value by the resultant moment intensity scalar value to obtain the intensity product, and divide the direction vector product by the intensity product to obtain the instantaneous force-moment direction similarity; Perform a time differential operation on the instantaneous force-moment direction similarity to obtain the force-moment co-variation rate index value; Determine the co-variation adaptation coefficient based on the historical surgical feedback feature information, and multiply the force-moment co-variation rate index value by the co-variation adaptation coefficient to obtain the optimized co-variation feature value; Perform a linear superposition on the normalized force intensity index value, the scene-optimized moment intensity index value, and the optimized co-variation feature value to obtain the comprehensive real-time mechanical state score; Determine the mechanical sensor data through the comprehensive real-time mechanical state score assistance.

5. The real-time surgical feedback classification method according to claim 4, wherein The steps of preprocessing the multi-dimensional information specifically include: Extract the multi-view images in the microscopic endoscope data, and register the multi-view images through the affine transformation matrix T. The relational expression existing in the corresponding process is: ； Among them, is the original image, is the aligned image, is the affine transformation matrix; Calculate the contact force F between the surgical instrument and the tissue based on the mechanical sensor data. The relational expression existing in the corresponding process is: ； in, is the mass matrix, is the damping matrix, is the stiffness matrix, is the displacement, For time, For and processing, For small changes; Calculate the heart rate variability index based on the sliding window and the physiological monitoring data, and the window length is 30 seconds.

6. The real-time surgical feedback classification method according to claim 5, characterized in that The steps of extracting features and fusing the preprocessed multi-dimensional information to obtain the surgical feedback feature information specifically include: Extract the endoscope features, use the pre-trained ResNet-50 or EfficientNet as the backbone network to extract features from the microscopic images, and obtain the deep feature vectors; Extract the mechanical features, calculate the mean and variance of the contact force between the surgical instrument and the tissue, perform a fast Fourier transform on the contact force signal, and extract the spectral energy distribution; Extract the physiological features, calculate the mean and standard deviation of the heart rate, the heart rate variability, and the energy of the power spectral density of the electroencephalogram signal in the set frequency band; Fuse the endoscope features, mechanical features, and physiological features into a high-dimensional vector to obtain the surgical feedback feature information.

7. The real-time surgical feedback classification method according to claim 6, wherein Extract mechanical features, calculate the mean and variance of the contact force between the surgical instrument and the tissue, perform a fast Fourier transform on the contact force signal, and extract the spectral energy distribution. The specific steps are as follows: Calculate the square root of the variance of the contact force to obtain the standard deviation of the contact force, and divide the mean of the contact force by the standard deviation of the contact force to obtain the time-domain stability index value; Perform a fast Fourier transform on the contact force signal to obtain the contact force spectrum distribution, divide the contact force spectrum distribution into a preset number of equal-width frequency bands, and calculate the energy value of each frequency band; Dynamically weight the energy value of each frequency band using an S-shaped curve function to obtain the weighted energy value of the frequency band, and sum the weighted energy values of all frequency bands to obtain the comprehensive frequency-domain energy index; Multiply the time-domain stability index value by the comprehensive frequency-domain energy index to obtain the final operation stability evaluation value; Determine the safety threshold based on the surgical stage, and compare the operation stability evaluation value with the safety threshold in real time to generate a safety warning signal; Process the contact force spectrum distribution by combining the operation stability evaluation value and the safety warning signal to generate the spectral energy distribution; 8. The real-time surgical feedback classification method according to claim 7, characterized in that Fuse the endoscopic features, mechanical features, and physiological features into a high-dimensional vector. The specific steps are as follows: Use a convolutional neural network to extract endoscopic features to obtain the independent intensity representation values of each channel, and perform a square sum operation on the independent intensity representation values of all channels to obtain the original energy accumulation value; Perform a square root operation on the original energy accumulation value to obtain the overall intensity value of the visual feature vector; Obtain the instantaneous change rate of the contact force through the mechanical sensor data, and after anti-zero processing of the instantaneous change rate of the contact force, add it to the overall intensity value of the visual feature vector to obtain a combined quantity; Perform a non-linear mapping on the combined quantity to generate a visual feature dynamic weight coefficient; Determine the maximum contact force based on the contact force spectrum energy distribution, extract the cumulative energy value within a certain frequency band according to the energy distribution to obtain the contact force cumulative energy value; Perform a non-linear transformation on the ratio of the contact force cumulative energy value to the maximum contact force using the hyperbolic tangent function to obtain the normalized mechanical sensitivity coefficient; Perform a weighted average on the mechanical features through the sliding window integration method to obtain a standardized time-domain mechanical vector; Perform a fast Fourier transform on the contact force signal to obtain the transformed mechanical features; Obtain the resonance frequency of the surgical instrument, and divide the transformed mechanical features by the resonance frequency of the surgical instrument to obtain the characteristic frequency band; Calculate the KL divergence between the energy of each characteristic frequency band and the limit energy spectrum based on the characteristic frequency band to obtain the frequency-domain anomaly index vector; Perform a Gaussian kernel mapping on the standardized time-domain mechanical vector to obtain the time-domain mechanical component, and perform a Hadamard product operation on the time-domain mechanical component and the frequency-domain anomaly index to obtain the multi-dimensional mechanical feature vector; Obtain the brain wave power value through the brain wave signal, and multiply the heart rate variability index by the brain wave power value to obtain the physiological regulation coefficient; Perform a standardized process on the heart rate mean and heart rate standard deviation to obtain the standardized heart rate mean and heart rate standard deviation, and construct a representation through the standardized heart rate mean and heart rate standard deviation to obtain the multi-dimensional physiological feature vector; Perform dimensionality expansion processing on the visual feature vector to obtain an expanded visual feature vector, and use the visual feature dynamic weight coefficient to adjust and scale the expanded visual feature vector to obtain a weighted visual feature vector; Perform element-wise weighting on the multi-dimensional mechanical feature vector using the normalized mechanical sensitivity coefficient to obtain a weighted mechanical feature; Perform an element-wise product operation on the multi-dimensional physiological feature vector and the physiological regulation coefficient to obtain a weighted physiological feature vector; Connect the weighted visual feature vector, the weighted mechanical feature, and the weighted physiological feature vector to obtain a high-dimensional vector.

9. The real-time surgical feedback classification method according to claim 8, characterized in that, The steps of expanding the surgical feedback data set using a generative adversarial network specifically include: Determine the basic architecture of the GAN model. For endoscopic features, choose DCGAN or StyleGAN; for mechanical features, choose LSTM-GAN or 1D-CNN-GAN; for physiological features, choose coupled GAN or cross-modal GAN; Determine the generator structure, discriminator structure, and loss function, and expand the surgical feedback data set through the generative adversarial network.

10. A real-time surgical feedback classification system, characterized in that, The system applies the real-time surgical feedback classification method described in any one of claims 1 to 9 above. The system includes: A multi-dimensional information acquisition module for acquiring multi-dimensional information, where the multi-dimensional information includes microscopic endoscopic data, mechanical sensor data, and physiological monitoring data; An information processing and extraction module for preprocessing the multi-dimensional information, and performing feature extraction and fusion on the preprocessed multi-dimensional information to obtain surgical feedback feature information; A feedback data set expansion module for constructing a surgical feedback data set based on historical surgical feedback feature information, and expanding the surgical feedback data set using a generative adversarial network; A classification model construction module for training and optimizing an SVM classification model through the surgical feedback data set to obtain a feedback classification model; A classification result output module for inputting the real-time surgical feedback feature information into the feedback classification model for real-time classification.

11. The real-time surgical feedback classification system according to claim 10, wherein The multi-dimensional information acquisition module includes: An endoscopic data acquisition unit for acquiring microscopic endoscopic data through a 4K or 8K microscopic endoscope to generate a surgical scene fusion image, and the microscopic endoscope supports RGB and near-infrared dual-modal imaging; A mechanical data acquisition unit for acquiring mechanical sensor data through a six-axis force sensor or a six-axis torque sensor, with a sampling frequency ≥ 1 kHz; A physiological monitoring acquisition unit for monitoring heart rate data, blood pressure data, blood oxygen saturation, and electroencephalogram signals, and integrating them to obtain physiological monitoring data.

Images