Force feedback monitoring method, medium and system for multi-sensor data fusion based on deep learning

Abstract

The invention relates to a robot-assisted surgery system, in particular to a force feedback monitoring method, medium and system for multi-sensor data fusion based on deep learning, and the method comprises the following steps: obtaining the magnitude and direction of the force applied by a tool, and the acceleration change during vibration and movement; preprocessing the data and performing feature extraction; inputting the extracted spatial features into a deep learning model for time sequence processing; judging a current stage state according to a result obtained by processing the deep learning model, and adjusting a transfer correction coefficient; and carrying out force feedback on the corrected force feedback result. Compared with the prior art, the problem that in the prior art, the touch sense of a doctor cannot be simulated during fine operation, so that surgical operation only depends on experience and visual speculation of the doctor is solved. According to the scheme, in combination with multi-sensor data, micro force changes in an operation are processed and analyzed in real time through a deep learning algorithm, and therefore the accuracy and safety of the system are improved.

Description

Technical Field

[0001] The present invention relates to a robot-assisted surgical system, and particularly to a force feedback monitoring method, medium and system based on multi-sensor data fusion of deep learning. Background Art

[0002] In current robot-assisted surgical systems, the lack of force feedback technology remains a key technical bottleneck. Although existing robot systems have improved the surgical precision to a certain extent through visual assistance and path planning, they lack real-time perception of the operating force. Especially during minimally invasive surgery, there are still significant risks. In delicate operations such as suturing blood vessels or cutting tissues, doctors cannot directly perceive the minute force changes generated when the tool contacts the tissue, which may lead to excessive operation, accidental injury, or affect the surgical effect due to insufficient force. Although existing robot systems can rely on visual information for auxiliary positioning and path adjustment, these visual feedbacks cannot reflect the mechanical changes during the operation in real time and directly. Therefore, doctors' perception of the minute force during the operation is very limited. The lack of force feedback makes the robot system unable to precisely control the force during the operation, thereby affecting the precision and safety of the surgery.

[0003] Most existing robot-assisted surgical systems rely on visual feedback for task execution, but this method cannot provide mechanical perception. Especially in minimally invasive surgery, especially for operations that require precise force control, such as suturing or cutting, it is difficult for doctors to perceive the force changes through touch when operating with the robot-assisted surgical system. This causes doctors to usually rely only on past experience and visual performance to speculate on the operation results, and cannot provide real-time feedback on whether the applied force is appropriate. At the same time, due to the lack of force feedback, the precision and safety of the surgery are severely affected. Especially in complex minimally invasive surgeries, the operation risks are significantly increased.

[0004] Although there are some force sensors in the prior art to monitor the operating force, the accuracy and response speed of these sensors cannot meet the high requirements in robot surgery. Most current force sensors can only detect large-amplitude force changes and usually cannot provide the real-time precise feedback required in a complex surgical environment. In addition, most existing force feedback technologies rely on a single type of sensor, resulting in limited feedback accuracy and stability. The signal differences between different types of sensors may lead to errors in data fusion, thereby affecting the accuracy of force feedback. Therefore, the existing force sensors and simple feedback mechanisms cannot provide sufficient accuracy to ensure the safety of surgical operations. Especially when dealing with complex tissues or performing delicate operations, they cannot simulate the doctor's touch and limit the performance of the robot system.

[0005] As disclosed in CN109620414A, a force feedback method and system for a surgical robotic gripper are provided. The method includes: recording the pressure information of the fingertips, the torque information of the finger joints, and the position information of the wrist during the doctor's surgery; integrating the pressure information and the torque information to generate a surgical force model according to the time line; integrating the position information to generate a surgical motion trajectory model according to the time line; integrating the surgical force model and the surgical motion trajectory model to generate a force application system model for the doctor's surgical actions; guiding the operation process of the same type of surgery according to the force application system model of the doctor's surgical actions, and supervising and correcting the operation process of the surgical actions with visual images and ultrasonic images. This solution records the pressure, torque, and position information generated by the doctor during the surgery, and generates surgical force and motion trajectory models based on these data. The core of this model is to model the surgical actions of individual doctors and guide and correct the surgical operation process based on this model. However, this modeling method based on individual doctor's motion data has obvious limitations: the actions of different doctors vary greatly during the surgery, and each doctor has unique operating habits and force preferences. Although the system can generate corresponding feedback for specific doctors, its universality is poor and it cannot effectively meet the needs of different doctors. Therefore, the application scope of the system is limited and it cannot be effectively promoted and applied across doctors.

[0006] Therefore, developing a force feedback method based on multi-sensor signal fusion is of great clinical significance for improving the accuracy and safety of robot-assisted surgery. Summary of the Invention

[0007] The object of the present invention is to provide a force feedback monitoring method, medium and system based on deep learning for multi-sensor data fusion to solve at least one of the above problems, so as to solve the problem that the force sensor and simple feedback mechanism in the prior art cannot simulate the doctor's touch during fine operations, resulting in relying only on the doctor's experience and visual speculation for judgment during surgical operations. This solution combines the data of multiple sensors and processes and analyzes the minute mechanical changes during the surgery through deep learning algorithms, thereby improving the accuracy and safety of the robot-assisted surgery system.

[0008] The object of the present invention is achieved by the following technical solutions:

[0009] The first aspect of the present invention discloses a force feedback monitoring method based on deep learning for multi-sensor data fusion, which is used for a robot-assisted surgery system to transmit real-time force feedback to a doctor, and includes the following steps:

[0010] S1: Obtain the magnitude and direction of the force applied by the tool through a force sensor, obtain the vibration of the tool through a vibration sensor, and obtain the acceleration change of the tool during movement through an acceleration sensor;

[0011] S2: Preprocess the acquired data and extract features;

[0012] S3: Input the extracted spatial features into a deep learning model for time series processing to obtain a force feedback result and predict the force change trend;

[0013] S4: Based on the force feedback result and the force change trend prediction result obtained by processing with the deep learning model, determine the current stage state, and adjust the transfer correction coefficient of the force feedback result according to the different stage states reached;

[0014] S5: Provide real-time force feedback of the corrected force feedback result to the doctor.

[0015] Preferably, in step S1, one or both of the following are included:

[0016] i) The force sensor is a strain type force sensor or a piezoelectric force sensor;

[0017] ii) The vibration sensor is a piezoelectric sensor.

[0018] Preferably, in step S2, the preprocessing includes normalization and filtering.

[0019] Preferably, the filtering includes low-pass filtering and median filtering.

[0020] Preferably, in step S2, the feature extraction is performed by a convolutional neural network.

[0021] Preferably, in step S3, the deep learning model is a long short-term memory network.

[0022] Preferably, in step S3, the deep learning model is trained by supervised learning and optimized by a multi-task loss function.

[0023] Preferably, in step S5, the real-time force feedback is vibration feedback and / or pressure simulation feedback.

[0024] A second aspect of the present invention discloses a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, it controls the device where the computer-readable storage medium is located to execute the method described in any one of the above.

[0025] A third aspect of the present invention discloses a force feedback monitoring system based on multi-sensor data fusion of deep learning for implementing the method described in any one of the above;

[0026] The system includes a force sensor, a vibration sensor, an acceleration sensor, a central processing unit, and a force feedback device;

[0027] The force sensor, vibration sensor and acceleration sensor are respectively installed on the tool or the robot-assisted surgery system to obtain the magnitude and direction of the force applied by the tool, the vibration of the tool and the acceleration change when the tool moves;

[0028] The central processing unit is in communication connection with the force sensor, the vibration sensor, the acceleration sensor and the force feedback device, and the central processing unit is used to receive data acquired by the force sensor, the vibration sensor and the acceleration sensor, and then process the data to obtain a corrected force feedback result;

[0029] The force feedback device feeds back the corrected force feedback result to the doctor in real time.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] (1) Existing technologies rely on a single sensor, and feedback information is incomplete: Existing force feedback systems mainly rely on a single sensor (such as a single force sensor) for feedback and cannot provide multi-dimensional perception of force.

[0032] In contrast, the present invention achieves multi-dimensional mechanical perception by combining force sensors, vibration sensors and acceleration sensors. This technology can comprehensively consider factors such as force size, direction, frequency and acceleration, and provide more comprehensive and accurate real-time feedback, thereby improving the safety and accuracy of surgical operations.

[0033] (2) Existing technologies lack real-time perception of tiny force changes: Traditional robotic surgical systems are usually unable to accurately perceive tiny force changes in real time, resulting in excessive cutting or improper operation.

[0034] The present invention combines the deep learning algorithm with the spatiotemporal feature extraction of CNN and LSTM to analyze the changing trend of force signals in real time and capture the subtle mechanical changes during surgery. Through this deep learning model, the system can identify small fluctuations in force in real time and provide timely feedback to doctors, solving the problem of insufficient perception of small force changes in existing technologies.

[0035] (3) Existing force feedback technology has slow response speed and poor accuracy: Traditional force feedback systems have major deficiencies in response speed and feedback accuracy, resulting in delayed or inaccurate feedback information.

[0036] The present invention significantly improves the response speed of the system through efficient deep learning models and fast sensor data processing, so that feedback information can be delivered to doctors in a timely manner within milliseconds. At the same time, the intelligent processing of deep learning can eliminate data differences between sensors, improve the accuracy of feedback, and solve the problems of slow response and poor feedback accuracy in the existing technology.

[0037] Among them, the key to fast sensor data processing lies in the close connection of each link:

[0038] First, multiple sensors are used to achieve real-time synchronous acquisition of data. The collected data can be instantly transmitted to the central processing unit through wireless transmission, which greatly reduces the delay in the data transmission process.

[0039] Secondly, in the data preprocessing stage, the original data collected by each sensor undergoes standardization processing to eliminate the dimensional differences between different sensors, ensuring that the data is processed on a unified scale. At the same time, the system uses low-pass filtering and median filtering, and both of these filtering methods can complete the calculation in a short time.

[0040] Subsequently, the preprocessed data enters an optimized convolutional neural network (CNN). Here, the multi-scale convolutional kernel design can efficiently capture the subtle changes and overall trends in the sensor data and quickly extract the key spatial features. The extracted features are then passed into the long short-term memory network (LSTM). The LSTM can efficiently process the time-series data, timely capture the dynamic changes of the signal, and complete the prediction. The entire process from data acquisition to the generation of the prediction result only requires a millisecond-level response.

[0041] Therefore, from real-time wireless transmission, multi-sensor synchronous acquisition, to data standardization and high-speed filtering, and then to the CNN and LSTM processing in the deep learning model, each link is carefully designed and optimized. This overall solution not only greatly improves the response speed but also ensures the accuracy of the feedback information.

[0042] (4) In the case where the prior art only relies on visual feedback, there is a lack of mechanical perception: Currently, robot-assisted surgical systems generally rely on visual feedback for positioning and operation, but this method cannot provide direct mechanical feedback, especially in complex surgeries, the applied force cannot be sensed in real time.

[0043] The present invention provides real-time feedback of force by combining multi-sensor data and deep learning algorithms, not only relying on visual information but also being able to reflect the change of force. Doctors can adjust the operation force according to the real-time feedback to avoid excessive cutting or accidental injury, improving the precision and safety of surgical operations.

[0044] (5) The prior art highly relies on the surgical process of specific doctors and lacks universality: For example, the solution disclosed in CN109620414A models the surgical actions of individual doctors and guides and corrects the surgical operation process based on this model; however, this method lacks universality for the surgical habits of different doctors.

[0045] The solution proposed in this application introduces multi-sensor data fusion technology and combines deep learning algorithms to process and analyze data collected from multiple sensors (such as force, pressure, displacement, etc.). Through this method, the system can comprehensively and accurately capture various mechanical information during the operation, and solve the problems of data accuracy and real-time performance that may exist in a single sensor through data fusion technology. Therefore, the solution of this application can provide more stable and reliable force feedback monitoring.

[0046] In addition, the application of deep learning algorithms enables this application to adaptively adjust according to the individual operation habits of different doctors through training models, thereby generating personalized surgical operation models. This forms a sharp contrast with the publicly disclosed solutions in the prior art, which only rely on the operation data of individual doctors and cannot be intelligently adapted to the differences among doctors. Through deep learning technology, this application can automatically optimize the model when facing different doctors, improving the universality and accuracy of the system.

[0047] Generally speaking, the advantages of this application not only lie in the fusion of multi-sensor data and the application of deep learning, enabling the system to provide accurate feedback in a wider range of scenarios, but also have stronger adaptability in dealing with individual differences among doctors. In contrast, the solutions of the prior art are relatively limited in dealing with the operation differences among doctors and are difficult to achieve intelligent and personalized adjustments. Brief Description of the Drawings

[0048] Figure 1 is a schematic flow chart of the force feedback monitoring method of the present invention;

[0049] Figure 2 is a schematic diagram of the LSTM network architecture in the force feedback monitoring method of the present invention. Detailed Description of the Embodiment

[0050] The present invention will be described in detail below with reference to the drawings and specific embodiments.

[0051] In the following description, if not otherwise specified, the devices used can be existing commercially available products that meet the functions, and other matters not covered can all adopt the prior art.

[0052] Embodiment 1

[0053] The technical solution of the present invention is as Figure 1 shown, and it is specifically as follows:

[0054] a. Multi - sensor data fusion system: First, the present invention constructs a system integrating multiple sensors, including force sensors, vibration sensors, and acceleration sensors, etc. The force sensor is used to monitor the magnitude and direction of the force generated when the tool (the tool at the end of the robotic - assisted surgical system for operation, hereinafter simply referred to as the tool) contacts the tissue. The vibration sensor is used to capture the tiny vibration signals that may be generated during the contact between the tool and the tissue, while the acceleration sensor is used to monitor the acceleration changes when the tool is moving. The data of these sensors are transmitted to the central processing unit in real - time through wireless transmission to ensure that all signals can be synchronously collected.

[0055] b. Application of deep - learning algorithms: The present invention uses a deep - learning model combining convolutional neural network (CNN) and long short - term memory network (LSTM) to process the data collected by the sensors. CNN is used to extract spatial features from different sensors, and effectively identify the force change patterns and the contact situation between the tool and the tissue through convolutional layers; while LSTM is used to process time - series data and can capture the time dynamics of force changes. After the combination of the two, it can comprehensively consider the data in spatial and temporal dimensions and provide more accurate mechanical feedback.

[0056] c. Implementation of real - time feedback mechanism: The data processed by the deep - learning model will be converted into force - feedback information in real - time and transmitted to the doctor through a haptic feedback device or a virtual - reality display screen. Specifically, the force - feedback device adopted can be transmitted to the handle operated by the doctor or the control interface of the surgical robot through various methods such as current signals, vibration feedback, and pressure simulation. According to the real - time feedback, the doctor can accurately perceive the force exerted by the current surgical tool, and thus adjust the operation force to ensure that situations such as excessive cutting or tissue damage will not occur.

[0057] d. Multi - dimensional mechanical change monitoring and optimization: The present invention monitors the multi - dimensional changes of force during the surgical process through multi - sensors in real - time, including cutting force, compression force, shear force, etc. In practical applications, the system will judge the mechanical requirements of the current surgical stage according to the data fed back by the sensors and dynamically adjust the feedback intensity and type. For example, when performing delicate blood - vessel suturing, the system will provide more refined force feedback to ensure that the force is not too large to cause bleeding or too small to result in operation failure.

[0058] This force - feedback monitoring method and system are used in a robotic - assisted surgical system to provide real - time force feedback to the doctor. More specifically:

[0059] According to the technical solution of the present invention, the implementation process mainly includes five core steps: multi - sensor data acquisition, data pre - processing and feature extraction, deep - learning model processing, force - feedback output, and multi - dimensional mechanical monitoring and dynamic adjustment. The implementation methods of each step will be described in detail below.

[0060] Step a, multi-sensor data acquisition:

[0061] In the implementation process of the present invention, data acquisition needs to be carried out first through a multi-sensor combination to comprehensively monitor the changes in the mechanical state during the operation. The multi-sensor combination mainly includes a force sensor, a vibration sensor, and an acceleration sensor. Each sensor is responsible for collecting different types of physical quantities and transmitting the collected signals to the central processing unit for subsequent data processing and deep learning model analysis.

[0062] The force sensor is mainly used to monitor the mechanical changes when the tool contacts the tissue. Especially during minimally invasive surgery, accurately measuring the force applied to the tissue is crucial. Common force sensors include strain gauge force sensors and piezoelectric force sensors. The working principle of the strain gauge force sensor is based on the resistance change of the strain gauge. When an external force acts on the sensor, the sensitive element of the sensor will deform, resulting in a change in resistance, which is then converted into an electrical signal. This electrical signal is proportional to the applied external force and can be described by the following formula:

[0063] V out = G·F·∈

[0064] where G is the sensitivity of the sensor, F is the applied external force, and ∈ is the strain value. The output signal of the force sensor is transmitted to the central processing unit after analog-to-digital conversion and serves as the basis for subsequent signal processing. This signal can reflect the magnitude and direction changes of the force when the robotic surgical tool contacts the tissue during the operation. Since the change in force directly affects the safety and accuracy of the operation, accurate force sensor data is crucial for real-time force feedback.

[0065] The vibration sensor is used to monitor the minute vibrations generated when the tool contacts the tissue. By sensing these vibration signals, the system can identify potential problems during the operation, such as abnormal situations like excessive cutting or accidental injury. The vibration sensor often adopts the piezoelectric effect principle. When the sensor deforms, charges will be generated at both ends, and the change in the amount of charge is proportional to the vibration intensity. The output signal of the vibration sensor can be expressed by the following formula:

[0066] V vib = k·A vib

[0067] where k is the sensitivity of the vibration sensor, and A vib is the vibration amplitude. The vibration signal is also converted into a digital signal through an ADC (analog-to-digital converter) and then transmitted to the central processing unit. The real-time acquisition of the vibration signal can provide additional information for subsequent force feedback. Especially during the process of tissue shearing or stretching, the change in the vibration signal can reflect potential operation abnormalities.

[0068] Acceleration sensors are used to monitor the acceleration changes of the tool, especially the instantaneous acceleration or deceleration that may occur during the movement of robotic surgical tools. Through the acceleration sensors, the inertial forces and dynamic changes during the operation can be sensed, which is crucial for timely adjusting the operation strategy and preventing operations that are too fast or too slow. The signals output by the acceleration sensors are also converted from analog to digital and then transmitted to the central processing unit for real-time analysis of the stability and accuracy of the robot's movement.

[0069] The output signals of all sensors are transmitted to the central processing unit in real time through a wireless transmission module. In the central processing unit, the signals will undergo preprocessing steps such as denoising, filtering, and standardization to ensure the stability and consistency of the signals. The processed sensor data will provide the necessary input for the subsequent deep learning model for further analysis and optimization of the mechanical feedback during the surgery.

[0070] Through this way of multi-sensor collaborative work, the system can comprehensively collect various mechanical signals interacting with the tissue and provide accurate data support for subsequent real-time feedback, thus ensuring the precise control of force during the surgery and reducing the occurrence of risks and misoperations.

[0071] Step b, data preprocessing and feature extraction:

[0072] After the sensor data is collected, it first needs to be preprocessed to ensure the quality and consistency of the data. Since the sensors may be affected by environmental noise, hardware errors, or other external factors, the raw data often contains noise and unstable components, which will interfere with subsequent analysis and modeling. Therefore, preprocessing is a key step to ensure that the subsequent deep learning model can effectively process the data. For the data of force sensors, vibration sensors, and acceleration sensors, we have adopted various processing methods such as denoising, filtering, and standardization.

[0073] During the data preprocessing process, the data is first standardized. The purpose of standardization is to eliminate the dimensional differences between the data of different sensors, so that the data of each sensor has the same mean and variance. Specifically, for the raw data F of the force sensor raw , we calculate the mean μ and standard deviation σ of this data and use the following formula to standardize the data:

[0074]

[0075] In this way, the data after standardization processing will have zero mean and unit variance, ensuring that the data is on the same scale.

[0076] The data of vibration sensors and acceleration sensors are also standardized using similar methods to ensure their comparability in subsequent analysis.

[0077] To remove the noise in the signal, we perform filtering on the sensor data. The purpose of filtering is to remove high-frequency noise or transient interference in the sensor data, ensure the smoothness of the signal, and improve the data quality. In the present invention, we first use a low-pass filter (such as a Butterworth filter) to remove the high-frequency noise in the signal, and then use median filtering to remove the sudden impulse noise.

[0078] The function of the low-pass filter is to allow signals with frequencies lower than the cut-off frequency to pass through while suppressing high-frequency signals, thereby smoothing the signal changes. In the present invention, we choose to use a Butterworth low-pass filter, and its transfer function is defined as:

[0079]

[0080] where H(s) is the transfer function of the filter, s is the complex frequency, ω c is the cut-off frequency of the filter, and n is the order of the filter. This formula describes how the Butterworth filter maintains a flat frequency response near the cut-off frequency and gradually attenuates the signal in the region above the cut-off frequency. By setting appropriate ω c and n, we can adjust the suppression effect of the filter on high-frequency noise to ensure that only low-frequency signals are retained and high-frequency noise is removed.

[0081] For discrete-time signals, we use a difference equation to implement low-pass filtering. The recursive filtering formula is adopted:

[0082] y(t) = αx(t) + (1 - α)y(t - 1)

[0083] where x(t) is the input signal at the current moment, y(t) is the output signal at the current moment, and α is the smoothing factor of the filter, which controls the ratio of the current input signal to the historical signal. The value of α is usually between 0 and 1. The larger the value, the greater the influence of the current signal on the output.

[0084] Median filtering is another commonly used denoising method. It removes sudden noise by replacing the current data point with the median value of the data within the window, especially suitable for removing salt-and-pepper noise in sensors. We use a window of size 2k + 1 for filtering. For each data point x(i), the operation of median filtering is as follows:

[0085] x filtered (i) = median(x i-k , x i-k+1 ,..., x i+k )

[0086] where x filteredis the filtered data, and median represents taking the median value of the data within the window. This method is particularly suitable for removing sudden noise data, retaining the main trend of the signal, and not overly affecting the smoothness of the signal.

[0087] By combining these two filtering methods, we can effectively remove the noise in the sensor data. The low-pass filter first removes the high-frequency signal noise, and the median filter further removes the sudden impulse noise, ultimately making the signal more stable and retaining the main trend and characteristics of the data. After this preprocessing, the quality of the signal is greatly improved, providing a more reliable data input for subsequent feature extraction and the training of deep learning models.

[0088] The feature extraction part adopts an improved convolutional neural network (CNN) architecture to efficiently extract key features from multi-sensor data. This network architecture extracts local and global features in the sensor data through multi-scale convolutional kernels, thereby enhancing the processing ability for mechanical data. Especially in complex surgical tasks, it can accurately capture the dynamic processes of different mechanical changes.

[0089] First, after being normalized and filtered, the sensor data is fed into the CNN model as input data. The dimension of the input data is N×T×C, where N represents the number of samples, T represents the time step, and C is the number of sensors. To better extract the spatial features in the signal, the improved CNN network adopts a multi-scale convolutional kernel structure, which can effectively capture different scale features in the signal, especially the local changes and global trends in the data of force sensors, vibration sensors, and acceleration sensors, etc.

[0090] In the design of the convolutional layer, we use convolutional kernels of sizes 3×3, 5×5, and 7×7 to extract local features in the signal in different scale ways. Convolutional kernels of different sizes can help the model identify trends from subtle changes to larger ranges. For example, for force sensor data, smaller convolutional kernels can capture subtle force changes, while larger convolutional kernels can extract more extensive mechanical patterns.

[0091] In the convolution operation, assuming the input signal is X, after convolution with convolutional kernels of different sizes, the resulting output feature map can be expressed as:

[0092] Z1 = Conv2D(X, filters = 32, kernel_size = 3, strides = 1, padding ='same')

[0093] Z2 = Conv2D(X, filters = 32, kernel_size = 5, strides = 1, padding ='same')

[0094] Z3 = Conv2D(X, filters=32, kernel_size=7, strides=1, padding='same')

[0095] Then, the feature maps extracted by these different convolutional kernels are concatenated together to form a multi-scale feature map:

[0096] Z multi_scale = concat(Z1, Z2, Z3)

[0097] This feature map integrates signal information from different scales and provides richer features for subsequent processing.

[0098] Next, after the convolutional layer, we use the max pooling (MaxPooling) operation to reduce the dimension of the convolutional feature map, reducing the data dimension while retaining the most significant features. The size of the pooling kernel is set to 2×2 and the stride is 2, thus reducing the size of the feature map by half, reducing the computational amount while retaining the most important information.

[0099] After multiple convolutional and pooling operations, the resulting feature map will enter the fully connected layer for more high-level feature learning and combination. At this stage, we will flatten the multi-dimensional features obtained after convolution and pooling through the fully connected layer and output them as a one-dimensional vector. The calculation formula of the fully connected layer is:

[0100] Z fc = W·X fc + b

[0101] where W is the weight matrix of the fully connected layer, X fc is the flattened input data, b is the bias term, and Z fc is the output of the fully connected layer.

[0102] To improve the expressiveness of the network and avoid overfitting, the output of the fully connected layer usually undergoes a non-linear transformation through an activation function (such as ReLU):

[0103] A fc = ReLU(Z fc )

[0104] At this time, the features after the convolutional and pooling layers already contain the key information in the sensor data, but these features are still spatial features. To further capture the dynamic changes in the time series, the next step is to feed the extracted spatial features into a long short-term memory network (LSTM) for processing.

[0105] Step c, deep learning model processing:

[0106] In the present invention, the feature extraction part has extracted rich spatial features from the sensor data through a Convolutional Neural Network (CNN), and the subsequent steps further perform time series processing on these features through a Long Short-Term Memory Network (LSTM) to capture the temporal dynamics of the signal.

[0107] LSTM, as a powerful Recurrent Neural Network (RNN), can effectively learn and remember long-term sequence dependencies through its internal gating mechanism, and is therefore particularly suitable for processing time-related data, such as the change of mechanical signals over time.

[0108] After the feature extraction by the CNN, the obtained output feature map contains the spatial information of the sensor data. To further analyze the dynamic changes of the signal over time, these spatial features will be input into the LSTM layer as input. Assume the output of the convolutional layer is F cnn , which is flattened into a one-dimensional vector to adapt to the input format of the LSTM. At this time, the input data of the LSTM will be the spatial features extracted by the CNN and the information of its time dimension.

[0109] The input sequence F of the LSTM cnn is input into the LSTM network via the time step t. The network architecture of the LSTM is as shown in the appendix Figure 2 . The main advantage of the LSTM is that its memory unit controls the information flow and update through the input gate, forget gate, and output gate, thereby capturing the temporal dependencies in the sequence.

[0110] The calculation process of the LSTM includes the following key steps:

[0111] First is the input gate, which controls the influence of the current input data on the memory unit. The input gate determines the proportion of the input information at each moment through the sigmoid activation function:

[0112] i t = σ(W i ·[h t-1 , F cnn,t +b i )

[0113] where, i t is the output of the input gate, σ is the sigmoid activation function, W i and b i are the weight matrix and bias term of the input gate respectively, h t-1 is the output of the LSTM at the previous time step, and F cnn is the convolutional feature at the current time step.

[0114] Then there is the Forget Gate, which controls the influence of the memory content at the previous moment on the current state.

[0115] The Forget Gate is also calculated through the sigmoid function to determine how much of the previous moment's memory to discard:

[0116] f t = σ(W f · [h t-1 , F cnn,t + b f )

[0117] where f t is the output of the Forget Gate, and W f and b f are the weight and bias term of the Forget Gate respectively. The output of the Forget Gate determines how much of the previous moment's memory needs to be retained.

[0118] Next is the Output Gate, which controls the output of the current memory cell and determines the content output by the LSTM cell at the current moment:

[0119] o t = σ(W o · [h t-1 , F cnn,t + b o )

[0120] where o t is the output of the Output Gate, and W o and b o are the weight matrix and bias term of the Output Gate respectively. Through the control of the Output Gate, LSTM determines the output h t at the current moment and combines the current memory state to generate the final prediction result.

[0121] Next, LSTM calculates the Cell State, which combines the outputs of the Input Gate and the Forget Gate to update the current memory state. The update formula for the Cell State is:

[0122] c t = f t · c t-1 + i t · tanh(W c · [h t-1 , F cnn,t + b c )

[0123] where c t is the memory cell at the current moment, c t-1 is the memory cell at the previous moment, and W c and bc It is used to calculate the weights and bias terms for updating the memory unit. The updated memory unit stores the key time-dependent information from historical inputs and passes it to the next time step.

[0124] Finally, the output h of the LSTM unit t is calculated by the following formula:

[0125] h t = o t · tanh(c t )

[0126] where h t is the output at the current moment, and tanh is the hyperbolic tangent activation function. The output h t captures the combination of the current moment and historical information and can effectively transmit the temporal features to the subsequent layers of the network.

[0127] In the architecture of the present invention, the output of the LSTM layer will be used for the subsequent decision-making layer to further perform mechanical change prediction and real-time feedback. Through the learning of LSTM, the model can extract information from historical mechanical signals, predict the future trend of mechanical changes, and provide accurate force feedback for doctors to help them adjust the operation force during the surgery.

[0128] The advantage of the LSTM model lies in its powerful time series modeling ability, especially when the mechanical signals during the surgery usually have long-term time dependence. Through LSTM, the system can dynamically capture these changes and adjust the feedback in a timely manner, thus providing more accurate surgical guidance for doctors.

[0129] In the present invention, the training process of the model involves constructing a high-quality training set covering multi-dimensional data of various mechanical changes during the surgery. To train the deep learning model, we first collect data from real surgical scenarios to ensure the diversity and representativeness of the data. We have collected sensor data from simulated surgical environments and real surgical procedures, and these data cover a variety of operation scenarios such as cutting, suturing, and compressing. Each operation process includes multi-dimensional signals from force sensors, vibration sensors, and acceleration sensors. The sensor data acquisition equipment includes: high-precision force sensors for monitoring the force applied to the tissue; vibration sensors for capturing minute vibration signals; and acceleration sensors for recording the acceleration changes during the movement of the tool.

[0130] These data are recorded in real time during the acquisition process and synchronously transmitted to the central processing unit via wireless transmission to ensure that all data can be synchronously acquired at the same time point. The data we collect are the full-course signals of each surgery, including all operation processes from the start to the end of the surgery. The sensor output at each time step contains information such as the magnitude of the force, the frequency of vibration, and the change in acceleration. All data will be processed through standardization, denoising, etc. after acquisition to ensure its quality is suitable for training.

[0131] In terms of data annotation, each data sample is accurately annotated by expert doctors. The annotation content includes the mechanical change information at each time step during the operation process, such as the force value applied at each moment, whether there are abnormal changes (such as excessive or too small force), and whether these changes affect the success of the surgery or pose risks (such as bleeding, accidental injury, etc.). These annotation information provide the necessary labels for subsequent supervised learning.

[0132] Based on the above method, we created a dataset containing 3000 surgical cases, and each case contains 30 minutes of real-time sensor data. The signal data within each surgical case is divided into multiple time steps according to the time step length, and each time step is accompanied by the operation record of the doctor and the label of the mechanical change. These data cover the operation processes from simple tissue cutting to complex blood vessel suture.

[0133] The goal during the training process is to learn the changing rules of mechanical signals in different surgical scenarios through a deep learning model. By learning these multi-dimensional data, the model can predict the changing trend of the magnitude of the force at each moment and output mechanical feedback based on real-time data to help doctors adjust the operation force and avoid misoperation or surgical failure.

[0134] During training, we used a convolutional neural network (CNN) to extract spatial features and then processed the time series signals through a long short-term memory network (LSTM). In the CNN processing stage before the LSTM layer, the model extracts spatial features at different scales from the sensor data through convolutional kernels. After dimensionality reduction and feature aggregation through the max-pooling layer, the output feature map will be fed into the LSTM for further temporal dynamic modeling. The LSTM network can capture the long-term dependencies in the data and understand the changing trend of mechanical signals over time.

[0135] During the training process of the model, we used standard supervised learning methods and optimized the performance of the model through a multi-task loss function. Since the mechanical changes during the surgery usually involve multiple subtasks, such as the prediction of cutting force, compression force, and shear force, we designed a multi-task loss function to enable the model to handle these tasks simultaneously and share information between tasks, thereby improving the prediction accuracy of each task.

[0136] The multi-task loss function can be expressed as:

[0137]

[0138] Wherein, respectively represent the losses for cutting force, compression force, and shear force,

[0139] λ1, λ2, and λ3 are the weight coefficients for each task, used to control the contribution of each task to the total loss. In this way, the model can share information during the multi-task learning process and improve the prediction accuracy of each task.

[0140] During model training, we optimize the above multi-task loss function to make the model perform more precisely on different mechanical tasks. By minimizing the total loss, the model can find a balance in each task, thereby improving the prediction accuracy of the mechanical signal changes during the surgical process.

[0141] The loss function calculates the error between the predicted result of the model and the true label. Assume is the true label, and y i is the output predicted by the model. Then the definition of the loss function in a single task is:

[0142]

[0143] The loss function measures the squared difference between the predicted value and the true value. By minimizing this loss, the model can gradually adjust its parameters and improve the prediction accuracy.

[0144] To improve the stability of the training process and prevent gradient explosion or vanishing, we also use the Adam optimizer. The Adam optimizer can accelerate the training process and improve the convergence speed of the model through an adaptive learning rate adjustment strategy. In this way, the model continuously optimizes its parameters during multiple rounds of iteration, and finally can effectively identify the key patterns in the mechanical signals and capture the temporal dynamic changes of the mechanical signals through the LSTM model.

[0145] The trained and optimized model can accurately predict the mechanical changes during the surgical process. Especially when the mechanical signals change, the model can quickly capture these dynamic changes. When the model is deployed to the actual surgical process, the trained LSTM layer will use the sensor data collected in real time to predict the future mechanical change trend and transmit the prediction result to the doctor through the real-time force feedback device.

[0146] Step d, real-time force feedback output:

[0147] In the present invention, through the processing of the deep learning model, the system can output force feedback information in real time according to the sensor data and the prediction results of the model, helping the doctor to accurately adjust the operation force during the operation and ensuring the safety and accuracy of the operation. This real-time force feedback mechanism is crucial in practical applications. It can not only feedback the current mechanical state in real time, but also effectively prompt the doctor about possible operation problems during the operation, avoiding unnecessary injuries or misoperations.

[0148] In the specific implementation of force feedback, the system transmits the feedback information to the doctor through several different devices or apparatuses. These feedback methods include two main forms: vibration feedback and pressure simulation. And during the operation, the system dynamically adjusts the intensity and method of the feedback signal according to the force applied by the doctor's operation and the contact situation between the tool and the tissue.

[0149] Vibration feedback is the most common and intuitive feedback form in the system. Through the vibration feedback device in the surgical robot handle or the glove worn by the doctor, when the system detects that the force applied by the doctor is too large or too small, the feedback device will emit vibration signals of different intensities. For example, when the doctor performs a fine cut, if the applied force is too large, the system will remind the doctor to reduce the force through the vibration on the handle, avoiding tissue damage or bleeding caused by cutting too deep. In this case, the system warns the doctor by increasing the vibration frequency and intensity to ensure that the doctor can sense the excessive force. However, if the operation force is too small, resulting in a slow surgical process or difficulty in execution, the system also reminds the doctor to increase the force through vibration feedback (using different vibration frequencies and intensities) to ensure the smooth progress of the operation. In practice, the intensity and frequency of vibration can be set according to the doctor's operation force and adjusted immediately through the device.

[0150] Pressure simulation is another important feedback method, especially suitable for high-precision operations in robot-assisted surgery. Through the pressure feedback device, the system can simulate the feeling when contacting the actual tissue. When the pressure applied by the robotic surgical tool reaches a predetermined threshold, the pressure feedback device will simulate the feeling of contacting the actual tissue, helping the doctor to perceive the magnitude of the surgical force. For example, when performing vascular suture, the doctor cannot directly touch the pressure of the surgical tool, but through pressure simulation, the doctor can perceive the pressure applied by the robot through the handle or wearable device. This simulation can effectively help the doctor judge whether the pressure is too large, thus avoiding damaging the blood vessel, or whether the pressure is too small, resulting in operation failure. The pressure sensor used in this pressure simulation device can accurately sense the pressure applied by the surgical tool and synchronize with the actual surgical situation, thus providing accurate force feedback for the doctor.

[0151] Step e, multi-dimensional force monitoring and dynamic adjustment:

[0152] The multi-dimensional mechanical monitoring system of the present invention can dynamically adjust the feedback force by real-time monitoring of various mechanical signals during the operation and combining the prediction and analysis of the operation stage by the deep learning model, ensuring that every step of the operation is carried out within the safe and accurate range.

[0153] The core function of the system is to automatically adjust the feedback mechanism according to different operation stages and tissue states to meet the needs of the current operation, avoid excessive pressure application, reduce the risk of tissue damage, and ensure the smooth completion of the operation.

[0154] In practical applications, the system can perform real-time analysis and processing based on multi-dimensional data collected by sensors, such as mechanical signals, vibration information, acceleration changes, etc. At the beginning of the operation, the system preprocesses the sensor data, identifies the current operation stage, and intelligently adjusts the feedback force (sets an appropriate correction coefficient) in combination with the prediction of different operations by the deep learning model. For example, during blood vessel suture, the system will detect the softness of the tissue and its sensitivity to force, and thus automatically reduce the feedback force. In this way, while avoiding excessive pressure application, the system can ensure that the doctor can precisely control the force and avoid unnecessary damage to soft tissues. At this time, the system will reduce the feedback intensity through the pressure simulation feedback and / or vibration feedback device without affecting the operation accuracy of the operation, providing a more delicate operation perception.

[0155] When the operation enters the cutting or other operations that require greater force, the system will adjust the feedback mechanism to provide appropriate cutting force feedback. At this time, the system will monitor the magnitude of the cutting force applied to the surgical tool in real time according to the data of the force sensor, and transmit relevant information to the doctor through the feedback device. By increasing the vibration feedback intensity and pressure simulation feedback, the doctor can more clearly perceive the magnitude of the current cutting force, ensure that the cutting force is appropriate, and avoid difficulties in operation caused by too small a force or tissue damage caused by too large a force. The system can dynamically adjust the feedback force, enabling the doctor to always maintain precise control of the surgical tool when performing high-risk operations such as cutting.

[0156] For example, in an actual heart valve operation, the doctor uses a robotic surgical system for blood vessel suture and valve repair. The system first monitors the contact of soft tissues. Through real-time sensor data analysis, the system identifies that this is a critical stage for suture and automatically adjusts to reduce the feedback force to ensure that the doctor's operation will not cause excessive pressure on the blood vessel or surrounding tissues. When the operation enters the valve cutting stage, the system identifies from the real-time data that a greater cutting force is required, and helps the doctor precisely control the cutting depth and force by increasing the intensity of the vibration feedback and the magnitude of the pressure simulation feedback, avoiding valve damage or incomplete cutting caused by uneven force.

[0157] Through this dynamic adjustment mechanism based on multi-dimensional mechanical monitoring, the system can effectively manage the feedback requirements at different surgical stages and make appropriate adjustments according to real-time sensor data. This not only enhances the precision of the surgery but also greatly improves the safety, ensuring that the doctor can adjust the operation force in real time according to the feedback, reducing the risks and complications during the surgery. Therefore, through the multi-dimensional mechanical monitoring and dynamic adjustment system of the present invention, the precise control of mechanical changes during the surgery is achieved, thus providing a safer and more accurate surgical environment for the doctor.

[0158] In summary, this solution provides real-time force feedback information by introducing multiple sensor signals and deep learning algorithms, enabling the doctor to accurately perceive the operation force during the surgery, thereby effectively avoiding problems such as accidental injury or excessive operation.

[0159] The above description of the embodiments is for the convenience of those of ordinary skill in the art to understand and use the invention. Obviously, those who are familiar with the technology in this field can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative labor. Therefore, the present invention is not limited to the above embodiments, and the improvements and modifications made by those skilled in the art without departing from the scope of the present invention according to the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A force feedback monitoring method based on deep learning for multi-sensor data fusion, used to transmit real-time force feedback from a robotic assisted surgery system to a doctor, characterized in that, It includes the following steps: S1: Obtain the magnitude and direction of the force applied by the tool through a force sensor, obtain the vibration of the tool through a vibration sensor, and obtain the acceleration change during the movement of the tool through an acceleration sensor; S2: Preprocess the acquired data and perform feature extraction; S3: Input the extracted spatial features into a deep learning model for time series processing to obtain a force feedback result and predict the force change trend; S4: According to the force feedback result and the force change trend prediction result obtained by the deep learning model processing, determine the current stage state, and adjust the transfer correction coefficient of the force feedback result according to the different stage states; S5: Provide real-time force feedback of the corrected force feedback result to the doctor.

2. The force feedback monitoring method based on multi-sensor data fusion of deep learning according to claim 1, characterized in that, In step S1, it includes one or both of the following: i) The force sensor is a strain type force sensor or a piezoelectric force sensor; ii) The vibration sensor is a piezoelectric sensor.

3. A force feedback monitoring method for multi-sensor data fusion based on deep learning according to claim 1, characterized in that, In step S2, the preprocessing includes normalization and filtering.

4. The force feedback monitoring method based on multi-sensor data fusion using deep learning according to claim 3, wherein, The filtering includes low-pass filtering and median filtering.

5. A force feedback monitoring method based on multi-sensor data fusion of deep learning according to claim 1, characterized in that, In step S2, the feature extraction is performed through a convolutional neural network.

6. The force feedback monitoring method based on multi-sensor data fusion using deep learning according to claim 1, characterized in that, In step S3, the deep learning model is a long short-term memory network.

7. The force feedback monitoring method based on multi-sensor data fusion of deep learning according to claim 6, characterized in that, In step S3, the deep learning model is trained through supervised learning and optimized through a multi-task loss function.

8. A force feedback monitoring method for multi-sensor data fusion based on deep learning according to claim 6, characterized in that, In step S5, the real-time force feedback is vibration feedback and / or pressure simulation feedback.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein when the program runs, it controls the device where the computer-readable storage medium is located to execute the method described in any one of claims 1-8.

10. A force feedback monitoring system based on multi-sensor data fusion of deep learning, characterized in that, For implementing the method described in any one of claims 1-8; The system includes a force sensor, a vibration sensor, an acceleration sensor, a central processing unit, and a force feedback device; The force sensor, the vibration sensor, and the acceleration sensor are respectively installed on the tool or the robot-assisted surgical system to obtain the magnitude and direction of the force applied by the tool, the vibration of the tool, and the acceleration change during the movement of the tool; The central processing unit is communicatively connected to the force sensor, the vibration sensor, the acceleration sensor, and the force feedback device. The central processing unit is used to receive the data acquired by the force sensor, the vibration sensor, and the acceleration sensor, and then process the data to obtain a corrected force feedback result; The force feedback device provides the corrected force feedback result to the doctor in real time.

Images