Lumbar plate cutting state sensing method based on deep learning and multi-modal fusion

Abstract

The invention relates to the technical field of medical information, and discloses a lumbar vertebra plate cutting state sensing method based on deep learning and multi-modal fusion, and the method comprises the following steps: S1, synchronously collecting a multi-modal physical signal in a lumbar vertebra plate cutting process in real time through a sensor group installed at the tail end of a surgical robot, the signals at least comprise force / torque signals, sound signals and vibration signals; s2, performing preprocessing and feature extraction on the collected modal signals to obtain feature representation of each modal; s3, inputting the feature representation of each modal into a pre-trained deep learning multi-modal fusion model for fusion and state recognition; and S4, according to the output of the deep learning multi-modal fusion model, identifying and outputting the type of a tissue layer where the tail end of the current cutting tool is located in real time, and predicting the current cutting depth. By means of the scheme, the cutting state can be sensed accurately in real time in a lossless mode, and accurate and intelligent control over the SPR operation process is achieved.

Description

Technical Field

[0001] This application relates to the field of medical information technology, and in particular to a method for perceiving the state of lumbar laminectomy based on deep learning and multimodal fusion. Background Technology

[0002] Selective posterior rhizotomy (SPR) is an effective surgical procedure for treating spastic cerebral palsy, and one of its core key steps is the precise cutting of the lumbar lamina to expose the target spinal nerve root. Precise positioning and intelligent manipulation are the future directions for orthopedic robotics. Currently, this procedure mainly relies on the surgeon's personal experience, touch, and visual observation, which has significant limitations. Lack of precise depth perception: Surgeons cannot perceive the cutting depth in real time and quantitatively, and can only make a rough judgment based on empirical indicators such as bone color and bleeding. To avoid damaging the deep spinal cord and nerve roots, the operation tends to be conservative, which may lead to insufficient laminectomy and affect the surgical outcome; or require repeated trials, prolonging the operation time. Inability to provide real-time status feedback: Existing surgical navigation systems (such as those based on preoperative CT / MRI) mainly provide static anatomical structure navigation and cannot dynamically provide real-time feedback on the interaction between the tool and bone tissue (such as whether it is currently in the cortical bone or cancellous bone layer). Limited sensing information: Although existing research has made attempts, most of them use single signals such as force, sound or vibration for monitoring. Due to the complex anatomy of the spine and the many interferences in the cutting environment, single-modal signal information is limited and has poor anti-interference ability, making it difficult to establish a stable and reliable sensing model, which cannot meet the clinical requirements for high reliability.

[0003] Therefore, the field of orthopedic surgical robots urgently needs to develop a system that can perceive the cutting state in real time, accurately, and without damage, in order to break through the bottleneck of "blind" cutting. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application provides a method for perceiving the state of lumbar laminectomy based on deep learning and multimodal fusion. This method solves the technical problems of existing technologies, which mainly rely on the surgeon's personal experience, touch, and visual observation, lacking precise depth perception, unable to provide real-time feedback, and having a single perception signal. It provides a key technology for quantitative feedback to doctors or robot control systems, thereby achieving precise and intelligent control of the SPR surgical process.

[0005] In a first aspect, embodiments of this application provide a method for perceiving the state of lumbar laminectomy based on deep learning and multimodal fusion, comprising the following steps: S1: Multimodal physical signals during the lumbar lamina cutting process are collected in real time and synchronously through a sensor group installed at the end of the surgical robot. The multimodal physical signals include at least force / torque signals, sound signals and vibration signals. S2: Preprocess and extract features from each of the acquired modal signals to obtain the feature representation of each modality; S3: Input the feature representations of each modality into a pre-trained deep learning multimodal fusion model for fusion and state recognition; the deep learning multimodal fusion model is used to learn the nonlinear mapping relationship between multimodal signals and cutting states; S4: Based on the output of the deep learning multimodal fusion model, identify and output the tissue layer type at the end of the cutting tool and predict the current cutting depth in real time.

[0006] In one possible implementation, the preprocessing and feature extraction in step S2 include: The force signal is subjected to noise reduction processing using Kalman filtering, and its time-domain and frequency-domain features are extracted. The sound signal is subjected to a Fast Fourier Transform to be converted to the frequency domain, and its Mel-frequency cepstral coefficients (MFCC) features are extracted. The vibration signal is subjected to bandpass filtering and fast Fourier transform to extract the amplitude characteristics of its harmonic components related to the spindle speed.

[0007] In one possible implementation, the deep learning multimodal fusion model described in step S3 is a fusion framework based on convolutional neural networks (CNN) and long short-term memory networks (LSTM), combined with a dynamic attention mechanism, including: The CNN module is used to perform one-dimensional convolution operations on the input feature sequences of each modality in order to extract local spatial features within each modality. The LSTM module is used to receive the feature sequences output by the CNN module and capture the long-short-term time dependencies in each modality feature through its gating mechanism. The dynamic attention mechanism module is used to perform weighted fusion of multimodal feature sequences processed by CNN and LSTM. By calculating the importance weights of features at different modalities and time steps, it adaptively focuses on key discriminative information.

[0008] In one possible implementation, the dynamic attention mechanism module performs the following operations: The hidden state sequences of multiple modalities of LSTM are concatenated to form a comprehensive feature matrix H; The multimodal feature sequences are transformed linearly to generate the Query matrix Q, Key matrix K, and Value matrix V; For target time step i and source time step j, calculate their attention scores; The attention scores are normalized to obtain the attention weights; The Value matrix V is weighted and summed according to the attention weights to obtain the context vector representation of the i-th time step; The obtained context vector representation is fused with the original input features to obtain the final updated fused feature representation rich in context information.

[0009] In one possible implementation, the training process of the deep learning multimodal fusion model employs an Adam optimizer incorporating Nesterov momentum for parameter updates. The parameter update process of the optimizer includes: Calculate the first moment estimate of the current time step and its Nesterov look-ahead momentum; The first-order moment estimate and Nesterov look-ahead momentum are corrected for bias. The parameters are updated based on the bias-corrected Nesterov look-ahead momentum and second-order moment estimates.

[0010] In one possible implementation, the correction process for the second-order moment estimation includes the following steps: The original second moment estimate is calculated using the exponential moving average (EMA): in, The gradient at the current time step t, which is obtained through the loss function. Partial derivatives with respect to the model parameter θ, ; The element-wise square of the gradient; : The second moment estimate of the previous time step; The exponential decay rate of the second moment; : Current second-moment estimate without bias correction; Bias correction: in, Second-order moment estimate after bias correction; : The decay rate raised to the power of t.

[0011] In one possible implementation, the tissue layer type in step S4 includes: outer cortical bone, cancellous bone, and inner cortical bone; the predicted current cutting depth is the predicted remaining distance from the target spinal nerve root.

[0012] In one possible implementation, the training process of the deep learning multimodal fusion model includes: constructing a training dataset using historical surgical data, wherein the labels of the dataset are obtained by aligning and annotating postoperative imaging data with intraoperative multimodal signals.

[0013] In one possible implementation, the step of extracting features from each acquired modal signal to obtain a feature representation of each modality includes: Wavelet packet decomposition is performed on the force / torque signal, and the Shannon entropy of its energy distribution is calculated to obtain the cutting force energy distribution entropy characteristic that characterizes the stability of the cutting process. The sound signal is subjected to short-time Fourier transform, and the amplitude ratio of harmonic clusters synchronized with the harmonics of the spindle speed is extracted to construct bone voiceprint features. After performing adaptive bandpass filtering on the vibration signal, the offset of its main harmonic frequency relative to the theoretical rotational speed of the spindle is calculated to obtain the characteristics of the main harmonic offset.

[0014] Secondly, embodiments of this application also provide a lumbar laminectomy state perception system based on deep learning and multimodal fusion, comprising: A multimodal data acquisition module is used to synchronously acquire multimodal physical signals during the lumbar lamina cutting process in real time through a sensor group installed at the end of the surgical robot. The multimodal physical signals include at least force / torque signals, sound signals, and vibration signals. The multimodal feature extraction module is used to preprocess and extract features from the acquired modal signals to obtain feature representations for each modality. The cutting state recognition module is used to input the feature representations of each mode into a pre-trained deep learning multimodal fusion model for fusion and state recognition; the deep learning multimodal fusion model is used to learn the nonlinear mapping relationship between multimodal signals and cutting states; The cutting depth prediction module is used to identify and output the tissue layer type at the end of the cutting tool and predict the current cutting depth in real time based on the output of the deep learning multimodal fusion model.

[0015] Compared with existing technologies, the lumbar laminectomy state perception method and system based on deep learning and multimodal fusion provided in this application have the following significant advantages: 1. This application analyzes multimodal physical signals during the cutting process, without penetrating tissue or relying on frequent intraoperative imaging scans, and can determine the type of tissue layer (such as cortical bone or cancellous bone) where the cutting tool is located in real time and predict the remaining distance to the target nerve. This transforms the surgery from an experience-based "blind" operation to a data-driven "visual" and "quantifiable" precise operation, achieving real-time, non-destructive, and precise state perception during the operation.

[0016] 2. This application fully utilizes the complementarity between different signals by deeply integrating the signal characteristics of three modes: force / torque, sound, and vibration (e.g., force signals directly reflect resistance, sound signals reflect high-frequency vibration characteristics, and vibration signals reflect changes in system stiffness). This significantly improves the accuracy and robustness of perception, overcomes the shortcomings of limited information sources and susceptibility to interference of single sensors, and demonstrates extremely high reliability and stability when facing complex and ever-changing intraoperative environments.

[0017] 3. The CNN-LSTM-Attention fusion network model designed in this application can automatically extract deep features from the original signal and simultaneously capture complex spatial (local modes) and temporal (long-range dependencies) correlations, thereby learning the highly nonlinear mapping relationship between multimodal signals and cutting states, which is difficult to describe using traditional mathematical models. This is the core of achieving high-precision perception. By introducing a Query-Key-Value structure to construct a learnable attention weight matrix, it dynamically focuses on the key segments in the input sequence that are most relevant to the current prediction, which is particularly suitable for handling high-dimensional sparse features and improving the model's discriminative power and prediction accuracy. Combining the Adam optimizer and Nesterov momentum mechanism and introducing a learning rate decay strategy: This application adopts a forward-looking parameter optimization method that integrates momentum, which suppresses gradient oscillations while maintaining the model's convergence speed. Combined with the learning rate decay strategy, it can improve long-term training stability and prevent overfitting and premature convergence.

[0018] 4. This application targets the physical mechanism of bone cutting, extracts multimodal features that are closely coupled with changes in the physical properties of the bone layer, and has extremely strong discriminative power, greatly enhancing the discriminative power of state recognition and providing high-quality information input for deep learning models.

[0019] 5. This application can effectively prevent accidental damage to the spinal nerve roots during surgery, greatly improving surgical safety; at the same time, by providing clear depth warnings, it can optimize the cutting path, reduce unnecessary bone structure damage, and help maintain postoperative spinal stability. It is a key technology to promote the SPR procedure from invasive to minimally invasive or non-invasive, and is of great significance to promoting the development of intelligent orthopedic surgery. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1A flowchart illustrating a lumbar laminar fusion-based method for perceiving the state of lumbar vertebral laminar fusion, provided for an embodiment of this application; Figure 2 A technical roadmap for a lumbar lamina cutting state perception method based on deep learning and multimodal fusion is provided for embodiments of this application; Figure 3 A system architecture diagram of a lumbar laminar lamina cutting state perception method based on deep learning and multimodal fusion provided for embodiments of this application; Figure 4 This is a schematic diagram of a lumbar lamina cutting state perception system based on deep learning and multimodal fusion, provided as an embodiment of this application. Detailed Implementation

[0022] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0023] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them. It should also be understood that in the embodiments of this disclosure, "multiple" can refer to two or more, and "at least one" can refer to one, two, or more. It should also be understood that any component, data, or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless explicitly limited or given a contrary indication in the context. Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this disclosure generally indicates that the related objects before and after are in an "or" relationship. It should also be understood that the descriptions of the various embodiments in this disclosure emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be elaborated upon one by one.

[0024] Furthermore, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. Techniques, methods, and apparatus known to those skilled in the art will not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it need not be further discussed in subsequent drawings.

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Figure 1 A flowchart illustrating a lumbar laminar fusion-based method for perceiving the state of lumbar vertebral laminar fusion, provided for an embodiment of this application; Figure 2 A technical roadmap for a lumbar lamina cutting state perception method based on deep learning and multimodal fusion is provided for embodiments of this application; Figure 3 This document presents a system architecture diagram for a lumbar laminectomy state perception method based on deep learning and multimodal fusion, as provided in an embodiment of this application. Figures 1-3 As shown, a lumbar laminectomy state perception method 100 based on deep learning and multimodal fusion includes the following steps: S1: Multimodal physical signals during the lumbar lamina cutting process are collected in real time and synchronously through a sensor group installed at the end of the surgical robot. The multimodal physical signals include at least force / torque signals, sound signals and vibration signals. In step S1, force signals, sound signals, and vibration signals are acquired in real time using multimodal signals. This facilitates subsequent analysis of the state characteristics and mapping mechanism of sound signals in different tissue layers during the lumbar lamina cutting process, research on the bone biomechanical properties of different tissue layers of the lumbar lamina and force sensing feedback and preprocessing methods, and analysis of the spectral characteristics of vibration signals in different tissue layers and their mapping relationship with the lumbar lamina cutting state.

[0027] Specifically, during the vertebral laminectomy performed by the surgical robot, force signals (Fx, Fy, Fz, Mx, My, Mz) (including force / torque signals) from a six-dimensional force sensor, audio signals from a high-sensitivity microphone, and vibration signals from a vibration sensor (such as an accelerometer or laser displacement sensor) are simultaneously acquired. All signals are sampled synchronously at a high rate to ensure data timing alignment.

[0028] S2: Preprocess and extract features from each of the acquired modal signals to obtain the feature representation of each modality; In step S2, signal preprocessing and feature extraction are performed: S2.1 Force / Torque Signal: Kalman filtering is used to denoise and smooth the original force signal, eliminating noise interference and instantaneous impacts. Time-domain features (such as mean, variance, and root mean square), frequency-domain features (through FFT transform), and wavelet packet energy features are extracted.

[0029] Further processing and feature extraction of force / torque signals specifically include: Kalman filtering is used to denoise and smooth the original six-dimensional force / torque signal, specifically filtering out periodic high-frequency noise caused by spindle rotation and impact interference caused by instantaneous bone fractures. To further quantify the stability of the cutting force and bone layer characteristics, this application innovatively introduces the "cutting force energy distribution entropy" feature based on wavelet packet transform. The specific method is as follows: Performing N-level wavelet packet decomposition on the filtered axial feed force signal (Fz) yields a complete binary tree-like decomposition structure, generating a total of Each terminal node or sub-band. Let j be the index of the terminal node (j = 0, 1, 2, ..., ...). Then the energy of the j-th subband signal (i.e., the sum of squares of the moduli of the coefficients at that node) is: Where n is the node index. This energy distribution reflects the energy allocation of the cutting force signal across different frequency bands.

[0030] Calculate the Shannon entropy for this energy distribution: ,in This entropy value effectively characterizes the smoothness of the cutting process. A lower entropy value indicates a more homogeneous bone (such as homogeneous cortical bone), while a sudden increase in entropy may indicate that the tool is about to penetrate the bone layer or encounter a cavity region. Finally, the cutting force energy distribution entropy feature, which characterizes the smoothness of the cutting process, is extracted.

[0031] S2.2 Audio Signal: The audio signal undergoes pre-emphasis, framing, and windowing processing, followed by Fast Fourier Transform (FFT) to convert it to the frequency domain. Mel-frequency cepstral coefficients (MFCC), spectral centroid, and spectral bandwidth are extracted.

[0032] Further processing and feature extraction of sound signals specifically include: After pre-emphasis, framing, and windowing of the audio signal, a short-time Fourier transform (STFT) is performed to obtain the time spectrum, and features such as Mel-frequency cepstral coefficients (MFCC), spectral centroid, and spectral roll-off point are extracted. This application constructs a "bone voiceprint" feature specifically for the unique sound generated by high-speed grinding drills cutting bone tissue. By analyzing the difference in sound spectrum between the spindle motor's idle and loaded cutting states, the amplitude ratio of harmonic clusters synchronized with the spindle speed via phase-locked loop (PLL) technology is extracted as a feature to construct the bone voiceprint feature. For example, the amplitude ratios of the fundamental frequency (F0), second harmonic (F1), and third harmonic (F2) are calculated. and Different bone densities and structures exhibit varying absorption and reflection characteristics for specific harmonics, making this set of harmonic ratio characteristics highly distinguishable between cortical and cancellous bone.

[0033] To further capture the signal modulation phenomenon caused by changes in bone density during the cutting process, this application embodiment also extracts the time-frequency envelope features of the signal, which can be applied to the extraction process of sound and vibration signals. Taking sound signals as an example, for the time spectrum S(t, f) obtained after STFT, its spectral envelope Env(t, f) at each time frame t is calculated. This application calculates the first-order difference (ΔEnv / Δt) and second-order difference (ΔEnv / Δt) of this envelope signal. The statistical characteristics of bone density (such as mean and variance) show that changes in bone density lead to shifts in the resonance peak frequency and amplitude, which are reflected in the differential properties of the envelope. For example, when transitioning from high-density cortical bone to low-density cancellous bone, the rate of change of the envelope (first-order difference) may increase significantly, while the acceleration of the change (second-order difference) may exhibit a specific peak pattern. This set of features provides the model with key information to characterize the dynamic transition process of the cutting state.

[0034] S2.3 Vibration Signal: Bandpass filtering and Fast Fourier Transform (FFT) are performed on the vibration signal to remove high-frequency noise and low-frequency drift. Similarly, FFT is performed to extract the amplitude and frequency of the main harmonic components (especially the frequency components related to the spindle speed, i.e., harmonic components), as well as the energy of each node after wavelet packet decomposition.

[0035] Further processing and feature extraction of vibration signals specifically include: The vibration signal is subjected to adaptive bandpass filtering, and its passband cutoff frequency is dynamically adjusted to the real-time spindle speed. (Unit: Hz) Associated, set to [ , This method aims to retain frequency band information strongly correlated with the cutting process to the greatest extent possible, while removing irrelevant high-frequency noise and low-frequency motion drift. After STFT transformation, this application not only extracts the amplitude of the main harmonic components but also extracts the "vibration main harmonic offset" feature. That is, it monitors the fundamental frequency component of the spindle speed. actual peak frequency Deviation from theoretical value Increased cutting resistance leads to a slight decrease in the instantaneous rotational speed of the tool, resulting in a small shift in the dominant vibration frequency (ΔF increases). This feature is extremely sensitive to changes in cutting force and can serve as a strongly correlated feature for identifying high-density bone layers (such as the inner cortical bone layer), thereby extracting the dominant vibration frequency shift feature.

[0036] Through the above preprocessing and feature extraction, not only were conventional time-domain and frequency-domain features obtained, but also a series of specialized features closely related to the physical mechanism of lumbar lamina cutting state and with strong discriminative power were obtained, providing richer and more reliable information input for subsequent deep learning models.

[0037] S3: Input the feature representations of each modality into a pre-trained deep learning multimodal fusion model for fusion and state recognition; the deep learning multimodal fusion model is used to learn the nonlinear mapping relationship between multimodal signals and cutting states; Step S3 is used to realize deep learning fusion network modeling and state recognition, which is the core of this application. The preprocessed multimodal feature sequence is input into a specially designed deep learning fusion network model, which aims to realize end-to-end mapping from multimodal sequence to cutting state.

[0038] S31: Model Construction To improve the accuracy and generalization ability of cutting prediction, this application constructs a time series modeling method based on neural networks to capture the complex coupling relationship between load data at the spatial and temporal levels, and builds an efficient prediction framework. For example... Figure 2 The diagram shows the structure of this deep learning fusion network model. It employs a framework that combines convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), namely the CNN-LSTM fusion framework, and incorporates a dynamic attention mechanism, specifically including: Input layer 311: Receives the aligned multimodal feature sequence X; For example, a multimodal feature sequence X containing three modalities is: Each of them It is a time step of length L and feature dimension of The matrix.

[0039] The convolutional neural network (CNN) module 312 is used to extract local spatial features from the feature sequences of each modality; For the feature sequence of each mode Each modality signal is processed using a one-dimensional convolutional layer (1D-CNN). The convolutional kernel slides along the time axis, which can effectively extract the local spatiotemporal patterns in each modality signal, as shown in the following expression: Where * represents a one-dimensional convolution operation, which performs sliding weighted calculations on time series data; Nonlinear activation functions (such as ReLU) are used to enhance the network's ability to fit features nonlinearly. : Convolution kernel weights, used to extract local feature patterns along the time axis; : Convolutional layer bias term; The feature representation output by the convolutional layer captures the spatial dependency structure of the input data that changes over time.

[0040] The convolutional neural network module 312 maps the features of each modality from the original space to a higher-dimensional, more discriminative feature space to capture local patterns such as "abrupt changes in cutting force at a specific frequency" and "resonance of sound signals in a specific frequency band".

[0041] Long Short-Term Memory (LSTM) module 313 is used to capture long and short-term time dependencies in feature sequences of each modality; The high-level feature sequence output by each CNN module The input is fed into the corresponding LSTM layer. The gating mechanism of LSTM (input gate, forget gate, output gate) enables it to learn and remember long-distance temporal dependencies in the cutting process, such as the dynamic evolution of the entire process from "starting to contact the cortical bone" to "penetrating the cortical bone and entering the cancellous bone".

[0042] The LSTM ultimately outputs the hidden state sequence for each mode. .

[0043] More specifically, the core gating mechanism of LSTM is as follows: in, The convolution module at time step The output feature vector; The hidden state at the previous moment; The input gate controls the proportion of new information written to the current memory cell; The forgetting gate controls the degree to which memories from the previous moment are retained; The output gate controls the proportion of the current memory cell outputting to the hidden state; The state of the memory unit at the previous moment; : The updated state of the memory cell at the current moment; The hidden state at the current moment is passed to subsequent time steps or the output layer; Weight matrices for input gate, forget gate, output gate, and candidate units; : The bias term corresponding to the gating unit; The Sigmoid activation function outputs values ​​in the range of... Used to control the opening and closing of doors; Element-wise multiplication operation; : Hyperbolic tangent function, used to control the range of candidate memory cells.

[0044] The dynamic attention mechanism module 314 is used to calculate the importance weights of each modality feature at different time steps and to perform adaptive feature fusion. To further enhance the model's ability to perceive key features, especially when processing high-dimensional sparse data, this application introduces a dynamic attention mechanism to adaptively adjust feature weights. This enables the adaptive fusion of information from multiple modalities and highlights the most important modalities and features at the current moment. The dynamic attention mechanism module 314 performs the following operations: S3141: Concatenate the LSTM hidden state sequences of multiple modalities to form a comprehensive feature matrix H, which integrates feature sequences of multimodal information. Its shape should be (L, D), where L is the time step and D is the total feature dimension. .

[0045] S3142: Generate the Query matrix Q, Key matrix K, and Value matrix V from the multimodal feature sequence through linear transformation; Specifically, , , ; Where H: input feature sequence, each element in H The first matrix represents the fused feature vector at the i-th time step; Q: Query matrix, representing the query vector sequence used to retrieve key information from the input sequence; K: Key matrix, representing the keyword vector sequence, which constitutes the information being focused on; V: Value matrix, representing the value vector on which the attention output depends. : These are the linear transformation weight matrices (learnable parameters) corresponding to Query, Key, and Value, respectively.

[0046] These transformations enable the model to understand input features from different semantic dimensions and are the core parameters of the attention mechanism.

[0047] S3143: For target time step i and source time step j, calculate their attention scores: ; in, Attention score, used to measure the importance of features at time step j when computing the context of time step i, where i and j are indices of time step; The key vector at the j-th time step in the Key matrix; The dimension of the key vector, used as a scaling factor to prevent excessively large scores from causing gradient vanishing or exploding.

[0048] S3144: Normalize the attention scores to obtain the attention weights: Where L: the total time step of the sequence. Attention weights represent the degree of attention the model pays to information at time step j at time step i.

[0049] S3145: The Value matrix V is weighted and summed according to the attention weights to obtain the context vector representation at the i-th time step: in, : The value vector in the Value matrix corresponding to the j-th time step; : The attention output vector after weighted summation represents the feature representation after global context aggregation at the current time step i.

[0050] S3146: Obtain the context vector Compared with the original input features The features at the i-th time step are fused, for example through residual connections and layer normalization, to obtain the final updated feature representation rich in contextual information. : Here, Linear represents a linear transformation layer used to adjust the dimensions; The final updated, context-rich feature representation is used for the feature representation of the subsequent output layer 315.

[0051] Output layer 315: The final feature sequence H′, after fusion and updating (its elements are...), is... The input is fed into the output layer, which outputs the tissue type classification result at the current time step t and the predicted remaining distance to the target nerve root.

[0052] For classification tasks (identifying tissue types): H′ is passed through a fully connected layer and a Softmax activation function to output the probability distribution of the current time step t belonging to each tissue layer, such as: category 1 - cortical bone, category 2 - transition zone, category 3 - cancellous bone, etc.

[0053] For regression tasks (predicting remaining distance): H′ is passed through a fully connected layer (linear activation), and the output is a continuous scalar value (in millimeters) representing the predicted remaining distance to the target nerve root.

[0054] Specifically, using mean squared error as the objective function, the final hidden state H′ is input into the fully connected layer for mapping, obtaining the predicted value at the target time t: in, : The weight matrix of the output layer; : The bias term of the output layer; The estimated value of the target variable at time point t, predicted by the neural network, i.e., the output tissue type classification result or the remaining distance prediction value. For classification tasks, Followed by Softmax; for regression tasks, This means direct output.

[0055] S32: Model Training and Optimization (1) Data preparation: Collect a large amount of historical surgical data, including multimodal signals during the cutting process and corresponding “true labels”. Labels can be obtained through postoperative synchronous imaging data (such as intraoperative O-arm scans) or manual annotation by experts based on signal waveforms and video recordings. True labels include the tissue layer type (such as cortical bone, cancellous bone) and / or the precise distance (in millimeters) from the target nerve root at each time step.

[0056] Loss function: Cross-entropy loss is used for classification tasks, and mean squared error loss is used for regression tasks. The total loss is a weighted sum.

[0057] Optimizer: The training process of the deep learning multimodal fusion model uses the Adam optimizer, which incorporates Nesterov momentum, for model training and parameter updates.

[0058] Furthermore, during the training process, the Adam optimizer, which incorporates Nesterov momentum, is used for parameter updates. The parameter update process of the optimizer specifically includes the following steps: S321: Calculate the first moment estimate of the current time step t and its Nesterov look-ahead momentum: in, This is the gradient of the current loss function with respect to the model parameters. For the first-order moment estimation in the previous step, It is the first-order moment attenuation factor; The current revised first-order momentum estimate is used to smooth the gradient. The look-ahead first-order momentum, after introducing Nesterov momentum, is used to simulate the gradient trend of "one step ahead". S322: Correct the bias in the first-order moment estimate and Nesterov look-ahead momentum: in, These are the first-order momentum and the look-forward momentum after deviation correction, respectively. : is the first-order moment decay factor, used as a power-order decay factor to correct deviation; S323: Update parameters based on the bias-corrected Nesterov look-ahead momentum and second-order moment estimates. in, These are the updated model parameters at the current moment. Here are the model parameters from the previous step, and η is the learning rate. This is the second-order moment estimate after bias correction, and ε is the numerical stability factor.

[0059] : Model parameters updated at the current time step t; : The model parameters from the previous step; Learning rate, used to control the step size; : The second-order moment estimate after bias correction, used to adaptively adjust the learning rate; ε: Numerical stability factor.

[0060] Among them, the second-order moment estimation The correction process includes the following steps: S3231: Calculate the original second moment estimate using the exponential moving average (EMA): in, The gradient at the current time step t, which is obtained through the loss function. Partial derivatives with respect to the model parameter θ, ; The element-wise square of the gradient is specifically the square operation performed on each element of the gradient vector. : The second moment estimate of the previous time step; The exponential decay rate of the second moment; : Current second-moment estimate without bias correction; S3232: Deviation Correction in, Second-order moment estimate after bias correction; : Attenuation rate t Power of 1.

[0061] This optimizer combines Adam's adaptive learning rate with the "look-forward" capability of Nesterov momentum to accelerate convergence and improve generalization performance. It also employs strategies such as learning rate decay and Dropout to prevent overfitting.

[0062] The CNN-LSTM-Attention fusion network model constructed in step S3 of this application can automatically extract deep features from the original signal and simultaneously capture complex spatial (local modes) and temporal (long-range dependencies) correlations, thereby learning the highly nonlinear mapping relationship between multimodal signals and cutting states, which is difficult to describe using traditional mathematical models. This is the core of achieving high-precision perception. By introducing a Query-Key-Value structure to construct a learnable attention weight matrix, it dynamically focuses on the key segments in the input sequence that are most relevant to the current prediction, which is particularly suitable for handling high-dimensional sparse features and improving the model's discriminative power and prediction accuracy. Combining the Adam optimizer and Nesterov momentum mechanism and introducing a learning rate decay strategy: This application adopts a forward-looking parameter optimization method that integrates momentum, which suppresses gradient oscillations while maintaining the model's convergence speed. Combined with the learning rate decay strategy, it can improve long-term training stability and prevent overfitting and premature convergence.

[0063] S4: Based on the output of the deep learning multimodal fusion model, identify and output the tissue layer type at the end of the cutting tool and predict the current cutting depth in real time.

[0064] In step S4, the tissue layer types include: outer cortical bone, cancellous bone, and inner cortical bone; the predicted current cutting depth is the predicted remaining distance from the target spinal nerve root.

[0065] According to the above embodiments of this application, by synchronously acquiring multimodal sensor signals of force, sound, and vibration, specialized discriminative features closely related to bone layer characteristics are innovatively extracted. A CNN-LSTM-Attention deep learning fusion network is used to establish a precise mapping between signals and states, overcoming the technical bottlenecks of inaccurate state perception and delayed feedback during lumbar lamina cutting. This achieves real-time, non-destructive, and high-precision perception of tissue layer type and cutting depth, providing core intelligent perception capabilities for spinal surgery robots. This is a major breakthrough in realizing the minimally invasive, precise, and intelligent nature of SPR surgery.

[0066] Figure 4 This is a schematic diagram of a lumbar laminar fusion-based state perception system for lumbar vertebral laminar fusion, provided as an embodiment of this application. Figure 4 As shown, this application provides a lumbar laminar laparotomy state perception system 200 based on deep learning and multimodal fusion, comprising: The multimodal data acquisition module 210 is used to synchronously acquire multimodal physical signals during the lumbar lamina cutting process in real time through a sensor group installed at the end of the surgical robot. The multimodal physical signals include at least force / torque signals, sound signals, and vibration signals. The multimodal feature extraction module 220 is used to preprocess and extract features from each of the acquired modal signals to obtain the feature representation of each modality. The cutting state recognition module 230 is used to input the feature representations of each mode into a pre-trained deep learning multimodal fusion model for fusion and state recognition; the deep learning multimodal fusion model is used to learn the nonlinear mapping relationship between multimodal signals and cutting states; The cutting depth prediction module 240 is used to identify and output the tissue layer type at the end of the cutting tool and predict the current cutting depth in real time based on the output of the deep learning multimodal fusion model.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0068] Furthermore, embodiments of this application also provide an electronic device, including: a processor, a memory, and a system bus; the processor and the memory are connected via the system bus; the memory is used to store one or more programs, the one or more programs including instructions, which, when executed by the processor, cause the processor to perform any of the methods described above.

[0069] Furthermore, embodiments of this application also provide a computer program product, which, when run on a terminal device, causes the terminal device to execute any of the methods described above.

[0070] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0071] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0072] It should also be noted that, in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0073] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in the embodiments of this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown in this application, but is to be accorded the widest scope consistent with the principles and novel features disclosed in the embodiments of this application.

Claims

1. A method for perceiving the state of lumbar laminectomy based on deep learning and multimodal fusion, characterized in that, Includes the following steps: S1: Multimodal physical signals during the lumbar lamina cutting process are collected in real time and synchronously through a sensor group installed at the end of the surgical robot. The multimodal physical signals include at least force / torque signals, sound signals and vibration signals. S2: Preprocess and extract features from each of the acquired modal signals to obtain the feature representation of each modality; S3: Input the feature representations of each modality into a pre-trained deep learning multimodal fusion model for fusion and state recognition; the deep learning multimodal fusion model is used to learn the nonlinear mapping relationship between multimodal signals and cutting states; S4: Based on the output of the deep learning multimodal fusion model, identify and output the tissue layer type at the end of the cutting tool and predict the current cutting depth in real time.

2. The method for sensing the cutting state of the lumbar lamina according to claim 1, characterized in that, The preprocessing and feature extraction described in step S2 include: The force signal is subjected to noise reduction processing using Kalman filtering, and its time-domain and frequency-domain features are extracted. The sound signal is subjected to a Fast Fourier Transform to be converted to the frequency domain, and its Mel-frequency cepstral coefficients (MFCC) features are extracted. The vibration signal is subjected to bandpass filtering and fast Fourier transform to extract the amplitude characteristics of its harmonic components related to the spindle speed.

3. The method for sensing the cutting state of the lumbar lamina according to claim 1, characterized in that, The deep learning multimodal fusion model described in step S3 is a fusion framework based on convolutional neural networks (CNNs) and long short-term memory networks (LSTMs), combined with a dynamic attention mechanism, including: The CNN module is used to perform one-dimensional convolution operations on the input feature sequences of each modality in order to extract local spatial features within each modality. The LSTM module is used to receive the feature sequences output by the CNN module and capture the long-short-term time dependencies in each modality feature through its gating mechanism. The dynamic attention mechanism module is used to perform weighted fusion of multimodal feature sequences processed by CNN and LSTM. By calculating the importance weights of features at different modalities and time steps, it adaptively focuses on key discriminative information.

4. The method for sensing the cutting state of the lumbar lamina according to claim 3, characterized in that, The dynamic attention mechanism module performs the following operations: The hidden state sequences of multiple modalities of LSTM are concatenated to form a comprehensive feature matrix H; The multimodal feature sequences are transformed linearly to generate the Query matrix Q, Key matrix K, and Value matrix V; For target time step i and source time step j, calculate their attention scores; The attention scores are normalized to obtain the attention weights; The Value matrix V is weighted and summed according to the attention weights to obtain the context vector representation of the i-th time step; The obtained context vector representation is fused with the original input features to obtain the final updated fused feature representation rich in context information.

5. The method for sensing the cutting state of the lumbar lamina according to claim 1, characterized in that, The training process of the deep learning multimodal fusion model uses an Adam optimizer incorporating Nesterov momentum for parameter updates. The parameter update process of the optimizer includes: Calculate the first moment estimate of the current time step and its Nesterov look-ahead momentum; The first-order moment estimate and Nesterov look-ahead momentum are corrected for bias. The parameters are updated based on the bias-corrected Nesterov look-ahead momentum and second-order moment estimates.

6. The method for sensing the cutting state of the lumbar lamina according to claim 1, characterized in that, in, The correction process for the second-order moment estimate includes the following steps: The original second moment estimate is calculated using the exponential moving average (EMA): in, The gradient at the current time step t, which is obtained through the loss function. Partial derivatives with respect to the model parameter θ, ; The element-wise square of the gradient; : The second moment estimate of the previous time step; The exponential decay rate of the second moment; : Current second-moment estimate without bias correction; Bias correction: in, Second-order moment estimate after bias correction; : The decay rate raised to the power of t.

7. The method for sensing the cutting state of the lumbar lamina according to claim 1, characterized in that, The tissue layer types mentioned in step S4 include: outer cortical bone, cancellous bone, and inner cortical bone; the predicted current cutting depth is the predicted remaining distance from the target spinal nerve root.

8. The method for sensing the cutting state of the lumbar lamina according to claim 1, characterized in that, The training process of the deep learning multimodal fusion model includes: constructing a training dataset using historical surgical data, wherein the labels of the dataset are obtained by aligning and annotating postoperative imaging data with intraoperative multimodal signals.

9. The method for sensing the cutting state of the lumbar lamina according to claim 2, characterized in that, The step of extracting features from each of the acquired modal signals to obtain feature representations for each modality includes: Wavelet packet decomposition is performed on the force / torque signal, and the Shannon entropy of its energy distribution is calculated to obtain the cutting force energy distribution entropy characteristic that characterizes the stability of the cutting process. The sound signal is subjected to short-time Fourier transform, and the amplitude ratio of harmonic clusters synchronized with the harmonics of the spindle speed is extracted to construct bone voiceprint features. After performing adaptive bandpass filtering on the vibration signal, the offset of its main harmonic frequency relative to the theoretical rotational speed of the spindle is calculated to obtain the characteristics of the main harmonic offset.

10. A lumbar laminectomy state perception system based on deep learning and multimodal fusion, characterized in that, include: A multimodal data acquisition module is used to synchronously acquire multimodal physical signals during the lumbar lamina cutting process in real time through a sensor group installed at the end of the surgical robot. The multimodal physical signals include at least force / torque signals, sound signals, and vibration signals. The multimodal feature extraction module is used to preprocess and extract features from the acquired modal signals to obtain feature representations for each modality. The cutting state recognition module is used to input the feature representations of each mode into a pre-trained deep learning multimodal fusion model for fusion and state recognition; the deep learning multimodal fusion model is used to learn the nonlinear mapping relationship between multimodal signals and cutting states; The cutting depth prediction module is used to identify and output the tissue layer type at the end of the cutting tool and predict the current cutting depth in real time based on the output of the deep learning multimodal fusion model.

Images