Force sensing methods, systems and training methods for puncture robots

Abstract

This invention relates to a force sensing method, system, and training method for a puncture robot, belonging to the field of image analysis technology. The force sensing system includes an initial processing unit, an intermediate processing unit, and an output unit. The intermediate processing unit includes at least an SLGA submodule and a TLGA submodule. The SLGA submodule includes a four-dimensional convolution module, an attention module, and a matrix summation module. The TLGA submodule includes a four-dimensional convolution module, a temporal segmentation module, a window partitioning module, an attention module, a matrix summation module, and a stitching module. The SLGA submodule can aggregate multi-scale features in the spatial domain, thereby endowing the system with the ability to capture local details and global features of the puncture site. The TLGA submodule, through local window learning and global bidirectional learning processes, enhances the system's ability to capture minute deformations of the puncture site in the temporal domain. The combination of these two submodules enables the system to achieve accurate force sensing, thereby ensuring that the puncture robot can quickly and accurately intervene in the puncture operation.

Description

Technical Field

[0001] This invention belongs to the field of image analysis technology, and more specifically, relates to a force sensing method, system and training method for a puncture robot. Background Technology

[0002] Puncture is a common method in the medical field for obtaining information from deep within the body. Traditional manual procedures rely on the operator's experience, and their efficiency and accuracy are affected by factors such as hand tremors and fatigue. Compared to manual biopsies, biopsies based on puncture robots offer advantages such as high automation and millimeter-level accuracy, significantly reducing the workload of doctors and shortening operation time. Thanks to these advantages, puncture robots have received widespread attention. Sensing the interaction forces between surgical instruments and soft tissue, utilizing feedback force and torque signals to assist motion planning, robot control, and surgical manipulation are crucial to ensuring rapid and accurate interventional procedures using puncture robots.

[0003] Currently, there are direct and indirect force sensing methods. In direct force sensing, sensors are mounted on the proximal or distal end of the surgical instrument to measure force. While distal sensor-based measurements can represent the most direct interaction between the surgical instrument and tissue, they are affected by factors such as sensor biocompatibility, sterilization requirements, size, and cost. Furthermore, proximal sensor-based measurements can produce inaccurate force measurements when the surgical instrument bends due to tissue compression. Unlike direct force sensing, indirect force sensing can be achieved by exploring soft tissue deformation in images using vision-based force sensing (VFS) methods. VFS methods are easier to deploy than sensor-based methods and do not have biocompatibility or sterilization requirements.

[0004] With the development of deep learning, many VFS (Vibration Force Detection) methods based on neural networks have been proposed. For example, Marban et al. used four RGB-D cameras to capture liver phantom deformation and a recurrent convolutional neural network to estimate contact force. Gessert et al. designed four neural networks with different spatiotemporal sequences and demonstrated the feasibility of using 4D optical coherence tomography (OCT) to achieve VFS. However, these methods are mainly based on optical images of the tissue surface or superficial layers, making them unsuitable for force estimation in biopsies of small, deep organs such as the prostate. Furthermore, the recurrent neural networks (RNNs) or convolutional neural networks (CNNs) used in these methods can only model local information and cannot effectively capture long-distance spatiotemporal correlations, making it difficult to meet the demands of precise force sensing in clinical applications.

[0005] Therefore, improving the accuracy of force perception in the VFS method is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a force sensing method, system, and training method for a puncture robot. The purpose is to improve the accuracy of force sensing in the VFS method, ensuring that the puncture robot can quickly and accurately intervene in puncture operations.

[0007] A force sensing system for a puncture robot, the force sensing system comprising:

[0008] The initial processing unit is used to perform spatial size downsampling on the four-dimensional ultrasound sequence at the puncture site using a four-dimensional convolution module to obtain an initial feature map.

[0009] Intermediate stage processing unit, used to extract spatiotemporal features of the initial feature map;

[0010] The output unit is used to pool the output feature map of the last intermediate stage processing unit and project it into three-dimensional components of force information.

[0011] The intermediate processing unit includes at least an SLGA submodule and a TLGA submodule:

[0012] The SLGA submodule includes a four-dimensional convolution module, an attention module, and a matrix summation module. The input feature map SX of the SLGA submodule is input into the three four-dimensional convolution modules to generate a query matrix SQ, a key matrix SK, and a value matrix SV. The key matrix SK and the value matrix SV are downsampled by the four-dimensional convolution modules with relatively small strides to generate a fine-grained key matrix SK. fgSum matrix SV fg The bond matrix SK with coarse granularity is generated by downsampling through a four-dimensional convolution module with a relatively large stride. cg Sum matrix SV cg ; Key matrix SK fg Sum matrix SV fg The query matrix SQ is processed by one of the attention modules to generate the attention matrix SA. fg SK bond matrix cg Sum matrix SV cg The query matrix SQ is processed by another attention module to generate the attention matrix SA. cg Attention Matrix SA fg and attention matrix SA cg After summing by the matrix summing module, the attention matrix SA is generated. The attention matrix SA is then downsampled by the four-dimensional convolution module containing only spatial convolution kernels to output the feature map SY.

[0013] The TLGA submodule includes a four-dimensional convolution module, a temporal segmentation module, a windowing module, an attention module, a matrix summation module, and a concatenation module. The input feature map TX of the TLGA submodule is first processed by three four-dimensional convolution modules to generate a query matrix TQ, a key matrix TK, and a value matrix TV. Then, it is processed by the temporal segmentation module to divide the query matrix TQ into the first half of the temporal domain. f Key matrix TK f Sum Matrix TV f And the query matrix TQ in the second half of the time domain r Key matrix TK r Sum Matrix TV r Each matrix in the first and second half of the time domain is divided into multiple corresponding spatiotemporal small windows by the window partitioning module. The query matrix of the same spatiotemporal small window Key matrix Sum matrix An attention matrix is ​​generated after processing by one of the attention modules. Query matrix in the same spatiotemporal small window Key matrix Sum matrix An attention matrix is ​​generated after processing by another attention module. The superscripts f and r indicate that the matrix is ​​in the first half of the time domain and the second half of the time domain, respectively; the subscripts ijkl indicate that the matrix is ​​in the (i,j,k,l)th spatiotemporal window, where i,j,k,l represent the partition indices of the spatiotemporal window in the four dimensions; attention matrices are used in all spatiotemporal windows. The attention matrix TA is generated after summation by the matrix summation module. fr Attention matrix within all spatiotemporal small windows The attention matrix TA is generated after summation by the matrix summation module. rf Attention Matrix TA fr Attention Matrix After being concatenated by the concatenation module, a complete attention matrix TA is generated. The attention matrix TA is then downsampled by a four-dimensional convolution module containing only temporal convolution kernels to output a feature map TY.

[0014] In one embodiment, the processing formula for each attention module is:

[0015]

[0016] In the formula, Z is the output of the attention module, A, B, and E are the query matrix, key matrix, and value matrix of the corresponding attention module, respectively, and b is the feature length of the input key matrix.

[0017] In one embodiment, the SLGA submodule and the TLGA submodule are combined in any of the following ways: serial, parallel, or sequential residual connections.

[0018] In one embodiment, the SLGA submodule and the TLGA submodule constitute an STLGA module. The intermediate stage processing unit includes at least one STLG block or multiple cascaded STLG blocks. Each STLG block includes a position encoding module, the STLGA module, and a feedback forward network module. The position encoding module is used to process the input feature map X of the STLG block. l-1 After performing depth-aware four-dimensional convolution processing, the output feature map PE(X) is completed. l-1 The input feature map X l-1 and the output feature map PE(X) l-1 Superimposed feature map X1 l-1 The STLGA module is then used to extract spatiotemporal features, generating a feature map STLGA(BN(X1)). l-1 The feature map X1 l-1 and the feature map STLGA(BN(X1) l-1 Superimposed feature map X2 is generated. l-1 The feature map FFN(BN(X2) is then output after being input into the feedforward network module to enhance the feature spatial domain. l-1 The feature map FFN(BN(X2)) l-1 )) and the feature map X2 1-1 The output feature map X corresponding to the STLG block is generated by superposition. l The subscript l indicates the sequence number index of the STLG block.

[0019] In one embodiment, the force sensing system includes multiple intermediate stage processing units, namely, the first to the Nth stage processing units. The first stage processing unit uses the initial feature map as the input feature map, and the output of the previous stage processing unit is used as the input feature map of the next stage processing unit. Each stage processing unit is used to extract the spatiotemporal features of the corresponding input feature map.

[0020] In one embodiment, the output unit includes a spatiotemporal global average pooling layer and a linear layer. The spatiotemporal global average pooling layer is used to perform spatiotemporal global average pooling operation on the output feature map of the intermediate stage processing unit, and the linear layer is used to project the pooling result into three-dimensional components of force information.

[0021] In one embodiment, in the SLGA submodule, the stride of the four-dimensional convolution module that generates the coarse granularity matrix is ​​twice the stride of the four-dimensional convolution module that generates the fine granularity matrix.

[0022] In one embodiment, the puncture robot is a puncture robot that assists in prostate puncture.

[0023] A training method for the force sensing system of a puncture robot as described above uses a four-dimensional ultrasound sequence at the puncture location in historical data as input and a three-dimensional component of the force information corresponding to the puncture robot as output to train the force sensing system so that the output of the force sensing system converges to a preset degree.

[0024] A force sensing method for a puncture robot involves inputting a four-dimensional ultrasound sequence at the current puncture location into a force sensing system trained using the training method described above, to obtain the three-dimensional components of the force information for puncture robot-assisted puncture.

[0025] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0026] This application designs a force sensing system for a puncture robot, which includes an SLGA submodule and a TLGA submodule. The SLGA submodule can aggregate multi-scale features in the spatial domain, thereby giving the system the ability to capture local details and global features of the puncture site. The TLGA submodule, on the other hand, enhances the system's ability to capture minute deformations of the puncture site in the temporal domain through local window learning and global bidirectional learning processes. The combination of the two enables the system to achieve accurate force sensing, thereby ensuring that the puncture robot can quickly and accurately intervene in the puncture procedure. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the force sensing system of a puncture robot according to some embodiments of this application;

[0028] Figure 2 The present application provides some embodiments of the combination of SLGA submodules and TLGA submodules, wherein (a), (b), (c), and (d) represent connection diagrams for serial, parallel, first type of front-to-back residual connection, and second type of front-to-back residual connection, respectively.

[0029] Figure 3(a) is a schematic diagram of the structure of the SLGA submodule in some embodiments of this application;

[0030] Figure 3(b) is a flowchart of key processing in the SLGA submodule of some embodiments of this application;

[0031] Figure 3(c) is a schematic diagram of feature extraction of key steps in the SLGA submodule of some embodiments of this application;

[0032] Figure 4(a) is a schematic diagram of the structure of the TLGA submodule in some embodiments of this application;

[0033] Figure 4(b) is a flowchart of key processing in the TLGA submodule of some embodiments of this application;

[0034] Figure 4(c) is a schematic diagram of feature extraction of key steps in the TLGA submodule of some embodiments of this application;

[0035] Figure 5(a) is a schematic diagram of the structure of the STLG block in some embodiments of this application;

[0036] Figure 5(b) is a schematic diagram of the structure of each stage processing unit in some embodiments of this application;

[0037] Figure 6 This is a diagram illustrating the feature processing of a force sensing system that includes four intermediate processing units. Detailed Implementation

[0038] To facilitate understanding of this application and to make the aforementioned objectives, features, and advantages of this application more apparent, a detailed description of specific embodiments of this application is provided below in conjunction with the accompanying drawings. Numerous specific details are set forth in the following description to provide a thorough understanding of this application, and preferred embodiments are shown in the accompanying drawings. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of this application. This application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0039] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. In the description of this application, "several" means at least one, such as one, two, etc., unless otherwise explicitly specified.

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0041] Example 1

[0042] like Figure 1 The diagram shown is a structural schematic of the force sensing system of a puncture robot in one embodiment of this application, which includes an initial processing unit, an intermediate stage processing unit and an output unit, with each different unit corresponding to a different operation stage.

[0043] The initial processing unit is used to perform spatial size downsampling on the four-dimensional ultrasound sequence at the puncture site using a four-dimensional convolution module to obtain an initial feature map.

[0044] A four-dimensional ultrasound sequence can be represented as... Where T, D, H, and W represent the temporal length, spatial depth, spatial height, and spatial width of the feature map, respectively. The initial processing unit can contain a four-dimensional convolutional module (Conv4D), which is used to process the input ultrasound sequence. The spatial size is downsampled, for example, the spatial size is downsampled to 1 / 2 of the original size.

[0045] Intermediate processing units are used to extract spatiotemporal features from the initial feature map. The number of intermediate processing units can be adjusted according to the actual situation. The figure shows N intermediate processing units, which are the first to the Nth processing units. The first processing unit uses the initial feature map as the input feature map, and the output of the previous processing unit is used as the input feature map of the next processing unit. Each processing unit is used to extract the spatiotemporal features of the corresponding input feature map.

[0046] The output unit is used to pool the output feature map of the last intermediate stage processing unit and project it into three-dimensional components of force information. In one embodiment, the output unit includes a spatio-temporal global average pooling layer and a linear layer. The spatio-temporal global average pooling layer is used to perform spatio-temporal global average pooling (ST-GAP) on the output feature map of the last intermediate stage processing unit, and the linear layer is used to project the pooling result into three-dimensional components of force and torque. The force information may include force and torque.

[0047] The key to this application lies in the design of the intermediate processing unit. This unit incorporates an SLGA (Spatial Local Global Attention) submodule for spatial feature extraction and a TLGA (Temporal Local Global Attention) submodule for temporal feature extraction. These two submodules capture both temporal and spatial features of the image, accurately perceiving the puncture force information of the puncture robot. The combination of the SLGA and TLGA submodules is not limited; for example, ... Figure 2 As shown, the SLGA submodule and TLGA submodule are combined in any of the following ways: serial, parallel, or front-to-back residual connections.

[0048] The following sections will introduce these two key sub-modules separately.

[0049] Figure 3(a) shows a structural diagram of the SLGA submodule, Figure 3(b) shows a flowchart of key processing steps in the SLGA submodule, and Figure 3(c) shows a feature extraction diagram of key steps in the SLGA submodule. The SLGA submodule includes a four-dimensional convolution module, an attention module, and a matrix summation module.

[0050] In the first step, the input feature map SX of the SLGA submodule is input into three four-dimensional convolutional modules to generate the query matrix SQ, the key matrix SK, and the value matrix SV.

[0051] The input feature map SX can be either the feature map of the intermediate stage processing unit or the processed feature map.

[0052] Specifically, three four-dimensional spatiotemporal convolutional modules are used to process the feature maps input to SLGA. Spatial features are extracted, where T, C, D, H, and W represent the temporal length, number of channels, spatial depth, spatial height, and spatial width of the feature map, respectively. The query (SQ), key (SK), and value (SV) matrices of the SLGA are generated accordingly, and their calculation process is as follows:

[0053]

[0054]

[0055]

[0056] in, This represents the convolution kernel corresponding to the four-dimensional convolution module Conv4D, with a temporal dimension of 1,k. c and k d ×k h ×k w These represent the channel and spatial dimensions, respectively; therefore, the four-dimensional convolution module Conv4D here only contains spatial information.

[0057] Typically, the four-dimensional convolutional module Conv4D is defined as follows:

[0058]

[0059] Wherein, Conv3D represents three-dimensional spatial convolution. This represents a convolutional kernel that simultaneously contains spatial and temporal information, with temporal, channel, and spatial dimensions k, respectively. t k c and k d ×k h ×k w X (l-1) and X l Here, i and j represent the input and output of the four-dimensional convolution module, respectively, and K represents the channel index and temporal index. (l) (i) represents the convolution kernel of the i-th channel, K (l-1) (j) represents the input at time j.

[0060] The second step involves downsampling the bond matrix SK and the value matrix SV using a four-dimensional convolution module with a relatively small stride to generate a fine-grained bond matrix SK. fg Sum matrix SV fg The bond matrix SK with coarse granularity is generated by downsampling through a four-dimensional convolution module with a relatively large stride. cg Sum matrix SV cg .

[0061] Specifically, two more four-dimensional convolutional modules are used to further downsample the key matrix SK, generating a fine-grained key matrix SK. fg The bond matrix SK with coarse grain size cg Two four-dimensional convolutional modules are used to further downsample the value matrix SV, generating a fine-grained value matrix SV. fg and the value matrix SV of roughness granularity cgThis leads to a set of fine-grained key-value features for SK. fg SV fg And a set of coarse-grained key-value features for SK cg SV cg The calculation process is as follows:

[0062]

[0063]

[0064]

[0065]

[0066] In an optional implementation, the convolution kernel The stride can be the convolution kernel. 2 times, for example, and These represent spatial convolution kernels with strides of 1×2×2×2 and 1×4×4×4, respectively.

[0067] Step 3, SK bond matrix fg Sum matrix SV fg The query matrix SQ is processed by one of the attention modules to generate the attention matrix SA. fg SK bond matrix cg Sum matrix SV cg The query matrix SQ is processed by another attention module to generate the attention matrix SA. cg .

[0068] Specifically, based on the SQ obtained in the first and second steps, SK fg SV fg SK cg and SV cg Calculate the fine-grained attention matrix SA fg Attention matrix SA with coarse granularity cg .

[0069] In an optional implementation, the processing formula of the attention module satisfies:

[0070]

[0071] In the formula, Z is the output of the attention module, A, B, and E are the query matrix, key matrix, and value matrix of the input attention module, respectively, b is the feature length of the input key matrix, and Softmax() is the activation function.

[0072] At this point, the attention matrix SA in the third step fgand attention matrix SA cg The specific calculation formula is as follows:

[0073]

[0074]

[0075] In the formula, and SK fg and SK cg The characteristic length.

[0076] Step 4, Attention Matrix (SA) fg and attention matrix SA cg The attention matrix SA is generated after summing by the matrix summing module.

[0077] The formula for summation is as follows:

[0078] SA = SA fg +SA cg

[0079] The fifth step is to output the feature map SY after the attention matrix SA is downsampled by a four-dimensional convolution module containing only spatial convolution kernels.

[0080] Specifically, the SA is converted into the output feature map SY of SLGA using a Conv4D module containing only spatial convolution kernels, as follows:

[0081] SY=Conv4D(K s SA)

[0082] in This indicates a convolution kernel that contains only spatial information.

[0083] By using the SLGA submodule and going through the first to fifth steps above, multi-scale features can be aggregated in the spatial domain, thereby giving the system the ability to capture local details and global features of the puncture site.

[0084] Figure 4(a) shows a structural diagram of the TLGA submodule, Figure 4(b) shows a flowchart of key processing steps in the TLGA submodule, and Figure 4(c) shows a schematic diagram of feature map changes in key steps of the TLGA submodule. The TLGA submodule includes a four-dimensional convolution module, a temporal segmentation module, a window partitioning module, an attention module, a matrix summation module, and a concatenation module.

[0085] In the first step, the input feature map TX of the TLGA submodule is first processed by three four-dimensional convolutional modules to generate the query matrix TQ, the key matrix TK, and the value matrix TV.

[0086] The input feature map TX can be either the feature map of the intermediate stage processing unit or the processed feature map.

[0087] Similar to the first step of the SLGA submodule, the difference lies in that the first step of the SLGA submodule extracts spatial features, while the first step of the TLGA submodule extracts temporal features. Specifically, three four-dimensional spatiotemporal convolutional modules are used to process the feature map input to the TLGA. Extract time-domain features to generate the TLGA query TQ, key TK, and value TV matrices, respectively. The calculation process is as follows:

[0088]

[0089]

[0090]

[0091] in This indicates a convolution kernel in the corresponding convolution module that contains only temporal information.

[0092] The second step involves dividing the query matrix TQ, key matrix TK, and value matrix TV into equal parts in the time domain using a time-domain segmentation module to generate the first half of the time-domain query matrix TQ. f Key matrix TK f Sum Matrix TV f And the query matrix TQ in the second half of the time domain r Key matrix TK r Sum Matrix TV r .

[0093] Specifically, TQ, TK, and TV are equally divided along the time-domain axis to generate the matrix TQ in the first half of the time domain. f TK f and TV f And the matrix TQ in the second half of the time domain r TK r and TV r The superscripts f and r indicate that the matrix is ​​in the first half of the time domain and the second half of the time domain, respectively:

[0094] (TQ f TQ r =Split(TQ)

[0095] (TK f TK r =Split(TK)

[0096] (TV f TV r =Split(TV)

[0097] Split(·) represents the operation of equally dividing the input feature map along the temporal dimension.

[0098] The third step involves generating multiple non-overlapping spatiotemporal small windows for each matrix in the first and second half of the time domain after it is divided into windows.

[0099] Specifically, for TQ f TK f TV f TQ r TK r and TV r Formation of non-overlapping spatiotemporal small windows as well as The superscripts f and r indicate that the matrix is ​​in the first half of the time domain and the second half of the time domain, respectively. The subscripts ijkl indicate that the matrix is ​​within the (i,j,k,l)th spatiotemporal window, where i,j,k,l represent the indices of the spatiotemporal window in the four dimensions of time domain, spatial depth, spatial height, and spatial width, respectively. For example... TQ f The matrix within the (i,j,k,l)th small window.

[0100] The fourth step is to query the matrix within the same spatiotemporal window. Key matrix Sum matrix An attention matrix is ​​generated after processing by one of the attention modules. Query matrix in the same spatiotemporal small window Key matrix Sum matrix An attention matrix is ​​generated after processing by another attention module.

[0101] In an optional implementation, the processing formula of the attention module satisfies:

[0102]

[0103] In the formula, Z is the output of the attention module, A, B, and E are the query matrix, key matrix, and value matrix of the input attention module, respectively, b is the feature length of the input key matrix, and Softmax() is the activation function.

[0104] At this point, in the fourth step, the temporal attention matrix from front to back is calculated for each small window. and the temporal attention matrix from back to front This process can be represented as:

[0105]

[0106]

[0107] in and They represent and The characteristic length.

[0108] Step 5: Attention matrix within all spatiotemporal windows The attention matrix TA is generated after summation by the matrix summation module. fr Attention matrix within all spatiotemporal small windows The attention matrix TA is generated after summation by the matrix summation module. rf .

[0109] The specific calculation formula is as follows:

[0110]

[0111]

[0112] Step 6, Attention Matrix (TA) fr Attention Matrix After being stitched together by the stitching module, a complete attention matrix TA is generated.

[0113] TA = Concat(TA) fr ,TA rf )

[0114] Concat(·) represents the concatenation operation of two feature maps along the time domain dimension.

[0115] Step 7: The attention matrix TA is downsampled by a four-dimensional convolutional module containing only temporal convolutional kernels and then output as a feature map TY.

[0116] Specifically, the TA is converted into the output feature map TY of TLGA using a Conv4D module containing only temporal convolutional kernels, as follows:

[0117] TY = Conv4D(K t ,TA)

[0118] in This represents a convolution kernel that contains only time-domain information.

[0119] By utilizing the TLGA submodule and going through the first to seventh steps above, the system's ability to capture minute deformations of the puncture site in the time domain is enhanced through local window learning and global bidirectional learning processes.

[0120] Therefore, by combining the SLGA and TLGA submodules, accurate force sensing can be achieved, thereby ensuring that the puncture robot can quickly and accurately intervene in the puncture procedure.

[0121] In some implementations, additional auxiliary modules can be added to each intermediate processing unit to assist feature extraction. The module formed by combining the SLGA and TLGA sub-modules described above is defined as the STLGA (Spatial-Temporal Local Global Attention) module. A Positional Embedding (PE) module and a Feed Forward Network (FFN) module can be added to further process the input and output features of the STLGA module. The module combining the STLGA module, the Positional Embedding module, and the Feed Forward Network module is defined as the STLG block. Figure 5(a) shows a schematic diagram of the STLG block structure, where the Positional Embedding module is used to process the input feature map X of the STLG block. l-1 After performing depth-aware four-dimensional convolution processing, the output feature map PE(X) is completed. l-1 Input feature map X l-1 and output feature map PE(X) l-1 Superimposed feature map X1 l-1 After inputting into the STLGA module for spatiotemporal feature extraction, a feature map STLGA(BN(X1) is generated. l-1 Feature map X1 l-1 and feature map STLGA(BN(X1) l-1 Superimposed feature map X2 is generated. l-1 The feature map FFN(BN(X2)) is then output after the feedforward network module is used to enhance the feature spatial domain. l-1 Feature map FFN(BN(X2) l-1 )) and feature map X2 l-1 The corresponding STLG block output feature map X is generated by superposition. l The subscript l indicates the sequence number index of the STLG block.

[0122] Therefore, the calculation process of the l-th STLG module is as follows:

[0123] X1 l-1 =PE(X) l-1 )+X l-1

[0124] X2 l-1 =STLGA(BN(X1) l-1 ))+X1 l-1

[0125] X l =FFN(BN(X2) l-1 ))+X2 l-1

[0126] Among them, X l-1 and X l Let x represent the input and output of the l-th STLG module, respectively. BN(x) represents batch regularization.

[0127] Specifically, a PE can be composed of a depth-aware Conv4D module, and the specific calculation process is as follows:

[0128] PE(X)=DWConv4D(X)+X

[0129] Here, DWConv4D represents the depth-aware Conv4D module.

[0130] Specifically, FFN is constructed from two cascaded spatiotemporal Conv4Ds to enhance the feature spatial domain.

[0131] In some implementations, as shown in Figure 5(b), each stage processing unit consists of multiple STLG blocks, and the multiple STLG blocks are connected in series, which can enhance the effect of feature extraction.

[0132] like Figure 6 The diagram illustrates the feature processing of a force sensing system comprising four intermediate processing units. Each intermediate processing unit contains an STLG block, corresponding to steps 1, 2, 3, and 4 respectively. In each step, the signal is input to the STLG block, first undergoing conventional patch embedding processing, and then input back into the STLG block. Finally, pooling and linear processing are used to generate the spatial three-dimensional components of the force and torque.

[0133] In one embodiment, the force sensing system described above can be applied to a puncture robot that assists in prostate puncture. Prostate cancer is the most common malignant tumor in the male reproductive system and one of the leading causes of cancer death in men. Applying the force sensing system to a puncture robot that assists in prostate puncture enables effective local-global integrated modeling of the spatiotemporal deformation of the prostate, thereby achieving rapid and accurate force sensing, which allows the robot to perform prostate puncture operations more precisely.

[0134] Example 2

[0135] This application also relates to a training method for the force sensing system of the puncture robot in Embodiment 1. The training method uses historical data as training data, specifically using the four-dimensional ultrasound sequence at the puncture site as input and the three-dimensional components of the force information corresponding to the puncture robot as output to train the force sensing system, so that the output of the force sensing system converges to a preset degree.

[0136] Specifically, during training, the mean squared error (MSE) can be used as the loss function for training the power-aware system. The specific calculation process is as follows:

[0137]

[0138] in F represents the loss value. i and T represents the actual force and predicted force in the x, y, and z directions of the spatial domain, respectively. j and These represent the actual torque and the predicted torque in the x, y, and z directions of the spatial domain, respectively.

[0139] For example, by inputting N sets of ultrasound body sequences-force and torque pairs from four-dimensional prostate biopsy punctures, and continuously training the force sensing system constructed in Example 1 until the loss value converges, a well-trained force sensing system can be obtained.

[0140] Example 3

[0141] This application also relates to a force sensing method for a puncture robot. During robot-assisted puncture, the four-dimensional ultrasound sequence at the current puncture site is input into a force sensing system trained using the method described in Example 2 to obtain the three-dimensional components of the force information for robot-assisted puncture. This method enables rapid and accurate acquisition of the robot's puncture force information, thereby ensuring rapid and accurate intervention of the puncture robot during the puncture operation.

[0142] The following compares the predictive performance of the force sensing system trained in Example 2 with that of various existing predictive models.

[0143] Comparative Example 1

[0144] VFS was implemented using the ResNet4d method described in (Med.Image Anal..64(2020),pp.101730). Specific parameters were: batch size of 8 and learning rate of 0.001 during training.

[0145] Comparative Example 2

[0146] VFS is implemented using the facResNet4d method in (Med.Image Anal..64(2020),pp.101730). The specific parameters are the same as those in Comparative Example 1.

[0147] Comparative Example 3

[0148] VFS is implemented using the ResNet3dGRU method in (Med.Image Anal..64(2020),pp.101730). The specific parameters are the same as those in Comparative Example 1.

[0149] Comparative Example 4

[0150] VFS is implemented using the ConvGRUResNet3d method in (Med.Image Anal..64(2020),pp.101730). The specific parameters are the same as those in Comparative Example 1.

[0151] Force sensing accuracy is evaluated using the MAE values ​​of force and torque, specifically defined as:

[0152]

[0153]

[0154] Among them, MAE F and MAE T These represent the MAE values ​​for force and torque, respectively.

[0155] Force sensing accuracy was tested using a dataset of four-dimensional ultrasound sequences from a prostate phantom and corresponding puncture forces and torques. All ultrasound sequences in this example were obtained via transabdominal ultrasound. Table 1 lists the MAE values ​​obtained by each algorithm. F and MAE T The mean and standard deviation of the values. As can be seen from Table 1, when performing VFS on a transabdominal ultrasound prostate phantom, the MAE provided in Example 2... F and MAE T The mean and standard deviation of the proposed method are all lower than those of other methods, indicating that the method proposed in this invention has the highest force sensing accuracy among all the compared methods.

[0156] Table 1. Comparison of force-sensing MAE values ​​for each method on the transabdominal imaging phantom dataset.

[0157]

[0158]

[0159] Force sensing accuracy was tested using a dataset of four-dimensional ultrasound sequences of the prostate gland in beagle dogs and corresponding puncture forces and torques. All ultrasound sequences in this example were obtained via transabdominal ultrasound. Table 2 lists the MAE values ​​obtained by each algorithm. F and MAE T The mean and standard deviation of the values. As can be seen from Table 2, when performing VFS on the prostate of a beagle dog via transabdominal ultrasound, the MAE provided in Example 2... F and MAE T The mean and standard deviation of the proposed method are all lower than those of other methods, indicating that the method proposed in this invention has the highest force sensing accuracy among all the compared methods.

[0160] Table 2 Comparison of force-sensing MAE values ​​for each method on the beagle dog dataset obtained through transabdominal imaging.

[0161]

[0162] In summary, this invention designs a force sensing system for a puncture robot. Its unique SLGA submodule can aggregate multi-scale features in the spatial domain, thereby endowing the system with the ability to fully capture local details and global features of the puncture site. The TLGA module, through local window learning and global bidirectional learning processes, enhances the system's ability to capture minute deformations of the puncture site in the temporal domain. The combination of these two modules enables the system to achieve accurate force sensing at the puncture site. Furthermore, the SLGA and TLGA modules proposed in this application incorporate many lightweight designs, which reduce the system's inference complexity and ensure rapid force sensing. This application is expected to provide a valuable reference method for the VFS field and become a feasible solution for clinical puncture force estimation.

[0163] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification. It should be noted that the terms "in one embodiment," "for example," and "as in another example" in this application are intended to illustrate the application and are not intended to limit the application.

[0164] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are quite specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the protection scope of this application.

Claims

1. A force sensing system for a puncture robot, characterized in that, Force sensing systems include: The initial processing unit is used to perform spatial size downsampling on the four-dimensional ultrasound sequence at the puncture site using a four-dimensional convolution module to obtain an initial feature map. Intermediate stage processing unit, used to extract spatiotemporal features of the initial feature map; The output unit is used to pool the output feature map of the last intermediate stage processing unit and project it into three-dimensional components of force information. The intermediate processing unit includes at least an SLGA submodule and a TLGA submodule: The SLGA submodule includes a four-dimensional convolution module, an attention module, and a matrix summation module. The input feature map SX of the SLGA submodule is input into the three four-dimensional convolution modules to generate a query matrix SQ, a key matrix SK, and a value matrix SV. The key matrix SK and the value matrix SV are downsampled by the four-dimensional convolution modules with relatively small strides to generate a fine-grained key matrix SK. fg Sum matrix SV fg The bond matrix SK with coarse granularity is generated by downsampling through a four-dimensional convolution module with a relatively large stride. cg Sum matrix SV cg ; bond matrix SK fg Sum matrix SV fg The query matrix SQ is processed by one of the attention modules to generate the attention matrix SA. fg SK bond matrix cg Sum matrix SV cg The query matrix SQ is processed by another attention module to generate the attention matrix SA. cg Attention Matrix SA fg and attention matrix SA cg After summing by the matrix summing module, the attention matrix SA is generated. The attention matrix SA is downsampled by the four-dimensional convolution module containing only spatial convolution kernels and the feature map SY is output. The TLGA submodule includes a four-dimensional convolution module, a temporal segmentation module, a windowing module, an attention module, a matrix summation module, and a concatenation module. The input feature map TX of the TLGA submodule is first processed by three four-dimensional convolution modules to generate a query matrix TQ, a key matrix TK, and a value matrix TV. Then, it is processed by the temporal segmentation module to divide the query matrix TQ into the first half of the temporal domain. f Key matrix TK f Sum Matrix TV f And the query matrix TQ in the second half of the time domain r Key matrix TK r Sum Matrix TV r Each matrix in the first and second half of the time domain is divided into multiple corresponding spatiotemporal small windows by the window partitioning module. The query matrix of the same spatiotemporal small window Key matrix Sum matrix An attention matrix is ​​generated after processing by one of the attention modules. Query matrix in the same spatiotemporal small window Key matrix Sum matrix An attention matrix is ​​generated after processing by another attention module. The superscripts f and r indicate that the matrix is ​​in the first half of the time domain and the second half of the time domain, respectively; the subscripts ijkl indicate that the matrix is ​​in the (i,j,k,l)th spatiotemporal window, where i,j,k,l represent the partition indices of the spatiotemporal window in the four dimensions; attention matrices are used in all spatiotemporal windows. The attention matrix TA is generated after summation by the matrix summation module. fr Attention matrix within all spatiotemporal small windows The attention matrix TA is generated after summation by the matrix summation module. rf Attention Matrix TA fr and attention matrix After being concatenated by the concatenation module, a complete attention matrix TA is generated. The attention matrix TA is then downsampled by a four-dimensional convolution module containing only temporal convolution kernels to output a feature map TY.

2. The force sensing system according to claim 1, characterized in that, The processing formula for each attention module is as follows: In the formula, Z is the output of the attention module, A, B, and E are the query matrix, key matrix, and value matrix of the corresponding attention module, respectively, and b is the feature length of the input key matrix.

3. The force sensing system according to claim 1, characterized in that, The SLGA submodule and the TLGA submodule are combined in any of the following ways: serial, parallel, or sequential residual connection.

4. The force sensing system according to claim 1, characterized in that, The SLGA submodule and TLGA submodule constitute an STLGA module. The intermediate stage processing unit includes at least one STLG block or multiple cascaded STLG blocks. Each STLG block includes a position encoding module, the STLGA module, and a feedback forward network module. The position encoding module is used to process the input feature map X of the STLG block. l-1 After performing depth-aware four-dimensional convolution processing, the output feature map PE(X) is completed. l-1 The input feature map X l-1 and the output feature map PE(X) l-1 Superimposed feature map X1 l-1 The STLGA module is then used to extract spatiotemporal features, generating a feature map STLGA(BN(X1)). l-1 The feature map X1 l-1 and the feature map STLGA(BN(X1) l-1 Superimposed feature map X2 is generated. l-1 The feature map FFN(BN(X2) is then output after being input into the feedforward network module to enhance the feature spatial domain. l-1 The feature map FFN(BN(X2)) l-1 )) and the feature map X2 l-1 The output feature map X corresponding to the STLG block is generated by superposition. l The subscript l indicates the sequence number index of the STLG block.

5. The force sensing system according to any one of claims 1 to 4, characterized in that, The force sensing system includes multiple intermediate stage processing units, namely the first to the Nth stage processing units. The first stage processing unit uses the initial feature map as the input feature map, and the output of the previous stage processing unit is used as the input feature map of the next stage processing unit. Each stage processing unit is used to extract the spatiotemporal features of the corresponding input feature map.

6. The force sensing system according to claim 1, characterized in that, The output unit includes a spatiotemporal global average pooling layer and a linear layer. The spatiotemporal global average pooling layer is used to perform spatiotemporal global average pooling operation on the output feature map of the intermediate stage processing unit, and the linear layer is used to project the pooling result into three-dimensional components of force information.

7. The force sensing system according to claim 1, characterized in that, In the SLGA submodule, the stride of the four-dimensional convolution module that generates the coarse granularity matrix is ​​twice the stride of the four-dimensional convolution module that generates the fine granularity matrix.

8. The force sensing system according to claim 1, characterized in that, The puncture robot is a puncture robot that assists in prostate puncture.

9. A training method for a force-sensing system of a puncture robot as described in any one of claims 1 to 8, characterized in that, The force sensing system is trained by using the four-dimensional ultrasound sequence at the puncture site in historical data as input and the three-dimensional component of the force information corresponding to the puncture robot as output, so that the output of the force sensing system converges to a preset degree.

10. A force sensing method for a puncture robot, characterized in that, The four-dimensional ultrasound sequence at the current puncture site is input into the force sensing system trained using the training method described in claim 9 to obtain the three-dimensional components of the force information for the puncture robot-assisted puncture.

Images