Real-time surgical instrument detection method based on regional collaborative convolution and non-parameter sensing block

Abstract

The invention discloses a real-time surgical instrument detection method based on regional collaborative convolution and a non-parameter sensing block. The method comprises the following steps: step 1, proposing a new lightweight convolution mode regional collaborative convolution; firstly, the number of convolution channels is determined according to the participation ratio, the convolution channels are divided into g groups to be subjected to grouping convolution, results are spliced along the channels and then spliced with non-participated parts, and a final feature map is obtained. 2, based on the regional collaborative convolution proposed in the step 1, introducing residual connection and 1 * 1 convolution capable of flexibly changing the number of channels, and constructing a regional collaborative convolution block; and step 3, proposing a non-parameter sensing block, generating an attention map through standardization of a spatial dimension mean value, difference square and variance, and multiplying the attention map with an original feature map after Sigmoid mapping to realize feature focusing. And 4, constructing a lightweight surgical instrument detection network, replacing a part of convolution modules with regional collaborative convolution blocks, and adding a non-parameter sensing block between the Backbone and the Neck to improve the feature focusing capability and the model performance.

Description

Technical Field

[0001] The present invention relates to the use of region-coconvolution, parameter-free perception blocks and real-time surgical instrument detection. Background Art

[0002] Surgical instrument detection is of great significance in the medical field and can significantly improve the safety and efficiency of surgery. On the one hand, by accurately locating and identifying surgical instruments, detection technology can assist surgeons in complex surgical environments to avoid instrument loss or misuse and reduce the occurrence of medical accidents. On the other hand, it can also provide a basis for the development of surgical robots, promote the realization of intelligent surgery, and improve the accuracy and success rate of surgical operations.

[0003] In recent years, image classification technology in surgical instrument detection has been widely studied and applied. With the development of deep learning technology, two-stage target detection algorithms represented by Faster R-CNN and Mask R-CNN, and single-stage target detection algorithms represented by SSD (Single Shot Multibox Detector) and YOLO (You Only Look Once) have been introduced into surgical instrument detection. These research methods mainly focus on the following two aspects: first, extracting higher-level abstract expressions from the underlying features of the image; second, classifying and locating surgical instruments based on the extracted features. However, these algorithms still have certain defects: the two-stage detection method improves the detection accuracy by separating candidate region generation and detection, but also increases the calculation steps and time consumption. Therefore, in applications with high real-time requirements such as surgical instrument detection, the two-stage algorithm is difficult to cope with; in order to improve the detection speed, the single-stage method integrates all the steps of target detection into an end-to-end process, which greatly improves the calculation efficiency, but due to the lack of fine processing of the candidate region, the accuracy is often sacrificed. Therefore, incorrect classification and positioning will bring great threats to the life and health of patients.

[0004] Recently, the explosive development of various large models has pushed the field of object detection to a new height. For example, the DETR (DEtection TRansformers) model led by the Transformer architecture has greatly improved the accuracy of object detection through the self-attention mechanism. DETR abandons the traditional anchor mechanism by expressing the object detection task as a sequence-to-sequence learning problem, directly predicts the bounding box and category of the target, thereby reducing complex post-processing steps; models based on the hierarchical Transformer structure such as SwinTransformer further optimize the extraction of multi-scale features, making these large models perform well in dealing with complex scenes and small target detection. In addition, the language model BERT (Bidirectional Encoder Representations from Transformers) based on the Transformer architecture captures the complex relationship between the target and other objects or backgrounds through the self-attention mechanism, greatly improving the accuracy of object detection, especially in detection tasks in complex scenes. However, despite the excellent performance of these large models in terms of accuracy, their deployment faces many challenges in specific application scenarios such as medical device detection. Surgical instrument detection in medical scenarios often requires high real-time performance and high reliability. The high computational complexity and huge parameters of large models make it difficult to run in real time on resource-constrained medical equipment. The inference speed is far slower than that of traditional small models. The delays that may be caused will affect the efficiency and safety of the operation, thereby having a certain impact on the patient's safety and the credibility of the medical team.

[0005] In summary, in order to solve the problem of high-precision models being difficult to deploy on medical devices with limited computing resources, we have weighed the number of model parameters and detection accuracy, and proposed a real-time surgical instrument detection method based on regional collaborative convolution and parameter-free perception blocks. First, a new convolution method, regional collaborative convolution, is proposed. This convolution method effectively avoids the problem that the number of layers of traditional convolutional neural network models increases with the increase of similar redundant information between channels, thereby causing excessive repeated calculations, and effectively reduces the amount and complexity of convolution operations. Second, based on regional collaborative convolution, a new module, regional collaborative convolution block, is proposed. Compared with the traditional convolution module, it increases the network expression ability while improving the computational efficiency. In addition, the shortcut structure is introduced in this module, which effectively alleviates the problem of gradient vanishing of the model, reduces information loss, and makes the model easier to optimize. Third, a parameter-free perception block is proposed. By simulating the activation process of biological neurons, the statistical information of the input features (such as the mean, etc.) is directly used to construct the energy function, so that the important parts of the feature map obtain higher activation values, and the unimportant parts are suppressed, thereby improving the feature representation ability without introducing additional learnable parameters. Finally, based on the proposed regional collaborative convolution block and parameter-free perception block, a new lightweight target detection network architecture is constructed for real-time detection of surgical instruments. This network architecture not only has a small number of parameters, which is convenient for the deployment of mobile medical devices with limited computing, but also maintains high accuracy to ensure surgical safety. Summary of the invention

[0006] In order to solve the problem that the existing high-precision target detection model has too many parameters, which makes it difficult to deploy mobile medical devices with limited computing resources, this paper invents a real-time surgical instrument detection method based on regional collaborative convolution and parameter-free perception blocks. The invention mainly includes: regional collaborative convolution avoids the problem of excessive repeated calculations caused by the redundancy of similar information between channels in traditional convolutional neural network models, and on this basis, introduces 1×1 convolution and residual connection to propose a new regional collaborative convolution block. Compared with the traditional convolution module, it increases the network expression ability while improving the computational efficiency. The parameter-free perception block constructs an energy function for the features extracted by the backbone network to improve the network feature representation ability. A new lightweight target detection network architecture is constructed based on regional collaborative convolution blocks and parameter-free perception blocks.

[0007] This paper proposes a real-time surgical instrument detection method based on regional co-convolution and parameter-free perception block. The main steps are as follows:

[0008] Step 1: Propose a new lightweight convolution method, regional collaborative convolution. Assume that the width, height, and number of channels of the feature map input to this convolution are W, H, and C respectively. We set a convolution participation ratio r (0 < r < 1), that is, in this feature map, only the first C×r channels will participate in the subsequent operations. Subsequently, for the C×r channels obtained by calculation, we perform grouped convolution operations on them, divide them into g groups. After each group's calculation is completed, the resulting feature maps are concatenated along the channels to obtain the result feature map of the part participating in the operation. For the remaining C×(1 - r) channels of the feature map that did not participate in the grouped convolution operation originally, we use the identity mapping method to directly concatenate them with the result feature map of the part participating in the operation along the channels to obtain the final result feature map of the regional collaborative convolution.

[0009] Step 2: Based on the regional collaborative convolution proposed in Step 1, propose a lightweight feature extraction module, the regional collaborative convolution block. First, for the feature map input to this module, we will make a backup first, and then perform the local intelligent convolution calculation in Step 1 on this feature map. Then, for the resulting result feature map, we first perform a 1×1 convolution operation to expand the number of channels to map the input features to a higher-dimensional space to capture more feature information. Then, the obtained result is subjected to batch normalization processing and ReLU activation function calculation to suppress irrelevant information and enhance useful features. After that, we perform another 1×1 convolution operation to reduce the number of channels to restore the number of channels, and map the high-dimensional features back to the original dimension, further enhancing the network's non-linear modeling ability. Finally, the initial feature map of the initial backup is summed with the result feature map obtained through calculation through residual connection to obtain the final result feature tensor, which is used as the output of this regional collaborative convolution block.

[0010] Step 3: Propose a parameter-free perception block that can enhance the model performance without introducing additional learning parameters. When a feature map is input to this module, first, the mean value is calculated, and the features of each channel are averaged through the spatial dimensions (height and width) of the feature map. Then, the module calculates the square of the difference between each feature and its corresponding mean value, so as to obtain the difference degree between each pixel point in the feature map and its spatial context. Then, this module standardizes the difference degree by dividing these squared differences by the corrected variance to generate an attention map. This attention map represents the significance of each pixel point in its所在空间. Finally, this attention map is mapped to between 0 and 1 through the Sigmoid activation function to obtain the final attention weight map, and it is multiplied point by point with the original input feature map to obtain the final result feature map.

[0011] Step 4: Use the region collaborative convolution block and parameterless perception block proposed in Step 2 and Step 3 to construct a brand-new lightweight surgical instrument detection network. First, in the Backbone part for feature extraction and the Neck part for multi-level feature fusion, we discard some convolution modules with relatively large computational complexity and replace them with region collaborative convolution blocks with less computational complexity. Second, we place the parameterless perception block at the connection between the Backbone and Neck parts to better focus on the features extracted by the network. Description of the Drawings

[0012] Figure 1 . Method Framework Process

[0013] Figure 2 . Flowchart of the Region Collaborative Convolution Block

[0014] Figure 3 . Computational Flowchart of the Parameterless Perception Block Detailed Implementation Manner

[0015] Step 1: Propose a convolution method with relatively small computational complexity, namely region collaborative convolution.

[0016] For the feature map I input to this convolution, assume its tensor size is W×H×C (where W, H, and C represent the width, height, and number of channels of the feature map respectively).

[0017] (1) First, we set the convolution participation rate r (0 < r < 1), that is, only the first 1 / r number of channels participate in the convolution calculation. Therefore, the number of channels participating in the convolution calculation is:

[0018] C part = C × r

[0019] (2) To further reduce the number of parameters, we adopt grouped convolution as our main convolution method, and set the number of groups to g. Therefore, the number of channels in each group is:

[0020]

[0021] (3) For the feature I, let the convolution kernel be K, and the convolution kernel size be k h ×k w , and the bias term be b. The output feature map is O. Then, for the convolution operation of the i-th group, the calculation formula for the output feature map is:

[0022]

[0023] (4) For the outputs of all groups, the final output feature map O will be obtained by concatenating the outputs O i of each group:

[0024] O(x,y)=concat(O1(x,y),O2(x,y)...O g (x, y)

[0025] Step 2: Based on the region co-convolution proposed in step 1, we combine it with two 1×1 convolutions with different channel numbers and residual connections to construct a computationally lightweight region co-convolution block as the main module of our lightweight object detection model.

[0026] Assume that the input feature map of this module is 1, and its tensor size is W×H×C (where W, H, and C represent the width, height, and number of channels of the feature map, respectively).

[0027] (1) First, we back up the input features and perform two different operations on them: input the original feature map into the regional collaborative convolution proposed in step 1 to calculate the feature map I conv ; The backup features are directly connected through residual connection to obtain I shortcut .

[0028] (2) Next, we will perform feature I obtained through regional co-convolution conv A 1×1 convolution calculation is performed to enlarge the number of channels to map the features to a higher-dimensional space to capture more feature information. The enlargement factor is set to α. The calculation formula for this operation is as follows:

[0029]

[0030] Among them, c represents the index of the input channel, k represents the index of the output channel, K1(c, k) is the weight of the 1×1 convolution kernel, and b1(k) is the bias term.

[0031] (3) Then we perform batch normalization on the feature map after expanding the number of channels and input the ReLU activation function, which not only stabilizes the gradient flow and alleviates the gradient explosion problem, but also introduces nonlinearity and enhances the feature expression ability of the model. The main calculation process is as follows:

[0032]

[0033] The above is the calculation process of batch normalization operation, μ k and Represent the mean and variance on channel k, γ k and β k are the scaling and offset parameters of channel k, respectively, and ∈ is a small constant used to avoid the denominator being 0. The ReLU activation function calculation formula is as follows:

[0034] I activated (x, y, k) = max(0, I bn(x, y, k)

[0035] (4) After that, the feature I after the activation function activate Perform a 1×1 convolution calculation to restore the number of channels and map the high-dimensional features back to the original dimension, further enhancing the nonlinear modeling ability of the network. Assuming the convolution kernel is K2 and the bias term is b2, the calculation process is as follows:

[0036]

[0037] (5) Finally, the feature map after the number of channels is restored and the backup feature map initially connected through the residual connection is summed to obtain the final output of this module:

[0038] O(x,y,c)=I reduce (x,y,c)+I shortcut (x,y,c)

[0039] Step 3: Propose a parameter-free perception block that enhances the model's feature learning ability without adding additional learnable parameters. This module simulates the process of neuron activation, can effectively perceive the significance of local features in 3D space during the calculation process, and dynamically adjust the importance distribution in the feature map so that the model can better focus on key parts.

[0040] (1) First, for each channel of the input feature map I (tensor size is W×H×C), we calculate the mean μ in the spatial dimension (i.e., height and width dimension) to determine the overall brightness level or intensity center of the feature map, laying the foundation for subsequent difference calculations. The calculation formula is:

[0041]

[0042] (2) Then calculate the square of the deviation between each value in the input feature map and its mean to reflect the difference between each position in the feature map relative to the whole, and calculate the square of the mean deviation σ in the spatial dimension 2 , the calculation process is as follows:

[0043]

[0044] (3) Then, use the mean and mean deviation square obtained above to assign weights to each position in the feature map. This weight value is normalized to adjust all deviation values ​​to a uniform range, making subsequent processing more stable. The weight calculation formula is:

[0045]

[0046] (4) Finally, the calculated weights are limited to the range of [0, 1] through the Sigmoid activation function and multiplied element by element with the original feature map to obtain the final output feature map O. This step enhances the useful information in the feature map by emphasizing or suppressing specific features, while suppressing redundant or noisy information:

[0047]

[0048] Step 4: Use the regional collaborative convolution block and parameter-free perception block proposed in steps 2 and 3 to build a new lightweight object detection model. The model is mainly divided into three parts: the Backbone network responsible for input image feature extraction, the Neck network responsible for aggregating and optimizing multi-scale features, and the Head detection head responsible for generating the final result.

[0049] (1) In the Backbone part, we use the CBS module (including a 3×3 convolution, batch normalization calculation and a SiLU activation function operation) and regional collaborative convolution to perform calculations alternately. For the image input into the Backbone network, a CBS module operation is performed first, followed by 4 combined calculations of the CBS module and regional collaborative convolution. Among them, downsampling and channel number expansion are performed after each CBS calculation. A parameter-free perception block is connected after the last regional collaborative convolution block to focus on the key parts of the previously extracted features.

[0050] (2) In the Neck part, we use the Feature Pyramid Network (FPN) and the Path Aggregation Network (PAN) to further fuse the features output by Backbone, and replace the ordinary convolution module with the regional collaborative convolution block we proposed. First, the feature map is fused layer by layer from high to low through FPN to obtain the intermediate feature map:

[0051]

[0052] Then, PAN is used to fuse features layer by layer from low to high to obtain the final feature map:

[0053]

[0054] (3) The Head part mainly performs target classification and positioning prediction on the feature map output by the Neck. At each prediction layer, the feature maps of different scales correspond to Generate category, bounding box, and confidence predictions through convolution operations:

[0055]

[0056] Among them, σ is the Sigmoid function, which is used to normalize the output.

[0057] (4) The loss functions used in training mainly include classification loss, bounding box regression loss, and confidence loss. We use P and S to represent the model’s predicted probability and true label, respectively. Among them, the classification loss L cls and confidence loss L conf Both use binary cross entropy loss:

[0058]

[0059] Bounding box regression loss L box The GIoU loss function is used:

[0060]

[0061] The final total loss is the weighted sum of the above three losses:

[0062]

[0063] in, and are the weight coefficients of each loss respectively.

[0064] After deploying the real-time surgical instrument detection method based on regional collaborative convolution and parameter-free perception blocks proposed in this article on medical equipment with limited computing resources, the inference speed and instrument positioning accuracy have been greatly improved compared with traditional models.

Claims

1. A real-time surgical instrument detection method based on regional collaborative convolution and parameter-free perception block, characterized by: At least the following steps are included: Step 1: A new lightweight convolution method, regional collaborative convolution, is proposed. For the input feature map, the number of channels participating in the convolution calculation is first calculated according to the convolution participation ratio r, and then the channels participating in the calculation are divided into g groups for group convolution. After each group of calculations is completed, each feature map obtained is spliced ​​along the channel to obtain the result feature map of the part participating in the operation. Finally, the remaining part of the obtained operation result feature that does not participate in the convolution calculation is spliced ​​along the channel to obtain the final result feature map of the regional collaborative convolution. Step 2: Based on the regional collaborative convolution proposed in step 1, a lightweight feature extraction module regional collaborative convolution block is proposed. For the input feature map, the regional collaborative convolution proposed in step 1 is first calculated, and then the result is successively calculated by a 1×1 convolution to expand the number of channels and a 1×1 convolution to reduce the number of channels. Finally, the result is summed with the initial feature map initially backed up to obtain the output feature. Step 3: Propose a parameter-free perception block that can enhance model performance without introducing additional learning parameters. First, the input feature map is used to calculate the mean of the features of each channel in the spatial dimension. Then, the square of the difference between each feature and its corresponding mean is calculated. Then, the difference is normalized by dividing these squared differences by the corrected variance to generate an attention map. Finally, the attention map is mapped to between 0 and 1 through the Sigmoid activation function to obtain the final attention weight map, which is then multiplied point by point with the original input feature map to obtain the final result feature map. Step 4: Using the regional collaborative convolution blocks and parameter-free perception blocks proposed in steps 2 and 3, we built a new lightweight surgical instrument detection network. First, in the Backbone part used for feature extraction and the Neck part for multi-level feature fusion, we abandoned some convolution modules with large computational complexity and replaced them with regional collaborative convolution blocks with less computational complexity; secondly, we placed the parameter-free perception block at the connection between the Backbone and Neck parts to better focus on the features extracted by the network.

2. The real-time surgical instrument detection method based on regional collaborative convolution and parameter-free perception block according to claim 1 is characterized in that: In step 1, a new lightweight convolution method, regional collaborative convolution, is proposed. Assuming that the width, height and number of channels of the feature map input to the convolution are W, H and C respectively, we set a convolution participation ratio r, that is, in the feature map, only the first C×r channels will participate in the subsequent operations. Then, for the calculated C×r channels, we perform grouped convolution operations on them and divide them into g groups. After each group of calculations is completed, each feature map is spliced ​​along the channel to obtain the result feature map of the part participating in the operation. For the feature maps of the channels that did not originally participate in the grouped convolution operation, we use the identity mapping method to directly splice them with the result feature maps of the part participating in the operation along the channel to obtain the final result feature map of the regional collaborative convolution.

3. The real-time surgical instrument detection method based on regional collaborative convolution and parameter-free perception block according to claim 1 is characterized in that: In step three, a parameter-free perception block is proposed that can enhance model performance without introducing additional learning parameters. When a feature map is passed into this module, the mean calculation is first performed to average the features of each channel through the spatial dimension of the feature map. Then, the module calculates the square of the difference between each feature and its corresponding mean, thereby obtaining the difference between each pixel in the feature map and its spatial context. Then, this module normalizes the difference by dividing these squared differences by the corrected variance to generate an attention map, which represents the significance of each pixel in its space. Finally, the attention map is mapped to between 0 and 1 through the Sigmoid activation function to obtain the final attention weight map, which is then multiplied point by point with the original input feature map to obtain the final result feature map.

Images