Method for accurately identifying surgical instrument in real time

Abstract

The invention provides a real-time accurate recognition method for a surgical instrument, and the method comprises the following steps: 1) collecting pictures shot by the surgical instrument at various angles, and making a data set; (2) non-absolute mosaic and HSV are combined with data to enhance online pre-processing pictures, and the diversity of training pictures is increased; 3) adding label regularization and category weight coefficients, increasing the generalization performance of the identification network and balancing the identification accuracy of different categories; 4) adding a SimAM attention mechanism into a Bottleneck module of a backbone network in the YOLOV5 network model to enhance the learning ability of the network, and transmitting an obtained data set into the designed network model for training and learning; 5, the obtained weight file is stored after the appropriate epoch is trained, and 6, untrained data are tested through the weight of the weight file, and the surgical instrument is recognized.According to the technical scheme, the real-time performance and the accuracy are both achieved.

Description

Technical Field

[0001] The invention relates to a method for real-time and accurate identification of surgical instruments. Background Art

[0002] With the development of computer vision, deep learning has gradually penetrated into all aspects of medical care.

[0003] In the process of development, target detection technology is gradually divided into two-stage algorithms based on region proposal and one-stage algorithms based on regression. Among the two-stage algorithms, Faster R-CNN is more suitable for surgical instrument target detection, while among the one-stage target detection algorithms, YOLO series and SSD are more suitable for surgical instrument target detection.

[0004] In 2015, Joseph Redmon, Ali Farhadi and others proposed the original YOLO (You only look once) model, an object detection system based on global image information for prediction. Due to its good performance and scalability, it is widely used in the industrial field, and the iteration is also very rapid, and the performance of the model is getting better and better.

[0005] In order to solve the problem of multi-target counting, Egi Yunus et al. used drone footage of a greenhouse tomato dataset containing three categories (red tomatoes, green tomatoes and flowers) to train and test the counting model through YOLOv5 and Deep Sort cutting-edge deep learning algorithms, achieving a balance between speed and accuracy. As a single target detection method cannot complete the multi-target statistical counting task, and in order to simultaneously improve the detection accuracy and speed, Wen Nu et al. proposed a mobile target tracking and multi-lane vehicle counting model DCN-Mobile-YOLO based on YOLOv4, replacing the convolution kernel of YOLOv4 with the deformable convolutional network (DCNs) v2 convolution kernel, and replacing the backbone network with the mobile convolutional network MobileNet v3 framework, combined with the DeepSORT algorithm, to achieve tracking and counting of multiple targets. In order to improve the speed and accuracy of classification counting, Chen Binghuang optimized the k value through k-Means+ and configured a new anchor value based on YOLO, so that the classification and counting algorithms can quickly and intuitively obtain detection results, and achieve the goal of fast decision-making for drivers. Based on the YOLOv4 target detection model, Wang Xiushan et al. designed a multi-target classification counting algorithm that can avoid repeated counting. Yizhou Zhao et al. used the attention mechanism to propose a simple but effective architecture to solve the problem of inconsistent query support semantics and ignoring spatial structure and scale differences. In view of the difficulty of data set collection, Hsieh ChenChiung et al. used data enhancement of periodic generative adversarial networks (GAN) to increase the data set to achieve real-time identification and tracking of defects. Jung Hail and Rhee Jeongjin took advantage of the YOLO model and ResNet to improve the overall efficiency and accuracy of detecting hot pledge points from door trim images. In view of the difficulty in balancing detection speed and detection accuracy in the pedestrian target detection algorithm of infrared images and the large size of the algorithm model, Zhang Guoli et al. introduced the attention mechanism and improved the fusion network module to propose a lightweight infrared image pedestrian detection method based on the YOLO algorithm. Wang Xinchen et al. proposed a method of adding local perturbations to sample images to improve image recognition accuracy.

[0006] Wang Yan proposed a deep learning algorithm SI-YOLO suitable for surgical instrument detection. The SimAM attention mechanism was added to the backbone network of the YOLOv5 target detection network model to improve the model's ability to extract features of surgical instruments in interference environments. The specific location is in the residual module of the backbone network, that is, the Bottleneck module in models / common.py. The original PANet in the neck network Neck was replaced with the BiFPN node to improve the feature fusion strategy and solve the problem of inconsistent scale features of different surgical instruments. Finally, the α-IoU Loss was used to replace the original loss function to improve the speed and accuracy of detection box regression. However, the detection accuracy of this solution is still not high enough. Summary of the invention

[0007] In order to overcome the deficiencies of the prior art, an object of the present invention is to provide a method for real-time and accurate identification of surgical instruments.

[0008] The technical solution of the present invention is as follows:

[0009] A method for real-time and accurate identification of surgical instruments, the steps are as follows:

[0010] 1) Collect pictures of surgical instruments taken from various angles and create a data set;

[0011] 2) Use non-absolute mosaic and HSV data enhancement to pre-process images online to increase the diversity of training images;

[0012] 3) Add label regularization and category weight coefficients to increase the generalization performance of the recognition network and balance the recognition accuracy of different categories;

[0013] 4) Add the SimAM attention mechanism to the Bottleneck module of the backbone network in the YOLOV5 network model to enhance the learning ability of the network, and pass the obtained data set into the designed network model for training and learning;

[0014] 5) Save the obtained weight file after training the appropriate epoch;

[0015] 6) Test the untrained data using the weights from the weight file to identify surgical instruments.

[0016] In the data enhancement described in step 2), the occurrence coefficient of mosaic data enhancement is set to 0.9, and the occurrence coefficients of H, S, and V in HSV data enhancement are set to 0.015, 0.7, and 0.4, respectively.

[0017] The regularization coefficient and weight coefficient described in step 3) are as follows: during training, the regularization coefficient is set to 0.01, and the category weight is set according to the difficulty of the category to be identified.

[0018] In the method described, the network used to learn features is divided into a backbone network and a neck network, and a SimAM attention mechanism is added to the CSP structure of the Bottleneck module in the backbone network described in step 4), and then the CSP layer of the network is replaced with an improved one to enhance learning.

[0019] The beneficial effects of the present invention are:

[0020] At present, hospitals still use manual counting methods. A surgical operation often requires multiple counting. Manual counting takes up precious time for emergency patients and increases the workload of medical staff. It is easy to make mistakes due to human factors such as fatigue, which threatens the safety of patients. Using target detection to identify surgical instruments can significantly improve the recognition speed, and the simple and beautiful interactive interface design can facilitate the operation of medical staff.

[0021] At the same time, this solution selects the YOLOv5 network as the basic network, which is simpler to operate than other target detection networks. Improvements are made on the YOLOv5 network structure to design a network structure and parameters that are more suitable for surgical instrument identification and inventory. In view of the large difference in gloss between metal and non-metal in surgical instruments, and the small difference in parts between some categories, a solution combining mosaic and HSV data enhancement is designed; in view of the high cost of surgical instrument data collection and annotation, label smoothing regularization is added to present better training results with as few data sets as possible; in view of the large length-width ratio of some surgical instruments, a recalculation anchors algorithm is designed to correct the default anchors; in view of the difficulty of identifying some categories, the category loss weight is increased to strengthen the network's learning of the characteristics of such instruments; the SimAM attention mechanism is added to enhance the learning of target features. In general, this technical solution has both real-time and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flow chart of the present invention.

[0023] Figure 2 It is an enhanced effect diagram of HSV data (batchsize=4, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4).

[0024] Figure 3 It is an enhanced effect diagram of HSV data (batchsize=4, hsv_h=0, hsv_s=0, hsv_v=0).

[0025] Figure 4It is an enhanced effect diagram of HSV data (batchsize=4, hsv_h=0.05, hsv_s=0.8, hsv_v=0.5).

[0026] Figure 5 This is a diagram of the length-to-width ratio of common surgical instruments.

[0027] Figure 6 This is the principle diagram of the manual calculation of anchor box algorithm.

[0028] Figure 7 This is a schematic diagram of the Bottleneck module after the introduction of the SimAM attention mechanism.

[0029] Figure 8 It is the output diagram of the improved network structure.

[0030] Fig. 9 It is a table diagram of data set information.

[0031] Fig.10 It is an analysis chart of the parameters of various surgical instruments in the training set.

[0032] Fig.11 This is the improved network training result diagram.

[0033] Fig.12 This is the original network training result diagram.

[0034] Fig.13 It is the F1 score-confidence map of the improved network training process.

[0035] Fig.14 is the F1 score-confidence map of the original network training process.

[0036] Fig.15 It is the confusion matrix diagram of the improved network.

[0037] Fig.16 is the confusion matrix diagram of the original network.

[0038] Fig.17 This is the recognition effect diagram of the improved network.

[0039] Fig.18 This is the recognition effect diagram of the original network. DETAILED DESCRIPTION

[0040] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0041] like Figure 1 As shown, a method for real-time and accurate identification of surgical instruments, the steps are as follows:

[0042] 1) Collect pictures of surgical instruments taken from various angles and create a data set;

[0043] 2) Use non-absolute mosaic and HSV data enhancement to pre-process images online to increase the diversity of training images;

[0044] 3) Add label regularization and category weight coefficients to increase the generalization performance of the recognition network and balance the recognition accuracy of different categories;

[0045] 4) Add the SimAM attention mechanism to the Bottleneck module of the backbone network in the YOLOV5 network model to enhance the learning ability of the network, and pass the obtained data set into the designed network model for training and learning;

[0046] 5) Save the obtained weight file after training the appropriate epoch;

[0047] 6) Test the untrained data using the weights from the weight file to identify surgical instruments.

[0048] In the data enhancement described in step 2), the occurrence coefficient of mosaic data enhancement is set to 0.9, and the occurrence coefficients of H, S, and V in HSV data enhancement are set to 0.015, 0.7, and 0.4, respectively.

[0049] The regularization coefficient and weight coefficient described in step 3) are set to 0.01 during training, and the category weight is set according to the difficulty of the category to be identified. For example, similar scalpel No. 1 and scalpel No. 4, which only differ slightly in length, should be set to 3 and regarded as the most difficult category to identify, while the stainless steel dressing bowl can be set to 1 and regarded as the easiest item to identify.

[0050] In the method described, the network used to learn features is divided into a backbone network and a neck network, and a SimAM attention mechanism is added to the CSP structure of the Bottleneck module in the backbone network described in step 4), and then the CSP layer of the network is replaced with an improved one to enhance learning.

[0051] A more specific description is given below.

[0052] 1. Model size selection

[0053] Deep learning requires a large amount of training data in a supervised learning mode. For targets of the same category, different angles, lighting and other conditions need to be provided to simulate the environment with variable placement positions and lighting conditions at different times in the actual environment. Therefore, the more data there is, the better the generalization performance of the trained network model. However, considering the actual situation of hospital surgery, each time the data set images are collected, they must be specially operated and photographed by hospital nurses and doctors after the corresponding post-operative instruments have been sterilized at high temperature and high pressure, which has a high labor cost. In order to achieve higher results based on as few images as possible, an online data enhancement scheme is used at the input end, that is, the data set is only enhanced during training, but the number and characteristics of the uploaded data set are not changed. This not only does not destroy the original data set and facilitates multiple adjustments, but also can obtain a variety of transformed surgical instrument images during training, increasing the robustness of the training network.

[0054] We worked with hospitals to collect images and sorted out 88 categories of surgical instruments, with about 300 images in each category, a total of 26,070 images to create a dataset of commonly used surgical instruments for surgery. We used scripts to convert the format and divide the dataset for use in training. We made improvements on the YOLOv5 network structure and designed a network structure and parameters that are more suitable for surgical instrument identification and counting. In view of the large difference in gloss between metal and non-metallic gloss of surgical instruments, and the small difference between some categories, we designed a solution combining mosaic and HSV data enhancement; in view of the high cost of surgical instrument data collection and annotation, label smoothing regularization was added to present better training results with as few datasets as possible; in view of the large length-width ratio of some surgical instruments, we designed a recalculation anchors algorithm to correct the default anchors; in view of the difficulty of identifying some categories, we increased the category loss weight to strengthen the network's learning of the characteristics of such instruments; and added the SimAM attention mechanism to enhance the learning of target features.

[0055] 2Mosaic Data Augmentation

[0056] Mosaic randomly selects a picture from the training set. The input layer of YOLOv5 will process the picture into a 640×640 size picture by scaling and filling the edges by default. First, expand the length and width to twice the original size on the default size to set a large picture and randomly select a center point. Take a picture from the training set, and then randomly select three pictures. These four pictures are processed to the default size, and then the first picture is placed in the upper left corner of the large picture, and the other three pictures are placed in the upper right corner, lower left corner, and lower right corner of the large picture. The position of the label is calculated by calculating the offset from the four small pictures to the large picture, and the label information corresponding to the large picture synthesized by the four small pictures is obtained. The obtained integrated picture can still be randomly rotated, cropped, translated, and scaled, and then resized to integrate the image into a size of 640×640. When performing random processing, it is easy to have some images that have some target boxes that are too small after cutting off the parts that exceed the specified size. Set a threshold and discard the border when the border size is less than the threshold. In this example, the Mosaic coefficient is set to 0.5.

[0057] 3HSV data enhancement

[0058] In the task of identifying medical surgical instruments, the dataset is characterized by the presence of many reflective metal instruments. The images collected by such metal instruments are very sensitive to ambient lighting conditions and angles. HSV is composed of hue, saturation, and value, and is a color space that reflects the intuitiveness of color. The thresholds of the three parameters H, S, and V can be set in the hyperparameter file. During training, a multiplication factor within the corresponding parameter threshold is randomly generated. The parameters of the input image are multiplied by the corresponding multiplication factor and then integrated into an HSV image to obtain an image with a wider range of hue, saturation, and value. Figure 1 , 2 , 3 show the effects of HSV data enhancement with three different thresholds. From the analysis of the three pictures, we can see that appropriate HSV data enhancement can simulate the image in the uncertain lighting environment of the recognition scene caused by the position of medical staff or other uncertain factors, and also balance the metal reflectivity well.

[0059] 4-fold calculation anchor frame design

[0060] The original network parameters of YOLOv5 have given a set of nine original anchor boxes with initial length and width for the COCO dataset, as shown in Table 1.

[0061] Table 1 YOLOv5 original anchor box setting length and width value table

[0062]

[0063]

[0064] However, the length and width of the objects in the COCO dataset are relatively uniform, with a maximum aspect ratio of 4, while the maximum aspect ratio of some surgical instruments, such as long scalpels, is around 12. Figure 4 The following are examples of the aspect ratios of some surgical instruments at different angles. It can be seen that some commonly used surgical instruments are slender, and their aspect ratios are much greater than 4. Therefore, it is not appropriate to use the original anchor frames for the COCO dataset. Therefore, this example uses the method of manually calculating the anchor frames.

[0065] After verification, the maximum aspect ratio of the target in the dataset is about 12, so the aspect ratio threshold is set to 12, and then the manual calculation anchor frame algorithm is set. The principle is as follows Figure 5 Where bpr (best possible recall) represents the maximum number of target boxes that can be recalled / the total number of target boxes, and the maximum value is 1.

[0066] When using k-means combined with genetic algorithm to calculate better anchors, the k-means algorithm randomly generates 9 anchors for each target. Since adjacent pixels in the image have a strong correlation, they are redundant when participating in training, so each data needs to be normalized by standard deviation, that is, whitening operation. In this way, the correlation between the anchor features obtained is reduced, and the variance of the features is 1. Among them, fitness is a custom fitness calculation formula, that is, survival of the fittest, which is used in genetic algorithms to measure whether the mutation is effective. If it is effective, the selection operation is performed, and if it is not effective, the next round of mutation is continued.

[0067] 5 Label Smoothing Regularization

[0068] For the task of surgical instrument identification and counting, there are many types of surgical instruments. When collecting data set images, medical staff cannot occupy all surgical packages for this purpose. In addition, since image collection cannot take into account all angles, light, shooting distance, etc., there will be some deviation between the data set images and the images taken in the actual test. In the training of deep learning neural network models based on multi-classification tasks, the cross entropy loss function is generally used. One-hot is used to encode the labels, and the loss between the predicted probability and the true label is minimized to obtain the optimal predicted probability. The loss of erroneous samples is not considered. The traditional softmax formula is as shown in formula 1-1, where P i represents the probability value of the current sample category i, z i It refers to the logit of sample i corresponding to the current sample, n is the total number of categories, and the cross entropy loss is as shown in formula 1-2. Formula 1-3 is the weight of positive and negative samples, i=y is a positive sample, and i≠y is a negative sample.

[0069]

[0070]

[0071]

[0072] If only positive samples are considered, maximizing the log-likelihood of the correct label is equivalent to minimizing the cross-entropy loss function, and the optimal prediction distribution is as shown in Equation 1-4.

[0073]

[0074] When the error samples are not considered, the network assigns the probability of each training sample to the true label during learning, which is equivalent to not learning the situation with error samples, resulting in overfitting. At the same time, the cross entropy loss makes the logarithm tend to a larger value, thereby increasing the difference with all other logarithms, which will reduce the model's adaptive ability. The network trusts the current data set too much during training, so the generalization performance of the obtained model will be poor.

[0075] Label smoothing is a common regularization strategy that adds noise through soft one-hot to reduce the weight of the real sample label when calculating the loss function and improve model overfitting. The label smoothing formula after adding Label Smoothing is shown in the following formula 1-5, where y represents the original prediction probability value, λ is the smoothing factor, and N is the total number of classification categories.

[0076]

[0077] 6Customize Class Loss Weights

[0078] Three loss functions are designed in the YOLOv5 network, among which the CIoU loss function is used by default for box regression loss, and the BCE loss function is used by default for confidence loss and classification loss. For classification loss, taking the image divided into 80*80 grids as an example, the neural network will generate three nearby rectangular prediction boxes for each grid. In multi-classification tasks, each prediction box will contain N predicted category probabilities, that is, the output of the neural network is a 3*80*80*N probability matrix. Usually, in order to maintain the stability of training and reduce overfitting, label smoothing operations are added, as shown in the following formula 3-6, where label represents the category unique hot code value parsed from the label file, and α represents the smoothing coefficient. Let the label probability be denoted as the matrix Lsmooth, and the model prediction probability matrix be denoted as P, then the BCE loss calculation formula for each value in the matrix is ​​formula 1-7:

[0079] label smooth =label*(1-α)+α / N (1-6)

[0080] loss BCE =-L smooth *logP-(1-Lsmooth )*log(1-P) (1-7)

[0081] In general multi-classification tasks, the same weight is used by default for classification losses of all categories. In YOLOv5, sample images can be collected inversely according to the imbalance of the number of samples, that is, the groundthruth list of the training set samples is read, the number of label boxes of different categories is counted, and the inverse of the number of label boxes of each category is taken to form a histogram distributed by category. The category weight is the inverse value. During training, each epoch collects images according to the category weight for training. The larger the weight, the more likely it is that this category will be collected.

[0082] However, under this weight assignment mechanism, only the imbalance of the number of categories is considered. For special recognition tasks such as surgical instrument recognition, there are different levels of difficulty in recognizing different categories. In order to balance the difference in category difficulty, a category weight vector can be customized when calculating the BCE loss. A larger loss weight is given to the difficult-to-recognize class to enhance the model's learning of the class, thereby better balancing the different difficulty levels of the recognition. The BCE loss function after adding the loss weight is shown in Formula 1-8, where wn is the weight of the nth category target.

[0083] loss BCE =-w n L smooth *logP-w n (1-L smooth )*log(1-P) (1-8)

[0084] 7 SimAM Attention Mechanism

[0085] The attention mechanism is inspired by the visual processing mechanism of the human brain. When we see a picture, the brain does not instruct us to directly browse the entire picture and treat every element in the picture equally. Instead, it selectively observes the elements of the picture based on the brain's learning of the world and the purpose. The most basic and commonly used attention mechanisms are the channel attention mechanism SE and the channel and space attention mechanism CBAM. After the SE attention mechanism generates 1D weights, it expands the generated weights to channel attention, while after the CBAM attention mechanism generates 2D weights, it infers the feature maps along the two dimensions of space and channel respectively, combines the two attention modules of space and channel to enhance useful information and suppress the expression of useless information, and finally multiplies the obtained attention map with the input feature map. Many current attention modules have two problems. First, they can only obtain features through space and channels, which limits the flexibility of attention weights. At the same time, they have multiple convolutions, complex structures, and are affected by a series of factors.

[0086] The SimAM attention mechanism evaluates feature importance based on the energy function in the famous neuroscience theory. In neuroscience, activated neurons will produce spatial inhibition effects, and such neurons should be given more important weights. A simple way to measure the importance of neurons is to measure the linear separability of neurons. Therefore, an energy function is defined, and its calculation formula is as shown in Equations 1-9, 1-10, 1-11, and 1-12, where t represents the target neuron, x represents the adjacent neuron, and λ is a hyperparameter. The lower the calculated energy function value, the higher the distinction between the neuron and the adjacent neurons, that is, the higher the spatial inhibition effect, so through 1 / e t x The importance of neurons is weighted to strengthen the target information of surgical instruments and reduce the interference of non-important information.

[0087]

[0088]

[0089]

[0090]

[0091] In this example, the SimAM attention mechanism is added to the Bottleneck module in common.py, that is, SimAM is combined with the CSP structure in the backbone feature extraction network. The improved Bottleneck module structure is as follows: Figure 6 , the improved YOLOv5 network structure output is as follows Figure 7 .

[0092] SimAM module algorithm idea:

[0093] 1. Define X: input features [b, c, h, w];

[0094] e_lambda: hyperparameter λ;

[0095] Space size: n = w*h-1;

[0096] (tu) 2 :d=(XX.mean(dim=[2,3])).pow(2);

[0097] Channel variance d.sum() / n: v = d.sum(dim = [2,3]) / n.

[0098] 2. Importance grouping basis: E_inv = d / (4*(v+e_lambda))+0.5.

[0099] 3. Return to add features: X*sigmoid(E_inv).

[0100] In summary, the present invention aims at the detection and counting tasks of surgical instruments in surgical operations, and designs a network model suitable for the characteristics of surgical instruments based on the improvement of the YOLOv5 network model. First, the input layer is designed, including mosaic data enhancement with a coefficient of 0.1, translation, upside down flipping, and left-right flipping image processing, and HSV data enhancement is used to improve the influence of lighting uncertainty and reflection. Then, predefined anchors suitable for surgical instrument data sets are designed. Label smoothing regularization is added to the model to improve the generalization ability of the model and prevent overfitting to a certain extent. The SimAM attention mechanism will be added to the backbone to enhance the feature extraction ability of surgical instruments. Finally, a counting function is designed at the output end to realize the visualization of the number of labels.

[0101] Unlike the residual module proposed by Wang Yan that adds the SimAM attention mechanism to the backbone, this solution combines the SimAM attention mechanism with the residual structure. Experiments have shown that the detection accuracy is higher than that of adding a residual module. At the same time, the present invention also takes into account that the default data enhancement of YOLOv5 will cut a large number of surgical instruments with similar features into the same instruments for confusing learning. Through experiments, a data enhancement scheme that is more suitable for surgical instrument detection is verified to further improve the detection accuracy. In addition, the present invention recalculates the anchor frame that is more in line with the surgical instrument dataset and calculates a more suitable anchor frame for surgical instruments. In order to make the model calculation faster and simpler as much as possible, the present invention no longer adds more complex structures, and only improves the generalization ability of the model by setting the label smoothing coefficient to 0.01. It also gives greater weight to some difficult-to-identify categories for focused learning to enhance recognition capabilities.

[0102] Application Examples

[0103] Taking into account the time cost and hardware limitations of training, this experiment selected some pictures to form a sample data set. In each experiment, only 6 types of surgical instruments were selected for training and verification. The six representative categories are 4# surgical handle (with blade), 4# surgical handle (without blade), curved comprehensive tissue scissors, abdominal wall spreader, cap, and stainless steel dressing bowl. The morphological difference between surgical handles with blades and without blades is only in a small part of the tip, which can represent surgical instruments with very similar shapes and only slight differences. The curved comprehensive tissue scissors and surgical handles are similar in a small part and are both slender instruments, while the abdominal wall spreader is an instrument with little difference in length and width. The stainless steel dressing bowl and cap are similar in shape, but one is made of metal and the other is made of non-metal. Therefore, the quality of the experimental results of this set of data sets can, to a certain extent, reflect the effectiveness of the network in the task of surgical instrument recognition. Figure 8 .

[0104] Visualize the label information of the training set as follows Fig. 9 The number of labels of the six surgical instruments in the training set is about 630 for surgical knife handles without handles, about 600 for surgical knife handles with handles, about 640 for integrated tissue scissors, about 600 for abdominal wall spreaders, about 290 for caps, and about 280 for stainless steel dressing bowls. The labels are evenly distributed horizontally and vertically, mainly concentrated in the middle of the image. The label box sizes are mostly large and medium-sized objects, and there are no particularly small targets, which means that no special processing is required for small targets.

[0105] The original YOLOv5 network is compared with a network that uses a customized data augmentation scheme, manually calculates anchors, and takes label smoothing regularization. The label smoothing coefficient is set to 0.01. According to experience, the two forms of surgical handles are difficult to identify. At the same time, the surgical handle and the comprehensive tissue scissors also have certain similarities. Therefore, the weight coefficient is initially set to [2,2,2,1,1,1], and the initial learning rate is set to 0.001. After training for 30 epochs, the learning rate is reduced to 0.000032 and then trained for another 10 epochs. At this time, the training results of the improved network are shown in Table 2, and the running results of the original network are shown in Table 3. In the improved network, the two indicators of average recognition accuracy of the six surgical instruments, mAP@.5 and mAP@.5:.95, are both increased. For a single category analysis, the surgical handle with a blade and without a blade is due to similar categories, which is difficult to identify, and the recognition accuracy will be significantly lower than that of the other categories. However, after adding category weights, the recognition accuracy of these two categories is also improved.

[0106] Table 2 Improved YOLOv5 network training results

[0107]

[0108] Table 3 Original YOLOv5 network training results

[0109]

[0110] The box regression loss, confidence loss, category loss and mAP index of the train and val sets during training are visualized as follows: Fig.10 , Fig.11 As can be seen from the loss curve in the figure, the original network has converged at this time, and the improved network can continue to be trained until it is fully converged. The convergence in the improved network is smoother, and the average accuracy also increases with the increase of training epochs.

[0111] The relationship function diagram of the F1 score and confidence for each category and the overall average is as follows Fig.12 , Fig.13. From the relationship between F1 score and confidence, it can be seen that the improved network still has a high F1 score above 0.5 confidence for comprehensive tissue scissors, abdominal wall spreaders, caps and stainless steel dressing bowls and two types of surgical handles, and the detection effect has been significantly improved. Among them, the worst effect is the surgical handle without a blade, and the F1 score in most confidence intervals is somewhat different from other types. Overall, the F1 peaks of all curves have increased, and the curves are more prominent to the upper right.

[0112] Fig.14 is the confusion matrix of the improved network on the validation set, Fig.15 is the confusion matrix of the original network. As can be seen from the figure, both forms of surgical handles are misjudged as background before and after the improvement. After the improvement, the recognition effect of 4#ssdb.1, i.e., surgical handle with blade, is greatly improved, while the improvement effect on other categories is small. At the same time, before the improvement, the background was also identified as surgical handle 1, surgical handle 2, comprehensive tissue scissors, abdominal wall spreader, and cap. After the improvement, the situation of background being mistakenly identified as objects has improved, but the situation of background being identified as cap suddenly increased. It may be that the cap, as a non-metallic instrument, is closer to the background in gloss and color. However, in the task of surgical instrument recognition, considering that the probability of the cap being missed is very small, identifying the background as a cap will lead to more objects being identified, which is a recognition error that is easy to troubleshoot.

[0113] Modify the detect.py file to add output images after detection to the network and display the counting results and category information on the image. Load the weight file into detect.py for testing. The test results are as follows: Fig.16 , run the original network as Fig.17 Comparing the two recognition effect pictures, the target detection frame of the improved network is obviously more in line with the surgical instrument. This is because the improvement of the custom anchor can make the frame regression more accurate and close to the actual target frame. The detection count result of the current image is displayed in the upper left corner of the image. The number of labels 4#ssdb.2, 4#ssdb.1, 20cmwWDzhzzjJC2509, and fbckq.B30160 are all 1.

[0114] The embodiments described above can be further combined or replaced, and the embodiments are only descriptions of preferred embodiments of the present invention, and do not limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various changes and improvements made by ordinary technicians in this field to the technical solution of the present invention belong to the protection scope of the present invention. The protection scope of the present invention is given by the attached claims and any equivalents thereof.

Claims

1. A method for real-time and accurate identification of surgical instruments, characterized by: steps as follows: 1) Collect pictures of surgical instruments taken from various angles and create a data set; 2) Use non-absolute mosaic and HSV data enhancement to pre-process images online to increase the diversity of training images; 3) Add label regularization and category weight coefficients to increase the generalization performance of the recognition network and balance the recognition accuracy of different categories; 4) Add the SimAM attention mechanism to the Bottleneck module of the backbone network in the YOLOV5 network model to enhance the learning ability of the network, and pass the obtained data set into the designed network model for training and learning; 5) Save the obtained weight file after training the appropriate epoch; 6) Test the untrained data using the weights from the weights file to identify surgical instruments.

2. The method according to claim 1, characterized in that: In step 2), the occurrence coefficient of mosaic data enhancement is set to 0.9, and the occurrence coefficients of H, S, and V in HSV data enhancement are set to 0.015, 0.7, and 0.4, respectively.

3. The method according to claim 1, characterized in that: The regularization coefficient and weight coefficient described in step 3) are as follows: during training, the regularization coefficient is set to 0.01, and the category weight is set according to the difficulty of the category to be identified.

4. The method according to claim 1, characterized in that: The network used to learn features is divided into a backbone network and a neck network. The SimAM attention mechanism is added to the CSP structure of the Bottleneck module in the backbone network described in step 4), and then the CSP layer of the network is replaced with an improved one to enhance learning.

Images