A deep learning-based compression method for surgical instrument storage environment inventory model

Abstract

The present invention discloses a compression method for a surgical instrument storage environment inventory model based on deep learning, which belongs to the field of target detection technology. A compression method for a surgical instrument storage environment inventory model based on deep learning, comprising the following steps: S1, photographing the surgical instruments to collect data, forming a data set, and labeling the photos in the data set as required; S2, dividing the data set into a training set, a validation set, and a test set, and performing data enhancement preprocessing on the data before training; S3, performing a lightweight structure design for the Swin transformer model, reducing the memory and computation required for the Swin transformer model, and achieving model compression; S4, encapsulating the target detection model code, and then inputting the data set into the model for training. The present invention performs model compression based on the target detection model of Swin Transformer, which can ensure the accuracy of the model while reducing the computation and parameter amount.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a deep learning-based compression method for a surgical instrument storage environment inventory model. Background Art

[0002] In recent years, surgical instruments have garnered widespread attention due to their close relationship with human health. Whether in clinical settings or storage environments, the types and quantities of surgical instruments must be counted to ensure the smooth execution of subsequent surgeries. However, currently, most surgical instrument counting is done manually, which consumes a lot of manpower and material resources and is inefficient. Especially in storage environments, counting large quantities of surgical instruments can seriously impact the normal progress of subsequent work. Therefore, the industry is increasingly demanding automated counting of surgical instruments in storage environments.

[0003] The current research methods for automatic inventory of surgical instruments mainly include RFID (Radio Frequency Identification), barcode and computer vision. Since the RFID-based and barcode-based methods are relatively complex and costly, current research mainly focuses on computer vision-based methods. The use of deep learning target detection algorithms for instrument inventory has been the most common method in recent years. However, ordinary target detection algorithms have shortcomings such as large model size and serious memory usage, and are not suitable for deployment and use in edge computing devices equipped for inventory in warehouse environments.

[0004] In order to deploy and use the algorithm on devices with limited computing resources, it is particularly important to achieve lightweight models through compression and acceleration without significantly affecting the recognition accuracy. Therefore, in order to realize the edge device deployment and use of the instrument inventory model in the warehouse environment, it is of great significance to invent a deep learning-based compression method for the surgical instrument warehouse environment inventory model. Summary of the Invention

[0005] The purpose of the present invention is to provide a surgical instrument storage environment inventory model compression method based on deep learning, to compress the target detection model based on Swin Transformer, and to ensure the accuracy of the model while reducing the amount of calculation and parameters.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A deep learning-based compression method for a surgical instrument storage environment inventory model includes the following steps:

[0008] S1. Surgical instrument data collection and labeling: Take photos of surgical instruments to collect data, form a data set, and label the photos in the data set as required;

[0009] S2. Dataset division and data enhancement: The dataset obtained in S1 is divided into training set, validation set and test set, and data enhancement preprocessing is performed on the data before training;

[0010] S3. Lightweight design of the object detection model: We designed a lightweight structure for the Swin transformer model to reduce the memory and computational load required, thereby achieving model compression.

[0011] S4. Target detection model training: Encapsulate the lightweight target detection model code in S3, then input the dataset into the model for training. After the pre-set model training wave is completed, save the model parameters with the highest accuracy for the inventory of surgical instruments in the warehouse environment.

[0012] Preferably, when the S1 is collecting photographic data, all surgical instruments to be inspected are placed in sequence according to five placement methods: instruments of the same type are placed close together, instruments of the same type are placed crosswise, instruments of different types are placed close together, instruments of different types are placed crosswise, and all devices are placed crosswise and close together. This is used to simulate various complex situations in actual applications and ensure the richness and balance of image information collection.

[0013] Preferably, the photo labeling work mentioned in S1 specifically includes the following steps:

[0014] A1. Assign a label name to the surgical instrument whose image data has been collected;

[0015] A2. Input all collected surgical instrument images into the image annotation software;

[0016] A3. Use annotation software to annotate all surgical instruments in each image with rectangular boxes. This means that the rectangular boxes outline the instruments in each image. The rectangular boxes provide the information required for training the object detection model, and are then labeled with the corresponding label names.

[0017] A4. After labeling is completed, the corresponding tag name, image name, rectangle outline, and position information are saved in a JSON file. The rectangle outline position information is obtained by establishing a coordinate system on the entire image and recording the x-axis and y-axis coordinates of each vertex of the polygon.

[0018] Preferably, the ratio between the training set, validation set and test set mentioned in S2 is 7:2:1; the training set is used to train the network; the validation set is used to verify the model detection effect during the training process and adjust the model's hyperparameters according to the effect; the test set is used for model effect testing.

[0019] Preferably, the specific enhancement method of the data enhancement preprocessing mentioned in S2 is Mosaic data enhancement, that is, randomly scaling, cropping and arranging multiple images together to enhance the generalization of the model.

[0020] Preferably, the lightweight design of the target detection model mentioned in S3 specifically includes the following contents:

[0021] B1. Given a set of surgical instrument images, denoted as The i-th image in the batch is represented as Among them, the superscript represents the processing sequence number, 0 represents the original image;

[0022] B2. Perform PatchEmdeding on the input image in the set, that is, use the grid division tool to divide the image into several non-overlapping blocks of equal size, each block is recorded as B i ∈R 3×4×4 , During the self-attention operation, it is converted into a series of 48 RGB pixels (T i ∈R 1×48 ), equivalent to the token in Transformer;

[0023] B3, convert the image into The transformed image is then input into a feature extraction module (hereinafter referred to as Block) consisting of five Swin Transformer Blocks. A single Block module divides the input feature map into several non-overlapping windows.

[0024] B3: Self-attention calculation is performed separately in each window. The other structures in the Block module are the same as ViT, including layer normalization, fully connected layers and residual connections;

[0025] B4. Modify the connection mode of the Swin transformer series block, add parallel connection, increase the width of the network, introduce different size window division, concat the output structure of two branches with different window sizes, and then connect a pooling layer to replace the original PatchEmerging layer to increase the receptive field and number of channels of the feature map, forming a hierarchical structure; after the two structures are the same but the window sizes are different, that is, the parallel structure of the blocks of different scales, there is a separate block module and a spatial attention module, so the overall structure consists of 4 levels, and the input of the four levels is

[0026] B5. Use the four-level input in B4 as the input of the FPN (Feature Pyramid Networks) structure of Faster R-CNN;

[0027] B6. Perform final target detection through Faster R-CNN to generate corresponding prediction boxes and confidence levels.

[0028] Compared with the existing technology, the present invention provides a deep learning-based surgical instrument storage environment inventory model compression method, which has the following beneficial effects:

[0029] The present invention proposes a deep learning-based compression method for the surgical instrument storage environment inventory model, which compresses the target detection model based on Swin Transformer and can ensure the accuracy of the model while reducing the amount of calculation and parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of a deep learning-based compression method for surgical instrument storage environment inventory model proposed by the present invention;

[0031] Figure 2 This is a schematic diagram of the model architecture after lightweight design proposed in Example 1 of the present invention;

[0032] Figure 3 Schematic diagram of the different-scale window division method proposed in Example 1 of the present invention. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0034] Example 1:

[0035] See also Figure 1-3;

[0036] A deep learning-based compression method for a surgical instrument storage environment inventory model includes the following steps:

[0037] S1. Surgical instrument data collection and labeling: Take photos of surgical instruments to collect data, form a data set, and label the photos in the data set as required;

[0038] When collecting photographic data, all surgical instruments to be inspected are placed in five different placements: instruments of the same type placed close together, instruments of the same type placed crosswise, instruments of different types placed close together, instruments of different types placed crosswise, and all instruments placed crosswise and close together. This is used to simulate various complex situations in actual applications and ensure the richness and balance of image information collection.

[0039] When performing photo labeling work, the specific steps include:

[0040] A1. Assign a label name to the surgical instrument whose image data has been collected;

[0041] A2. Input all collected surgical instrument images into the image annotation software;

[0042] A3. Use annotation software to annotate all surgical instruments in each image with rectangular boxes. This means that the rectangular boxes outline the instruments in each image. The rectangular boxes provide the information required for training the object detection model, and are then labeled with the corresponding label names.

[0043] A4. After labeling, save the corresponding tag name, image name, rectangle outline, and position information in a JSON file. The rectangle outline position information is obtained by establishing a coordinate system on the entire image and recording the x-axis and y-axis coordinates of each vertex of the polygon.

[0044] S2. Dataset division and data augmentation: The dataset obtained in S1 is divided into training set, validation set and test set, where the ratio between training set, validation set and test set is 7:2:1; the training set is used to train the network; the validation set is used to verify the model detection effect during the training process and adjust the model's hyperparameters based on the effect; the test set is used to test the model effect;

[0045] At the same time, data augmentation preprocessing is performed on the data before training. The specific enhancement method is Mosaic data augmentation, which randomly scales, crops, and arranges multiple images together to enhance the generalization of the model.

[0046] S3. Lightweight design of target detection model: We conduct lightweight structural design for the Swin transformer model to reduce the memory and computational load required by the Swin transformer model, achieve model compression, and redesign the overall model architecture. Figure 2 As shown, specifically including the following:

[0047] B1. Given a set of surgical instrument images, denoted as The i-th image in the batch is represented as Among them, the superscript represents the processing sequence number, 0 represents the original image;

[0048] B2. Perform PatchEmdeding on the input image in the set, that is, use the grid division tool to divide the image into several non-overlapping blocks of equal size, each block is recorded as B i ∈R 3×4×4 , During the self-attention operation, it is converted into a series of 48 RGB pixels (T i ∈R 1×48 ), equivalent to the token in Transformer;

[0049] B3, convert the image into The transformed image is then input into a feature extraction module (hereinafter referred to as Block) consisting of five Swin Transformer Blocks. A single Block module divides the input feature map into several non-overlapping windows.

[0050] The Swin transformer divides the image into non-overlapping windows and calculates self-attention separately within each window, ensuring that complexity grows only linearly with image size while allowing information from adjacent windows to flow between them. Furthermore, the Swin transformer uses a window shifting method, which shifts the entire feature map before performing window self-attention calculations. This ensures that the window contains information from adjacent windows from the previous calculation. However, after the shift, the pixels on both sides of the stitching line become unrelated, disrupting the spatial information of the image. To maintain the spatial position invariance of the image data, a mask must be added to the window data to compensate. The shifting process, which includes masking operations, is relatively complex and may not be optimized for mobile device speeds. In particular, the shifting operation uses the Torch.roll() function, which is memory-unfriendly and rarely supported by popular edge device inference frameworks (such as Nvidia TensorRT and Google Tensorflow-Lite). This hinders the Swin transformer's edge computing deployment in warehouse environments.

[0051] Therefore, in order to compress the model, self-attention calculation based on different-scale windows is adopted, by setting the sizes of the windows of two adjacent blocks to be inconsistent, such as Figure 3 As shown in the figure, the next window division already contains the adjacent window information from the previous one. At the same time, referring to the feature extraction ideas existing in convolutional neural networks, using windows of different sizes can extract richer features and enhance the modeling ability of the model. At the same time, the entire process only involves matrix multiplication that has been optimized by modern deep learning frameworks, avoiding the use of special matrix operations, which is more conducive to model deployment in application environments such as surgical instruments in warehouse environments.

[0052] B3: Self-attention calculation is performed separately in each window. The other structures in the Block module are the same as ViT, including layer normalization, fully connected layers and residual connections;

[0053] B4. Modify the connection mode of the Swin transformer series block, add parallel connection, increase the width of the network, introduce different size window division, concat the output structure of two branches with different window sizes, and then connect a pooling layer to replace the original PatchEmerging layer to increase the receptive field and number of channels of the feature map, forming a hierarchical structure; after the two structures are the same but the window sizes are different, that is, the parallel structure of the blocks of different scales, there is a separate block module and a spatial attention module, so the overall structure consists of 4 levels, and the input of the four levels is

[0054] The parallel connection between blocks adopted by the present invention is similar to the residual connection of ResNet. The difference is that ResNet is a direct addition, while the present invention adopts the Concat operation to directly splice the feature maps in the channel dimension and then perform the pooling operation to achieve the same hierarchical structure and avoid the introduction of the fully connected layer. Moreover, through the parallel connection, the feature maps calculated by windows of different sizes and stages are combined on the channel to establish a connection between two pixels at a certain distance on the image, which can achieve the effective fusion of high-pixel information and semantic information, ensure the efficient use of feature maps, and reduce the amount of calculation and parameters at the same time, reflecting the characteristics of lightweight design.

[0055] In addition, in a warehouse environment, the images captured by the camera often only contain a single object, with a relatively concentrated key information and most of the information being useless. Therefore, a global spatial attention module is added at the end of the network. The feature map size at this location is smaller, so the computational burden of performing global attention calculations is reduced. This can also improve the model's ability to extract target location information and suppress background noise.

[0056] B5. Use the four-level input in B4 as the input of the FPN (Feature Pyramid Networks) structure of Faster R-CNN;

[0057] B6. Perform final target detection through Faster R-CNN to generate corresponding prediction boxes and confidence levels.

[0058] S4. Target detection model training: Encapsulate the lightweight target detection model code in S3, then input the dataset into the model for training. After the pre-set model training wave is completed, save the model parameters with the highest accuracy for the inventory of surgical instruments in the warehouse environment.

[0059] In summary, the present invention proposes a surgical instrument storage environment inventory model compression method based on deep learning, which compresses the target detection model based on Swin Transformer, and can ensure the accuracy of the model while reducing the amount of calculation and parameters.

[0060] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A surgical instrument storage environment inventory model compression method based on deep learning, characterized in that: The following steps are involved: S1. Surgical instrument data collection and labeling: Take photos of surgical instruments to collect data, form a data set, and label the photos in the data set as required; S2. Dataset division and data enhancement: The dataset obtained in S1 is divided into training set, validation set and test set, and data enhancement preprocessing is performed on the data before training; S3. Lightweight Design of Object Detection Model: We designed a lightweight structure for the Swin transformer model to reduce the memory and computational load required, thereby achieving model compression. The lightweight design of the object detection model specifically includes the following: B1. Given a set of surgical instrument images, perform PatchEmbeding on the input images in the set. B2. Transform the image through the fully connected layer, and then input the transformed image into a feature extraction module composed of 5 Swin TransformerBlocks. The feature extraction module divides the input feature map into several non-overlapping windows; B3. Self-attention calculation is performed separately in each window. The feature extraction module includes layer normalization, fully connected layers and residual connections. B4. Modify the connection method of the Swin transformer series feature extraction module, add parallel connections, increase the width of the network, and introduce window divisions of different sizes to form a hierarchical structure. The inputs of different levels are used as the input of the FPN structure of Faster R-CNN; B5. Perform the final target detection through Faster R-CNN to generate the corresponding prediction box and confidence S4. Target detection model training: Encapsulate the lightweight target detection model code in S3, then input the dataset into the model for training. After the pre-set model training wave is completed, save the model parameters with the highest accuracy for the inventory of surgical instruments in the warehouse environment.

2. The method for compressing a surgical instrument storage environment inventory model based on deep learning according to claim 1, characterized in that: When the S1 is collecting photographic data, all surgical instruments to be inspected are placed in sequence according to five placement methods: instruments of the same type are placed close together, instruments of the same type are placed crosswise, instruments of different types are placed close together, instruments of different types are placed crosswise, and all devices are placed crosswise and close together. This is used to simulate various complex situations in actual applications and ensure the richness and balance of image information collection.

3. The method for compressing a surgical instrument storage environment inventory model based on deep learning according to claim 1, characterized in that: The photo labeling work mentioned in S1 specifically includes the following steps: A1. Assign a label name to the surgical instrument whose image data has been collected; A2. Input all collected surgical instrument images into the image annotation software; A3. Use annotation software to annotate all surgical instruments in each image with rectangular boxes. This means that the rectangular boxes outline the instruments in each image. The rectangular boxes provide the information required for training the object detection model, and are then labeled with the corresponding label names. A4. After labeling is completed, the corresponding tag name, image name, rectangle outline, and position information are saved in a JSON file. The rectangle outline position information is obtained by establishing a coordinate system on the entire image and recording the x-axis and y-axis coordinates of each vertex of the polygon.

4. The method for compressing a surgical instrument storage environment inventory model based on deep learning according to claim 1, characterized in that: The ratio between the training set, validation set and test set mentioned in S2 is 7:2:1; the training set is used to train the network; the validation set is used to verify the model detection effect during the training process and adjust the model's hyperparameters based on the effect; the test set is used to test the model effect.

5. The method for compressing a surgical instrument storage environment inventory model based on deep learning according to claim 1, characterized in that: The specific enhancement method of the data enhancement preprocessing mentioned in S2 is Mosaic data enhancement, which is to randomly scale, crop and arrange multiple images together to enhance the generalization of the model.

6. The method for compressing a surgical instrument storage environment inventory model based on deep learning according to claim 1, characterized in that: The PatchEmbeding operation mentioned in B1 on the input image in the set specifically includes the following: C1. Use the grid division tool to divide the image into several non-overlapping blocks of equal size; C2: During the self-attention operation, the block obtained in C1 is converted into a series of 48 RGB pixels, which is equivalent to the token in the Transformer.

7. The method for compressing a surgical instrument storage environment inventory model based on deep learning according to claim 1, characterized in that: The introduction of window division of different sizes mentioned in B4 specifically includes the following: The structures of the outputs of two branches with different window sizes are concat- ed , and then a pooling layer is connected to replace the original PatchEmerging layer, so that the receptive field and the number of channels of the feature map are increased, forming a hierarchical structure.

Images