Tiny weak soft tissue target detection method based on optimized DETR
By adopting deep learning network models based on optimized DETR in the target detection technology, and using the global context information optimization feature extraction, the problem of poor detection effects of small and weak soft tissues such as round window membranes has been solved. Essence
Patent Information
- Application Number
- CN202411351461.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2025-01-21
AI Technical Summary
The existing target detection technology has poor detection effect on tiny, weak soft tissues such as round window films. Especially in complex scenes of the ear canal, the translucent and low texture characteristics make the target detection more difficult and prone to false detection.
A deep learning target detection network model based on optimized DETR is used to utilize global context information to optimize the feature extraction process of the target, reduce post-processing steps, and improve detection accuracy and speed.
By optimizing the DETR model, the detection accuracy and speed of small and weak soft tissue targets such as round window films can be significantly improved, the false detection rate can be reduced, and the detection efficiency can be improved.
Smart Images
Figure CN119339369A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a method for detecting tiny weak soft tissue targets based on optimized DETR. Background Art
[0002] In clinical practice, the causes of some inner ear diseases such as sudden deafness, Meniere's disease, and tinnitus are still unknown, and there is a lack of targeted etiological treatment. Currently, drugs are still used for treatment. Due to the existence of the blood-labyrinth barrier, the concentration of drugs reaching the inner ear structure after systemic intravenous administration is low and the metabolism is rapid. The round window membrane is a membranous structure that connects the inner ear scala tympani to the middle ear tympanic cavity. It is permeable, and drugs can enter the inner ear perilymph through the round window niche-round window membrane pathway from the tympanic cavity to reach the drug therapeutic concentration. The round window area is an important area for modern ear surgery and otoneurosurgery. Whether it is endoscopic examination of the round window membrane, drug administration through the tympanic cavity-round window niche, or intraoperative exploration of the round window niche and round window membrane, the detection of the round window membrane is a key technical element in robot-assisted medical treatment for ear surgery.
[0003] The size of the round window membrane is about 2mm2. The existing target detection technology is not ideal for detecting tiny targets such as the round window membrane, and its low-texture, translucent membrane structure has the problem of difficult feature extraction. In the complex scene of the ear canal, these translucent, low-texture membranous tissues are difficult to distinguish from other tissue structures, and are prone to false detection. In addition, the local perception ability of traditional convolutional neural networks limits the modeling of global information. The existing target detection model is not effective for detecting tiny targets such as the translucent, low-texture round window membrane. Summary of the invention
[0004] The purpose of the present invention is to solve the problems raised in the above-mentioned background technology. The present invention provides a small weak soft tissue target detection method based on optimized DETR, which utilizes global context information to avoid the complex post-processing steps in traditional target detection methods, and achieves the purpose of improving the detection accuracy and speed of small targets such as round window membrane.
[0005] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:
[0006] A method for detecting tiny weak soft tissue targets based on optimized DETR, comprising:
[0007] Step 1: Obtain tiny semi-transparent weak soft tissue image data, and construct a data set including a training set and a validation set. Each sample in the data set is an image with the location and category of the tiny weak soft tissue target annotated;
[0008] Step 2: Build a deep learning target detection network model based on optimized DETR;
[0009] Step 3: Use the calibrated data set described in step 1 to input into the target detection network model described in step 2 for training, and after the training is completed, output the trained DETR target detection model as the network model for tiny weak soft tissue target detection;
[0010] Step 4: Get the image to be detected and input it into the DETR target detection network model trained in step 3 for detection.
[0011] Beneficial effects:
[0012] The present invention utilizes global context information, avoids the complicated post-processing steps in traditional target detection methods, optimizes the feature extraction process of the target, and achieves the purpose of improving the detection accuracy and speed of tiny targets such as the round window membrane. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0014] Figure 1 A schematic diagram of a DETR according to an embodiment of the present invention;
[0015] Figure 2 A schematic diagram of a Swin Transformer according to an embodiment of the present invention;
[0016] Figure 3 A schematic diagram of a Swin Transformer Block in a Swin Transformer according to an embodiment of the present invention;
[0017] Figure 4 FIG. 4 is a schematic diagram of an Encoder-Decoder in a DETR according to an embodiment of the present invention. DETAILED DESCRIPTION
[0018] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.
[0019] As shown in the figure, the embodiment of the present invention discloses a method for detecting tiny weak soft tissue targets based on optimized DETR, including:
[0020] Step 1: Obtain tiny semi-transparent weak soft tissue image data, and construct a data set including a training set and a validation set. Each sample in the data set is an image with the location and category of the tiny weak soft tissue target annotated;
[0021] Step 2: Build a deep learning target detection network model based on optimized DETR;
[0022] Step 3: Use the calibrated data set described in step 1 to input into the target detection network model described in step 2 for training. After the training is completed, output the trained DETR target detection model as the network model for detecting tiny weak soft tissue targets such as round window membranes;
[0023] Step 4: Get the image to be detected and input it into the DETR target detection network model trained in step 3 for detection.
[0024] Construct a dataset of tiny weak soft tissues such as the round window membrane. Specifically, image frames are captured from original pictures and videos obtained clinically as data samples to construct a dataset that contains the location and category information of tiny weak soft tissues such as the round window membrane.
[0025] In step three, the network is trained by using the divided data set and inputting it into the target detection network model for iterative training to obtain the optimal weight model.
[0026] like Figure 1 As shown, the construction of the deep learning target detection network model based on the optimized DETR is based on the DETR deep learning target detection network model, and the original Backbone is optimized from a convolutional neural network (CNN) to a SwinTransformer. The target detection network model includes Backbone, Encoder, Decoder and detection head. Backbone includes a split block layer (Patch Partition), a linear embedding layer (Linear Embedding), a block merging layer (PatchMerging) and a Swin Transformer Block layer. Among them, the linear embedding layer of Swin Transformer and the Swin Transformer Block together constitute Stage 1, and the block merging layer and the Swin Transformer Block are combined in sequence to constitute Stage 2, Stage 3 and Stage 4. For the input image, it is first processed by the split image module, that is, the split block layer, and then passes through the four stages of Stage 1, Stage 2, Stage 3 and Stage 4 in sequence to complete the feature extraction of the image.
[0027] In one embodiment of the present invention, a split block layer is used to divide the input image into non-overlapping pixel blocks of fixed size, each block is a patch, and then these patches are flattened in the channel direction;
[0028] Linear embedding layer: The flattened vectors are linearly embedded through a convolutional layer to map these vectors to a feature space of fixed dimension. All embedded patch vectors are arranged in a certain order to form a patch sequence.
[0029] The block merging layer divides the adjacent blocks of the input feature map with the number of channels C into new patches, flattens and concatenates each new patch to form a new feature map (with the number of channels 4C), and converts it into a new feature map (with the number of channels 2C) through linear transformation. After the block merging layer is processed, the resolution of the feature map is halved and the number of channels is doubled;
[0030] Swin Transformer Block layer, as the core building block of Swin Transformer, includes several main components: Layer Normalization (LN), self-attention mechanism, Multi-Layer Perceptron (MLP) and residual connection. The self-attention mechanism includes two structures, one using the Window-based Multi-head Self-Attention (W-MSA) structure and the other using the Shifted Window-based Multi-head Self-Attention (SW-MSA) structure. The two structures are stacked and used in pairs.
[0031] The input is layer normalized to standardize the data. W-MSA divides the input features into windows of fixed size. Each window performs self-attention calculation independently. The self-attention results of each window are merged to form a new feature map. SW-MSA divides the feature map into sliding windows with a certain step size, introduces cross-window connections, and enhances the capture of contextual information. The step size here is half of the window size.
[0032] Multi-head self-attention calculation is performed in each sliding window, and the self-attention results of each sliding window are merged to form a new feature map. MLP contains two fully connected layers and a nonlinear activation function for further feature conversion and nonlinear transformation. The residual connection is applied to (S)W-MSA and MLP respectively. (S)W-MSA is a combination of W-MSA and SW-MSA to ensure that the gradient can be effectively back-propagated through the network. The processing steps of the feature map in Swin Transformer Block are as follows:
[0033]
[0034] in, and z l They represent the (S)W-MSA module output features and the MLP module output features of the lth block respectively. Swin Transformer Block effectively reduces the computational complexity when calculating multi-head self-attention through window partitioning and moving window mechanisms. The combination of W-MSA and SW-MSA achieves efficient feature extraction through local self-attention and cross-window information exchange. At the same time, the stability and convergence speed of training are ensured through residual connections and layer normalization. This makes Swin Transformer perform well in visual tasks, capturing detailed features while retaining global information.
[0035] In one embodiment of the present invention, the encoder is composed of a plurality of stacked encoder layers, each of which includes two main sublayers: a multi-head self-attention mechanism and a feed-forward neural network;
[0036] The self-attention mechanism captures the relationship between each position and other positions in the input feature map by computing query, key, and value matrices; the feed-forward neural network applies two linear transformations at each position with a nonlinear activation function in between; residual connections and layer normalization are used after each sublayer to stabilize the training process;
[0037] The input feature map is encoded with position information, and the position-encoded feature map is used as input. It passes through multiple encoder layers in sequence. Each encoder layer performs multi-head self-attention calculation and feedforward neural network processing, and cooperates with residual connection and layer normalization to finally output the encoded global context feature as the input of the decoder.
[0038] The specific structure of Encoder is as follows:
[0039]
[0040] FFN(x)=Max(0,xW1+b1)W2+b2
[0041] LayerNorm(x+SubLayer(x))
[0042] where Q, K, and V are query, key, and value matrices respectively, and d k is the dimension of the key; W1, W2 are weight matrices, b1, b2 are bias vectors; SubLayer(x) represents the output of the sublayer, and x is the input of the sublayer.
[0043] In one embodiment of the present invention, the decoder is composed of a plurality of decoder layers stacked together, each of which includes three main sub-layers: a self-attention mechanism, a cross-attention mechanism, and a feed-forward neural network;
[0044] The self-attention mechanism calculates attention weights between target queries to capture the relationship between target queries;
[0045] The cross-attention mechanism interacts the target query with the feature map output by the encoder to calculate the relationship between the target query and the image features;
[0046] The feedforward neural network applies two linear transformations at each position, with a nonlinear activation function in between. After each sublayer, residual connections and layer normalization are used to stabilize the training process. A fixed number of target queries are taken as input, and the target queries are passed through multiple decoder layers in sequence. Each decoder layer performs self-attention calculations, cross-attention calculations, and feedforward neural network processing, and finally outputs an embedded representation of each target query with residual connections and layer normalization.
[0047] The specific structure is as follows:
[0048]
[0049] FFN(x)=Max(0,xW1+b1)W2+b2
[0050] LayerNorm(x+SubLayer(x))
[0051] Among them, Q, K, and V in SelfAttention are all projections of the target query, d k is the dimension of the key; Q in CrossAttention is the projection of the target query, K and V are the projections of the encoder output; W1, W2 are weight matrices, b1, b2 are bias vectors; SubLayer(x) represents the output of the sublayer, and x is the input of the sublayer.
[0052] In one embodiment of the present invention, the detection head includes two branches: a classification branch and a regression branch. The classification branch is used to predict the target category, and the regression branch is used to predict the target bounding box. It is a key part of converting the output of the decoder into a specific target detection result. The detection head converts each target query output by the decoder into an embedded representation. One branch predicts the target category, and the other branch predicts the target bounding box. These two branches jointly generate the final detection result of each target query. The specific structure is as follows:
[0053] P class =Softmax(W class query+b class )
[0054] B box =σ(W box query+b box )
[0055] Where Wclass is the weight matrix of the classification branch, Wbox is the weight matrix of the regression branch, bclass, bbox are bias terms, σ is the sigmoid function, and query is the output embedding representation of the decoder.
[0056] In one embodiment of the present invention, the splitting block layer divides the image input as [H, W, 3] into patches with every 4×4 adjacent pixels. After splitting, there will be H / 4×W / 4 patches, each of which has a size of 4×4×3. These patches are flattened in the channel direction, and each patch forms a 1×1×48 vector. After all patches are flattened, a feature map of [H / 4, W / 4, 48] is formed.
[0057] In the linear embedding layer of Stage 1, the flattened vector is linearly embedded through a convolutional layer, and the [H / 4, W / 4, 48] vector is mapped to the feature space of [H / 4, W / 4, C]. All embedded patch vectors are arranged in a certain order to form a patch sequence. After the feature map processed by the linear embedding layer, it passes through two Swin Transformer Blocks, and the size of the feature map remains unchanged, which is still [H / 4, W / 4, C].
[0058] Stage 2 divides the adjacent blocks of the input feature map of [H / 4, W / 4, C] into new patches, each of which has a size of 2×2×C. These patches are flattened and concatenated to form a feature map of [H / 8, W / 8, 4C], which is converted into a new feature map [H / 8, W / 8, 2C] through a linear transformation. After processing through the block merging layer, the resolution of the feature map is halved and the number of channels is doubled. The feature map processed through the block merging layer is subjected to feature extraction by two Swin Transformer Blocks, and the size of the feature map remains unchanged, which is [H / 8, W / 8, 2C].
[0059] Stage 3 is similar to Stage 2. The feature map of the input [H / 8, W / 8, 2C] is first processed by the block merging layer to generate a feature map of [H / 16, W / 16, 4C]. The feature map processed by the block merging layer is subjected to 6 Swin TransformerBlocks for feature extraction. The size of the feature map remains unchanged, which is [H / 16, W / 16, 4C].
[0060] Stage4 is similar to Stage2 and Stage3. The feature map of the input [H / 16, W / 16, 4C] is first processed by the block merging layer to generate a feature map of [H / 32, W / 32, 8C]. The feature map processed by the block merging layer is extracted by two SwinTransformer Blocks. The size of the feature map remains unchanged, which is [H / 32, W / 32, 8C].
[0061] In one embodiment of the present invention, during the training process in step 3, DETR matches each target query output by the decoder with a real target. Since the number of targets output by the decoder is fixed, while the number of real targets in the image is not fixed, DETR uses the Hungarian algorithm for matching.
[0062] Use the Hungarian matching algorithm to solve the set y and Best match for: Collection Arrangement
[0063]
[0064] According to the collection Best arrangement Bring it into the loss function to solve the loss and perform gradient update
[0065]
[0066] In step 4, the round window membrane image to be detected is obtained, and the image to be detected is input into the optimal weight model for detection to generate a detection result.
[0067] The present invention solves the current technical bottleneck of detecting tiny targets such as semi-transparent, low-textured round window films.
[0068] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for detecting tiny weak soft tissue targets based on optimized DETR, characterized in that: include: Step 1: Obtain tiny semi-transparent weak soft tissue image data, and construct a data set including a training set and a validation set. Each sample in the data set is an image with the location and category of the tiny weak soft tissue target annotated; Step 2: Build a deep learning target detection network model based on optimized DETR; Step 3: Use the calibrated data set described in step 1 to input into the target detection network model described in step 2 for training. After the training is completed, output the trained DETR target detection model as the network model for tiny weak soft tissue target detection; Step 4: Get the image to be detected and input it into the DETR target detection network model trained in step 3 for detection.
2. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 1, characterized in that: The target detection network model includes Backbone, Encoder, Decoder and detection head. Backbone contains split block layer, linear embedding layer, block merging layer and Swin Transformer Block layer. The linear embedding layer of Swin Transformer and Swin Transformer Block together constitute Stage 1, and the block merging layer and Swin Transformer Block are combined to form Stage 2, Stage 3 and Stage 4 in sequence. For the input image, it is first processed by the split image module, namely the split block layer, and then passes through the four stages of Stage 1, Stage 2, Stage 3 and Stage 4 in sequence to complete the feature extraction of the image.
3. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 2, characterized in that: The split block layer is used to divide the input image into non-overlapping pixel blocks of fixed size, each block is a patch, and then these patches are flattened in the channel direction; Linear embedding layer: The flattened vectors are linearly embedded through a convolutional layer to map these vectors to a feature space of fixed dimension. All embedded patch vectors are arranged in a certain order to form a patch sequence. The block merging layer divides the adjacent blocks of the input feature map with the number of channels C into new patches, flattens and concatenates each new patch to form a new feature map, and converts it into a new feature map through linear transformation. After the block merging layer is processed, the resolution of the feature map is halved and the number of channels is doubled; Swin Transformer Block layer, as the core building block of Swin Transformer, includes layer normalization, self-attention mechanism, multi-layer perceptron and residual connection. The self-attention mechanism includes two structures, one using window multi-head self-attention structure and the other using sliding window multi-head self-attention structure. The two structures are stacked and used in pairs. The input is layer normalized to standardize the data. The window multi-head self-attention structure divides the input features into windows of fixed size. Each window performs self-attention calculation independently, and the self-attention results of each window are merged to form a new feature map. The sliding window multi-head self-attention structure divides the feature map into sliding windows according to a certain step size, introduces cross-window connections, performs multi-head self-attention calculations in each sliding window, and merges the self-attention results of each sliding window to form a new feature map. The multi-layer perceptron contains two fully connected layers and a nonlinear activation function for further feature conversion and nonlinear transformation. The processing steps of the feature map in the Swin Transformer Block are as follows: in, and z l They respectively represent the output features of the window multi-head self-attention structure, the sliding window multi-head self-attention structure module, and the multi-layer perceptron module of the lth block.
4. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 2, characterized in that: The Encoder is composed of multiple encoder layers stacked together, each of which contains two sub-layers: a multi-head self-attention mechanism and a feed-forward neural network. The self-attention mechanism captures the relationship between each position in the input feature map and other positions by computing the query, key, and value matrices; The feed-forward neural network applies two linear transformations at each position, with a nonlinear activation function in between; residual connections and layer normalization are used after each sublayer to stabilize the training process; The input feature map is encoded with position information, and the position-encoded feature map is used as input. It passes through multiple encoder layers in sequence. Each encoder layer performs multi-head self-attention calculation and feedforward neural network processing, and cooperates with residual connection and layer normalization to finally output the encoded global context feature as the input of the decoder. The specific structure of Encoder is as follows: FFN(x)=Max(0,xW1+b1)W2+b2 LayerNorm(x+SubLayer(x)) where Q, K, and V are query, key, and value matrices respectively, and d k is the dimension of the key; W1, W2 are weight matrices, b1, b2 are bias vectors; SubLayer(x) represents the output of the sublayer, and x is the input of the sublayer.
5. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 2, characterized in that: The decoder is composed of multiple decoder layers stacked together, each of which contains three sub-layers: self-attention mechanism, cross-attention mechanism, and feed-forward neural network; The self-attention mechanism calculates attention weights between target queries to capture the relationship between target queries; The cross-attention mechanism interacts the target query with the feature map output by the encoder to calculate the relationship between the target query and the image features; The feedforward neural network applies two linear transformations at each position, with a nonlinear activation function in between. After each sublayer, residual connections and layer normalization are used to stabilize the training process. A fixed number of target queries are taken as input, and the target queries are passed through multiple decoder layers in sequence. Each decoder layer performs self-attention calculations, cross-attention calculations, and feedforward neural network processing, and finally outputs an embedded representation of each target query with residual connections and layer normalization. The specific structure is as follows: FFN(x)=Max(0,xW1+b1)W2+b2 LayerNorm(x+SubLayer(x)) Among them, Q, K, and V in SelfAttention are all projections of the target query, d k is the dimension of the key; Q in CrossAttention is the projection of the target query, K and V are the projections of the encoder output; W1, W2 are weight matrices, b1, b2 are bias vectors; SubLayer(x) represents the output of the sublayer, and x is the input of the sublayer.
6. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 2, characterized in that: The detection head consists of two branches: the classification branch and the regression branch. The classification branch is used to predict the target category, and the regression branch is used to predict the target bounding box. The detection head converts each target query output by the decoder into an embedded representation. One branch predicts the target category, and the other branch predicts the target bounding box. These two branches jointly generate the final detection result for each target query. The specific structure is as follows: P class =Softmax(W class ·query+b class ) B box =σ(W box ·query+b box ) Where Wclass is the weight matrix of the classification branch, Wbox is the weight matrix of the regression branch, bclass, bbox are bias terms, σ is the sigmoid function, and query is the output embedding representation of the decoder.
7. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 2, characterized in that: The split block layer divides the input image [H, W, 3] into patches with every 4×4 adjacent pixels. After splitting, there will be H / 4×W / 4 patches, each with a size of 4×4×3. These patches are flattened in the channel direction, and each patch forms a 1×1×48 vector. After all patches are flattened, a feature map of [H / 4, W / 4, 48] is formed. In the linear embedding layer of Stage 1, the flattened vector is linearly embedded through a convolutional layer, and the [H / 4, W / 4, 48] vector is mapped to the feature space of [H / 4, W / 4, C]. All embedded patch vectors are arranged in a certain order to form a patch sequence. After the feature map processed by the linear embedding layer, it passes through two Swin Transformer Blocks, and the size of the feature map remains unchanged, which is still [H / 4, W / 4, C]. Stage 2 divides the adjacent blocks of the input feature map of [H / 4, W / 4, C] into new patches, each of which has a size of 2×2×C. These patches are flattened and concatenated to form a feature map of [H / 8, W / 8, 4C], which is converted into a new feature map [H / 8, W / 8, 2C] through a linear transformation. After processing through the block merging layer, the resolution of the feature map is halved and the number of channels is doubled. The feature map processed through the block merging layer is subjected to feature extraction by two Swin Transformer Blocks, and the size of the feature map remains unchanged, which is [H / 8, W / 8, 2C]. In Stage 3, the feature map of the input [H / 8, W / 8, 2C] is first processed by the block merging layer to generate a feature map of [H / 16, W / 16, 4C]. The feature map processed by the block merging layer is subjected to 6 Swin Transformer Blocks for feature extraction. The size of the feature map remains unchanged, which is [H / 16, W / 16, 4C]. In Stage 4, the feature map of the input [H / 16, W / 16, 4C] is first processed by the block merging layer to generate a feature map of [H / 32, W / 32, 8C]. The feature map processed by the block merging layer is subjected to feature extraction by two Swin Transformer Blocks, and the size of the feature map remains unchanged, which is [H / 32, W / 32, 8C].
8. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 1, characterized in that: In the training process of step 3, DETR matches each target query output by the decoder with the real target. DETR uses the Hungarian algorithm for matching. Use the Hungarian matching algorithm to solve the set y and Best match for: Collection Arrangement According to the collection Best arrangement Bring it into the loss function to solve the loss and perform gradient update 9. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 1, characterized in that: In step 4, the round window membrane image to be detected is obtained, and the image to be detected is input into the optimal weight model for detection to generate a detection result.
10. The method for detecting tiny weak soft tissue targets based on optimized DETR according to claim 1, characterized in that: In step three, the network is trained by using the divided data set and inputting it into the target detection network model for iterative training to obtain the optimal weight model.