Ultrasonic liver detection tracking method based on DDet model

By improving the adaptive threshold of the DiffusionDet model and ByteTrack framework, the problem of insufficient accuracy in ultrasonic liver detection and tracking technology is solved, and more efficient detection and tracking of liver lesions is achieved, improving the accuracy and efficiency of detection and tracking.

CN118967641APending Publication Date: 2024-11-15CHANGZHOU UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411099144.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Existing ultrasound liver detection and tracking technologies are not accurate enough in the face of respiratory movements and noise interference, making it difficult to accurately capture liver lesions, especially smaller lesions, and are difficult to distinguish.

Method used

Using the ultrasonic liver detection and tracking method based on the DDet model, the DiffusionDet model was improved, and the ResNet50 feature extraction network was replaced as the Mix Transformer network, which enhanced global attention, and expanded the target box using the furthest point sampling algorithm, combined with the ByteTrack framework for adaptive threshold improvement , improve the model's spatial perception and tracking capabilities.

Benefits of technology

It improves the detection accuracy and tracking efficiency of liver lesions, can more accurately reflect the location of liver lesions, and moves with the lesion in real time, enhances the spatial perception ability of the model, and improves the detection and tracking accuracy and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967641A_ABST
    Figure CN118967641A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of image processing, in particular to an ultrasonic liver detecting and tracking method based on a DDet model, which comprises the following steps: acquiring an ultrasonic liver video, segmenting the ultrasonic liver video into a plurality of image frames, and marking a lesion area; the method comprises the following steps: constructing an improved Diffusion Det model, replacing a ResNet50 feature extraction network of the Diffusion Det model with a Mix Transform network, and enhancing the global attention of the Diffusion Det model; and the lesion part recognition capability is improved. According to the invention, the problem that the existing image detection tracking method cannot completely capture the diseased region and cannot capture the small diseased region is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an ultrasonic liver detection and tracking method based on a DDet model. Background Art

[0002] Ultrasound imaging has become a widely used technology in medical diagnosis due to its portability, low cost and non-invasiveness.

[0003] The liver, as the largest organ in the human abdomen, is often accompanied by diseases, so the diagnosis and treatment of liver disease has become a focus of attention.

[0004] Since the position of the liver can be affected by respiratory movement, any incorrect position or omission during diagnosis and treatment will have a huge impact. Taking into account factors such as severe noise interference and low contrast in ultrasound images or videos, traditional ultrasound liver detection and tracking technology is often not accurate enough and it is difficult to accurately control the guidance of liver respiratory movement. Therefore, a more advanced detection and tracking technology is urgently needed.

[0005] The patent publication number is CN102438529A, which discloses a method and system for automatically detecting lesions in medical images. However, this method simply divides the lesion site into light and dark colors, and then uses a clustering algorithm to detect the lesion site. However, the detection accuracy is not high, especially for lesions with smaller shapes that are difficult to distinguish from normal sites.

[0006] The paper Melanoma Detection Using Deep Learning-Based Classifications proposed a method for detecting melanoma based on deep learning. The use of deep learning methods requires the use of a large data set. When the data samples are limited, the accuracy will be greatly affected. In addition, the increase in training samples will affect the training time of the model. Summary of the invention

[0007] In view of the shortcomings of the existing methods, the present invention solves the problem that the existing image detection and tracking methods cannot completely capture the lesion site and cannot capture smaller lesion sites.

[0008] The technical solution adopted by the present invention is: an ultrasonic liver detection and tracking method based on a DDet model comprises the following steps:

[0009] Step 1: Obtain an ultrasound liver video, segment it into several image frames, and mark the lesion area;

[0010] Step 2: Build an improved DiffusionDet model, replace the ResNet50 feature extraction network of the DiffusionDet model with the Mix Transformer network, enhance the global attention of the DiffusionDet model, and improve the ability to identify lesions.

[0011] As a preferred implementation of the present invention, the Mix Transformer network is improved, and the improved Mix Transformer includes four overlapping patch merging operations and four Transformer blocks. The overlapping patch merging operation performs downsampling at four scales respectively, and the Transformer block connects efficient self-attention and Mix-FFN in series. A spatial perception module is inserted between two Transformer blocks, and the spatial perception module connects dilated partial convolution and strip pooling in series, and performs residual connection.

[0012] As a preferred implementation of the present invention, the original target frame of the DiffusionDet model is improved, and the farthest point sampling algorithm is used to expand the original target frame to obtain a new target frame.

[0013] As a preferred embodiment of the present invention, the calculation of the new target frame includes:

[0014] First, a number of points M are randomly generated in the original target frame, and a number of key points N are extracted using the farthest point sampling algorithm;

[0015] Then calculate the coordinates of the lower left corner of the new target box as (x min -d1,y min -d2), upper right corner coordinate (x max +d1,y max +d2);x min =min(x1,x2,...,x N ), x max =max(x1,x2,...,x N ), y min =min(y1,y2,...,y N ), y max =max(y1,y2,...,y N ); (x N ,y N ) is the coordinate of the random point in the Nth original target box.

[0016] As a preferred implementation manner of the present invention, improving the DiffusionDet model further includes: replacing the YOLOX framework in the DiffusionDet model with the ByteTrack framework.

[0017] As a preferred embodiment of the present invention, the adaptive threshold of the ByteTrack framework is improved, and the improved adaptive threshold formula is:

[0018]

[0019] Among them, α f is a fixed value, S det is the confidence of the detection box, The operation increases the impact of confidence.

[0020] As a preferred embodiment of the present invention, an ultrasonic liver detection and tracking system based on the DDet model includes: a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement an ultrasonic liver detection and tracking method based on the DDet model.

[0021] As a preferred embodiment of the present invention, a computer readable medium stores a computer program code, and when the computer program code is executed by a processor, the ultrasonic liver detection and tracking method based on the DDet model is implemented.

[0022] Beneficial effects of the present invention:

[0023] 1. The farthest point sampling algorithm is used to improve the position and size of the original target frame, so that the target frame can more accurately reflect the location of liver lesions and detect liver lesions more accurately and sensitively;

[0024] 2. Improve the adaptive threshold of the ByteTrack framework. When the probe is moved dynamically, the target frame can accurately follow the movement of the detected lesion in real time, improving the efficiency of the model in tracking the lesion.

[0025] 3. Enhance the spatial perception ability of the model so that the model can detect the lesion site in the liver ultrasound image more carefully; improve the accuracy and efficiency of the model in detecting and tracking the lesion site. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the ultrasonic liver detection and tracking method based on the DDet model of the present invention;

[0027] Figure 2 It is a partial structural diagram of the DiffusionDet model of the present invention;

[0028] Figure 3 It is a schematic diagram of constructing the farthest target frame of the present invention;

[0029] Figure 4 It is the Transformer block structure diagram;

[0030] Figure 5 This is the target detection and target tracking effect diagram. DETAILED DESCRIPTION

[0031] The present invention is further described below in conjunction with the accompanying drawings and embodiments. This figure is a simplified schematic diagram, which only illustrates the basic structure of the present invention in a schematic manner, and therefore it only shows the components related to the present invention.

[0032] like Figure 1 As shown, the ultrasonic liver detection and tracking method based on the DDet model includes the following steps:

[0033] The DiffusionDet model is referred to as the DDet model;

[0034] Step 1: Obtain an ultrasound liver video, segment it into several image frames, and mark the lesion area;

[0035] Divide the training set and test set into detection and tracking tasks respectively, and use point labels to generate target box labels;

[0036] For example, the liver video image frames are divided into 30 frames. The training set of the tracking task includes a 14×14×2048 training set and a test set of the detection task. 14*14 is the image size and 2048 is the number. The training set and test set of the detection task are divided on the same video sequence.

[0037] Liver video labeling is done by taking the point label as the center and obtaining a 60-pixel-long rectangle as the original target box label.

[0038] Step 2: Build an improved DiffusionDet model. The original DeffusionDet model includes the encoder feature extraction network ResNet50, target box filling, target box Gaussian noise and Sparse RCNN detection decoder.

[0039] The ultrasound image of 448×448×3 is input, and a multi-scale feature map is generated by ResNet50, and the deepest feature dimension is 14×14×2048. For the training process, the target box with padding and Gaussian noise is fed together with the deepest feature to the Sparse RCNN detection decoder, that is, the stage of training the model to learn from the noise box to the target box; for the inference process, since the new target box is separated from the DeffusionDet model, the noise box (that is, the original target box) is randomly sampled from the Gaussian distribution, and the rest is the same as the training process; the fed multi-scale feature map is cropped to obtain the ROI feature and then sent to the detection head to obtain the box regression and classification results.

[0040] The ResNet50 encoder only needs to be run once to reduce the amount of computation, and the Sparse RCNN detection decoder used for inference uses random target boxes instead of fixed learned boxes, and the detector head is reused during each sampling iteration.

[0041] The original target box of the DiffusionDet model is randomly filled by expanding the number of target boxes following the Gaussian distribution until the expected filling box value of 8 is reached.

[0042] The Gaussian noise of the target box noise adding process of the DiffusionDet model is given by the following formula: To control:

[0043]

[0044] Among them, z0 is the target box label sample without noise, z t is the target box label sample after adding noise, z0~Ν(0,Ι) indicates that the noise is a standard normal distribution, and the noise is proportional to A monotonically decreasing cosine scheduling is adopted.

[0045] The DiffusionDet model uses a collection of prediction losses, including l1 and GIoU losses between the predicted box and the true label, and the focal loss of the category, which is weighted as follows:

[0046]

[0047] Among them, L cls is the classification loss of the category, L l1 and L giou are the l1 and GIoU losses of the predicted box and the true label, respectively. λ1, λ2, and λ3 are correction coefficients, which take values ​​of 2, 5, and 2, respectively.

[0048] like Figure 2 As shown in the figure, the improved DiffusionDet model includes: replacing the ResNet50 backbone network with the Mix Transformer backbone network in Segformer to enhance global attention and fix the defect that the original ResNet50 network only focuses on local information; during detection, it can better distinguish between the lesion site and the normal site that is dark due to imaging reasons; at the same time, a spatial perception module is added after each scale of the MixTransformer backbone to enhance the pure Transformer architecture's ability to perceive thyroid nodules with noise interference; finally, the original target box in the DiffusionDet model is replaced with the farthest target box.

[0049] The MixTransformer backbone consists of four overlapping patch merges and four Transformer blocks, such as Figure 4 It is a Transformer block, in which the overlapping patch merging operation is downsampled at four scales respectively. The Transformer block is connected in series with efficient self-attention and Mix-FFN; efficient self-attention achieves less computation by reducing the size of K in the self-attention calculation, and Mix-FFN replaces the position encoding by capturing the position information through 3×3 convolution. The calculation method is as follows:

[0050] x Mix-FFN =MLP(GELU(Conv 3×3 (MLP(x in ))))+x in (3)

[0051] Among them, MLP is the fully connected layer, GELU is the activation function, and x in is the input of Mix-FFN.

[0052] A spatial perception module is inserted between every two Transformer blocks. The spatial perception module connects the dilated partial convolution and two strip pooling in series, supplemented by residual connections.

[0053] For the dilated partial convolution, the input features The channel dimension is evenly divided into They perform partial convolution calculations with different dilation rates, where partial convolution means only The features of the channel are calculated, and then the Concat operation is performed in the channel dimension to restore the feature map. The formula for this process is as follows:

[0054]

[0055] Among them, Pdc m Represents dilated partial convolutions with different dilation rates, m=1, 2, 5, 7 are dilation rates, and Conv1 is a 1×1 convolution.

[0056] Then, yes Perform two types of strip pooling separately, and then multiply them to restore the original dimension. The calculation process is as follows:

[0057]

[0058] Among them, Conv1 is a 1×1 convolution, Ap h and Ap w They are horizontal and vertical strip pooling operations respectively.

[0059] Finally, a residual connection is performed to obtain the final output.

[0060] Adding a spatial perception module between Transformer blocks enhances the position perception capability of the detection model, enables detection of all positions in the medical image of the liver, and accurately feedbacks the location of the lesion.

[0061] Construct the farthest target frame to expand the original target frame to meet the DiffusionDet model's (diffusion detection model) requirements for more targets;

[0062] like Figure 2 , 3 As shown, the construction of the farthest target box includes:

[0063] First, a number of points M are randomly generated in the original target frame. In this embodiment, M=20. Then, a number of key points N are extracted from the M points using the farthest point sampling algorithm (FPS). In this embodiment, N=8. Then, these 8 points are used as the center points to generate a new target frame. The specific method of generating the target frame is as follows:

[0064] Assume that the coordinates of the 8 points generated are (x i ,y i ),i∈[1,8], set: x min =min(x1,x2,…,x8),x max =max(x1,x2,…,x8),y min =min(y1,y2,...,y8),y max =max(y1,y2,…,y8);

[0065] calculate:

[0066] Generate the lower left corner coordinates of the new rectangular target box (x min -d1,y min -d2), upper right corner coordinate (x max +d1,y max +d2).

[0067] The farthest point sampling algorithm (FPS) first randomly selects one of the 20 points as the selected sampling point, and then searches for the point with the largest distance to the selected sampling point among the remaining points, classifies it as selected, and then keeps updating the distance and iterating the loop until the expected 8 points are found; using the farthest target box method not only reduces the amount of calculation required for using other methods, but also enables the trained target box to cover the detected target more accurately.

[0068] For the tracking of ultrasound liver targets, the reasoning of the DiffusionDet model is improved, and the detection box (i.e., the new target box) is input into the improved ByteTrack tracking framework, which processes the high-score detection boxes and the low-score detection boxes separately.

[0069] The ByteTrack framework first divides the detection model results into high-score frames and low-score frames, and uses Kalman filtering to predict the position of each trajectory in the current frame; then, the first association is performed, and the IoU cost matrix is ​​passed into the Hungarian algorithm for matching for the high-confidence detection frame to obtain the successfully matched trajectory and detection frame and the detection frame of the unmatched trajectory and the trajectory of the unmatched detection frame; the low-score frame is associated with the unmatched trajectory in the previous step for the second time, and the IoU matrix is ​​passed into the Hungarian algorithm, the remaining detection frames are deleted, and the unmatched trajectory is left for the next frame; finally, a new trajectory is created for the unmatched high-score detection frame, and the unmatched trajectory is allowed to wait for 30 frames before matching; the above process is iterated until the end of the video frame.

[0070] The ByteTrack framework is used to replace the original YOLOX framework in the DiffusionDet model, and the NSA Kalman filter is used to replace the original Kalman filter. Finally, the high-resolution frame is adaptively embedded in the appearance, where the adaptive value increases the influence of confidence.

[0071] NSA Kalman filter adds adaptive calculation of noise covariance on the basis of Kalman filter. The formula is as follows:

[0072]

[0073] Among them, R m is the fixed measurement noise covariance, c m is the confidence of the detection box in state m.

[0074] For adaptive appearance embedding, the pre-trained re-ID and exponential moving average (EMA) are used to update the appearance state. The EMA formula is as follows:

[0075] e k =αe k-1 +(1-α)e now (8)

[0076] Among them, e now The appearance of the matching detection added for the current step, e k-1 is the appearance of the matching detection added in the previous step, and α is the adaptive threshold.

[0077] The improved adaptive threshold α is as follows:

[0078]

[0079] Among them, α f is a fixed value, in this example it is 0.95, S det is the confidence of the detection box, The operation increases the impact of confidence.

[0080] The improved adaptive threshold α can make the size of the target box more appropriate to the detected object and can change more sensitively according to the size of the detected object.

[0081] Then the appearance cost A can be obtained by using the standard cosine similarity of the trajectory and the improved EMA. app ; Finally, the appearance cost A of the appearance model app And the IoU cost A of the motion model IoU Perform weighted incoming Hungarian algorithm, the formula is as follows:

[0082] C=λ4×A IoU +λ5×A app (10)

[0083] Among them, λ4 and λ5 are correction coefficients.

[0084] The ultrasonic liver detection and tracking system based on the DDet model includes:

[0085] Liver video processing module, farthest target frame module, detection module, tracking module and web display module; among them,

[0086] The liver video processing module is used to obtain ultrasound liver video segmentation into multiple image frames, divide the training set and test set according to the detection and tracking tasks respectively, and generate target box labels using point labels.

[0087] The farthest target box module uses the farthest target box extension to expand the original target box to meet the needs of the diffusion detection model for more targets;

[0088] The detection module uses the DiffusionDet model based on MixTransformer with a spatial perception module to perform ultrasound liver target detection tasks;

[0089] The tracking module uses the trained improved DiffusionDet model to substitute into the improved ByteTrack tracking framework.

[0090] The web display module realizes the detection, tracking and visualization of ultrasound liver objects in the web-based system for medical diagnosis, such as Figure 5 This is the actual effect diagram of detection and tracking.

[0091] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. An ultrasonic liver detection and tracking method based on the DDet model, characterized in that: The following steps are involved: Step 1: Obtain an ultrasound liver video, segment it into several image frames, and mark the lesion area; Step 2: Build an improved DiffusionDet model, replace the ResNet50 feature extraction network of the DiffusionDet model with the MixTransformer network, enhance the global attention of the DiffusionDet model, and improve the ability to identify lesions.

2. The ultrasonic liver detection and tracking method based on the DDet model according to claim 1, characterized in that: The MixTransformer network is improved, including four overlapping patch merging and four Transformer blocks. The overlapping patch merging operation downsamples at four scales respectively, and the Transformer block is connected in series with efficient self-attention and Mix-FFN; a spatial perception module is inserted between two Transformer blocks. The spatial perception module is connected in series with dilated partial convolution and strip pooling, and residual connection is performed.

3. The ultrasonic liver detection and tracking method based on the DDet model according to claim 2, characterized in that: The original target frame of the DiffusionDet model is improved, and the farthest point sampling algorithm is used to expand the original target frame to obtain a new target frame.

4. The ultrasonic liver detection and tracking method based on the DDet model according to claim 3 is characterized in that: The calculation of the new target box includes: First, several points M are randomly generated in the original target frame, and several key points N are extracted using the farthest point sampling algorithm; Then, calculate the coordinates of the lower left corner of the new target box as (x min -d1,y min -d2), upper right corner coordinate (x max +d1,y max +d 2 ); x min =min(x1,x2,...,x N ), x max =max(x1,x2,...,x N ), y min =min(y1,y2,...,y N ), y max =max(y1,y2,...,y N ); (x N y N ) is the Nth original, Coordinates of random points within the target box.

5. The ultrasonic liver detection and tracking method based on the DDet model according to claim 1, characterized in that: Improving the DiffusionDet model also includes: using the ByteTrack framework to replace the YOLOX framework in the DiffusionDet model.

6. The ultrasonic liver detection and tracking method based on the DDet model according to claim 5, characterized in that: Improve the adaptive threshold of the ByteTrack framework. The improved adaptive threshold formula is: Among them, α f is a fixed value, S det is the confidence of the detection box, The operation increases the impact of confidence.

7. The ultrasonic liver detection and tracking system based on the DDet model is characterized by: include: a memory for storing instructions executable by a processor; A processor, configured to execute instructions to implement the ultrasonic liver detection and tracking method based on the DDet model as described in any one of claims 1 to 6.

8. A computer readable medium storing a computer program code, characterized in that: When the computer program code is executed by a processor, the computer program code implements the ultrasonic liver detection and tracking method based on the DDet model as described in any one of claims 1 to 6.