Dynamic focus tracking method and device and computer storage medium

Abstract

The invention provides a lesion dynamic tracking method and device and a computer storage medium, and the method comprises the following steps: obtaining any frame of ultrasonic image containing a lesion; two different network models are combined to position a focus; extracting focus boundary pixel points and tracking the boundary pixel points; calculating the coordinates of the tracked boundary pixel points in the current frame of ultrasonic image; and performing curve fitting on the calculated boundary pixel point coordinates of the current frame of ultrasonic image and displaying the boundary pixel point coordinates in the current frame of ultrasonic image. According to the lesion dynamic tracking method, the position of the lesion can be accurately tracked under the dynamic condition, and the problem that the position of the lesion is very easy to change in the process that a puncture needle penetrates into the tissue due to the fact that the tissue to be punctured is soft tissue and the breathing of a patient in an ultrasound-guided biopsy puncture examination operation is solved; and the difficulty of an ultrasound-guided biopsy puncture operation is reduced.

Description

Technical Field

[0001] The present application relates to the technical field of medical diagnosis, and in particular to a method, device and computer storage medium for dynamic tracking of lesions. Background Art

[0002] Ultrasound-guided biopsy is a diagnostic method that uses a fine puncture needle to pierce the diseased tissue to remove a small portion of living tissue for examination. The diagnostic method of biopsy has the advantages of high accuracy, low trauma, and high reliability. The clinical difficulties that doctors often encounter during biopsy are the patient's dynamic breathing and the soft tissue to be punctured, which causes the lesion position to change in real time and increases the difficulty of the puncture. Summary of the invention

[0003] The purpose of the present application is to overcome the defects of the existing ultrasound-guided biopsy puncture examination surgery that the lesion position changes in real time due to the patient's dynamic breathing and the soft tissue to be punctured, which increases the difficulty of the puncture surgery.

[0004] To this end, the present application provides a method for dynamic tracking of lesions, comprising the following steps:

[0005] Acquire any frame of ultrasound image containing the lesion;

[0006] Two different network models were combined to localize the lesions;

[0007] Extract lesion boundary pixels and track boundary pixels;

[0008] Calculate the coordinates of the tracked boundary pixel points in the current frame ultrasound image;

[0009] The calculated boundary pixel coordinates of the current frame ultrasound image are curve-fitted and displayed in the current frame ultrasound image.

[0010] Furthermore, the use of two different network models to locate the lesion in combination includes:

[0011] Preprocessing ultrasound images containing lesions;

[0012] The preprocessed ultrasound image is input into the trained YOLOv5 network model for rough positioning of the lesion;

[0013] The rough positioning results output by the YOLOv5 network model are input into the U-Net network model to accurately locate the lesion.

[0014] Furthermore, the ultrasonic image containing the lesion includes: performing image denoising and contrast stretching on the ultrasonic image in sequence.

[0015] Furthermore, the image denoising includes: resetting the grayscale value of each pixel of the ultrasound image to the median value of all pixels in a fixed neighborhood;

[0016] And / or, the contrast stretching includes: homogenizing the grayscale histogram of the ultrasound image in a grayscale range with similar grayscale values.

[0017] Furthermore, the fixed neighborhood is selected from any one of a 3*3 neighborhood, a 5*5 neighborhood and a 7*7 neighborhood.

[0018] Further, the tracking boundary pixel points are the lesion boundary pixel points extracted by tracking using the LK optical flow pyramid algorithm, and the tracking of the lesion boundary pixel points extracted by using the LK optical flow pyramid algorithm includes:

[0019] The extracted pixel points of the lesion boundary are used as reference points to be tracked, and the ultrasound image where the reference points are located is used as the reference image;

[0020] The continuous frames of ultrasound images after the reference image are used as floating images;

[0021] The LK optical flow pyramid algorithm is used to track the reference points on the floating image to obtain matching point pairs, where the matching point pairs are pixel points that match the reference points.

[0022] Furthermore, using the LK optical flow pyramid algorithm to track the extracted lesion boundary pixels also includes:

[0023] The RANSAC algorithm is used to remove matching point pairs that fail to be tracked.

[0024] Furthermore, a binary image boundary extraction algorithm is used to extract lesion boundary pixels.

[0025] In a second aspect, the present application also provides a device for dynamic tracking of lesions, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the aforementioned dynamic tracking method of lesions is implemented.

[0026] In a third aspect, the present application further provides a computer storage medium having a program stored thereon, and when the program is executed by a processor, it is used to implement the aforementioned method for dynamic tracking of lesions.

[0027] The technical solution of this application has at least the following advantages:

[0028] 1. The dynamic lesion tracking method of the present application can accurately track the position of the lesion in a dynamic situation, overcoming the problem that the position of the lesion is very easy to change during the puncture needle penetrating the tissue due to the soft tissue to be punctured and the influence of the patient's breathing during the ultrasound-guided biopsy puncture examination operation. It helps doctors to accurately locate the lesion in real time and reduces the difficulty of ultrasound-guided biopsy puncture surgery.

[0029] 2. Compared with tracking the corner points of the target in the traditional dynamic tracking process, the dynamic tracking method of the lesion in the present application can extract and track the boundary pixel points of the lesion after accurately locating the lesion position, thereby improving the accuracy of lesion tracking.

[0030] 3. Traditional deep learning-based methods mainly use neural network models to autonomously learn feature representations for tracking, which requires a large amount of computing resources to infer. The dynamic lesion tracking method of the present application uses a combination of two different network models to locate the lesion, thereby achieving accurate positioning of the lesion, and then assists in lesion tracking with the LK optical flow pyramid algorithm. This improves the tracking accuracy of the lesion while improving the tracking efficiency, thereby improving the accuracy and efficiency of ultrasound-guided puncture surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flow chart of the method for dynamic lesion tracking provided in Example 1 of the present application;

[0033] Figure 2 This is an ultrasound image after locating the lesion during the puncture process of Example 2 of the present application;

[0034] Figure 3 This is an ultrasound image for tracking the location of the lesion during the puncture process of Example 2 of the present application;

[0035] Figure 4 Schematic diagram of a lesion dynamic tracking device provided in Example 3 of the present application. DETAILED DESCRIPTION

[0036] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0038] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, it can also be the internal connection of two components, it can be a wireless connection, or it can be a wired connection. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0039] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0040] When performing a biopsy puncture operation on a patient under ultrasound guidance, the patient's dynamic breathing and the fact that the tissue to be punctured is soft tissue cause the location of the lesion to change in real time, which increases the difficulty of the puncture operation.

[0041] Example 1

[0042] In view of the above problems, the present application provides a method for dynamic tracking of lesions. Figure 1 A flow chart of a method for dynamic lesion tracking provided in Example 1 of the present application. Although the processes described below include multiple operations that occur in a specific order, it should be clearly understood that these processes may also include more or fewer operations, which may be performed sequentially or in parallel (e.g., using a parallel processor or a multi-threaded environment).

[0043] The biopsy puncture procedure can be assisted by dynamically tracking the lesion, so as to accurately locate the lesion during the patient's dynamic breathing and when the punctured tissue is soft tissue, such as Figure 1 As shown, the lesion dynamic tracking method includes the following steps:

[0044] S101, acquiring any frame of ultrasound image containing a lesion;

[0045] Select a suitable ultrasound probe, adjust the gain, depth, focus and other parameters of the ultrasound image, and scan the part to be examined by manually or by controlling the ultrasound probe at the end of the robot to obtain an ultrasound image containing the lesion. The part to be examined can be the lungs, liver, kidneys, prostate, lymph nodes, cavities and other parts that require biopsy puncture.

[0046] S102, using two different network models to combine and locate the lesion;

[0047] The trained YOLOv5 network model is used to perform target detection to roughly locate the lesion, and then the trained U-Net network model is used to perform image segmentation to accurately locate the lesion, including:

[0048] S201: preprocessing an ultrasound image containing a lesion;

[0049] When preprocessing an ultrasound image, the ultrasound image is subjected to image denoising and contrast stretching in sequence. Specifically, the ultrasound image is denoised by resetting the grayscale value of each pixel of the ultrasound image to the median value of all pixels in a fixed neighborhood; the denoised ultrasound image is subjected to contrast stretching, which includes: homogenizing the grayscale histogram of the ultrasound image in a grayscale interval with similar grayscale. Image denoising removes the influence of noise, and contrast stretching improves the contrast of the lesion area in the ultrasound image, thereby improving the accuracy of locating the lesion in the ultrasound image and the accuracy of tracking the lesion in dynamic conditions.

[0050] When denoising the image, the fixed neighborhood is selected from any one of a 3*3 neighborhood, a 5*5 neighborhood and a 7*7 neighborhood, preferably a 5*5 neighborhood, that is, the grayscale value of each pixel of the ultrasound image is reset to the median value of all pixels in the 5*5 neighborhood.

[0051] S202: inputting the preprocessed ultrasound image into the trained YOLOv5 network model to perform rough positioning of the lesion;

[0052] The ultrasound image containing the lesion collected by the ultrasound probe is preprocessed and input into the trained YOLOv5 network model to output the center position, length, width and position of the lesion detection frame. The lesion detection frame is used to frame the position of the lesion in the ultrasound image to roughly locate the lesion.

[0053] S203: inputting the rough positioning result output by the YOLOv5 network model into the U-Net network model to accurately locate the lesion;

[0054] The detection frame output by the YOLOv5 network model is input into the trained U-net network model to accurately locate the lesion. When accurately locating, the Overlap-tile strategy of the U-net network is used together with the image segmentation. The lesion detection frame area of ​​the roughly located ultrasound image is first mirror-filled, and then reconstructed into image segments after mirror-filling. Then, multiple image segments are input into the trained U-net network model, and the ultrasound image with the lesion segmented is output. By using the Overlap-tile strategy of the U-net network together with the image segmentation, the image size of the initial input and final output of the U-net network model is consistent, which reduces the possibility of reducing the accuracy of lesion segmentation due to inconsistent image sizes between the input and output of the U-net network model.

[0055] S103, extracting lesion boundary pixel points and tracking the boundary pixel points;

[0056] The binary image boundary extraction algorithm is used to extract the coordinates of the lesion boundary pixel points in the ultrasound image from which the lesion is segmented; the LK optical flow pyramid algorithm is used to track the extracted lesion boundary pixel points. The LK optical flow pyramid algorithm is used to track the extracted lesion boundary pixel points, and the following steps are included:

[0057] The extracted pixel points at the boundary of the lesion are used as the reference points to be tracked, the ultrasound image where the reference points are located is used as the reference image, and the continuous frame ultrasound images after the reference image are used as floating images. The LK optical flow pyramid algorithm is used to track the reference points on the floating image to obtain matching point pairs, which are pixel points that match the reference points. The RANSAC algorithm is then used to remove the matching point pairs that fail to be tracked, and the remaining points that are successfully tracked on the reference image are the boundary pixel points of the lesion tracked based on the reference image.

[0058] Compared with tracking the corner points of the target in the traditional dynamic tracking process, the lesion dynamic tracking method of the present application can extract and track the boundary pixel points of the lesion after accurately locating the lesion position, thereby improving the accuracy of lesion tracking.

[0059] Specifically, the present application can be understood as follows: the ultrasound image containing the lesion obtained in step S101 is the reference image, and then in step S102, the YOLOv5 network model and the U-Net network model are combined to locate the lesion in the reference image, and then the binary image boundary extraction algorithm is used to extract the boundary pixel points of the lesion in the reference image, and the LK optical flow pyramid algorithm is used to take the boundary pixel points of the lesion in the reference image as the reference points, and the boundary pixel points of the lesion in the continuous frame ultrasound image after the reference image are tracked to obtain matching point pairs, and for each frame of ultrasound image after the reference image, there are matching point pairs that match the reference points of the reference image, and the matching point pairs that match the reference points of the reference image are the boundary pixel points in the frame of ultrasound image, and after the RANSAC algorithm is used to remove the matching point pairs that fail to track each frame of ultrasound image after the reference image, the remaining matching points are the boundary pixel points of the lesion in the frame of ultrasound image;

[0060] S104, calculating the coordinates of the tracked boundary pixel points in the current frame ultrasound image;

[0061] S105 , performing curve fitting on the calculated coordinates of the boundary pixel points of the current frame ultrasound image and displaying them in the current frame ultrasound image.

[0062] When the continuous frame images after the reference image are displayed in sequence, the coordinates of the boundary pixel points in the current ultrasound image that are tracked based on the reference points of the reference image in the current ultrasound image that is displayed in real time are calculated, and then the calculated coordinates of the boundary pixel points of the current ultrasound image are curve-fitted and displayed in the current ultrasound image. If the current ultrasound image is the reference image, the coordinates of the tracked reference points in the reference image are calculated and displayed in real time after curve fitting.

[0063] The dynamic lesion tracking method of the present application can accurately track the position of the lesion in a dynamic situation, overcoming the problem that the position of the lesion is very easy to change during the puncture needle penetrating the tissue due to the soft tissue to be punctured and the influence of the patient's breathing during the ultrasound-guided biopsy puncture examination operation. It helps doctors to accurately locate the lesion in real time and reduces the difficulty of ultrasound-guided biopsy puncture surgery.

[0064] The traditional deep learning-based method mainly uses the neural network model to autonomously learn feature representation for tracking, which requires a large amount of computing resources to infer. It uses two different network models in combination to locate the lesion, thereby achieving accurate positioning of the lesion, and then assists in lesion tracking with the LK optical flow pyramid algorithm. This improves the tracking accuracy of the lesion while improving the tracking efficiency, thereby improving the accuracy and efficiency of ultrasound-guided puncture surgery.

[0065] The training of the above U-Net network model includes the following steps:

[0066] Collecting a number of ultrasound images containing lesions, scaling, cropping, rotating and flipping the collected ultrasound images to increase sample data of various lesion locations;

[0067] Label the sample data and divide the labeled sample data into a training set, a verification set, and a test set. For example, the sample data may contain 3070 images, of which 2630 images are used for training, 300 images are used for verification, and the remaining images are used for testing;

[0068] Build a U-net network model, use the training set for training, use the verification machine to verify and adjust the parameters of the U-net neural network, and then use the test set to evaluate the model with accuracy and recall rate to obtain a trained U-net network model.

[0069] The training method of the YOLOv5 network model is basically the same as that of the U-net network model.

[0070] Example 2

[0071] In this embodiment, during an ultrasound-guided breast biopsy puncture operation on a patient, the lesion is tracked according to the lesion dynamic tracking method of Embodiment 1. Figure 2 This is an ultrasound image after locating the lesion during the puncture process of Example 2 of the present application. Figure 3 This is an ultrasound image of tracking the location of the lesion during the puncture process of Example 2 of the present application; Figure 2 and Figure 3 It can be seen that the dynamic lesion tracking method of the present application can accurately track the position of the lesion under dynamic conditions, overcoming the problem that the position of the lesion is very easy to change during the puncture needle penetrating the tissue due to the fact that the tissue to be punctured is soft tissue and the influence of the patient's breathing during the ultrasound-guided biopsy puncture examination operation. It helps doctors to accurately locate the lesion in real time and reduces the difficulty of ultrasound-guided biopsy puncture surgery.

[0072] Example 3

[0073] refer to Figure 4 The present application also provides a lesion dynamic tracking device, which includes a processor 410 and a memory 420.

[0074] The processor 410 may include one or more processing cores. The processor 410 uses various interfaces and lines to connect various parts of the entire device, and executes various functions of the device and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 420, and calling data stored in the memory 420. Optionally, the processor 410 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 410 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU) and a modem. Among them, the CPU mainly processes the operating system and application programs, etc.; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 410, but implemented separately through a chip.

[0075] Optionally, when the processor 410 executes the program instructions in the memory 420 , the dynamic lesion tracking method provided by the above-mentioned various method embodiments is implemented.

[0076] The memory 420 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 420 includes a non-transitory computer-readable storage medium. The memory 420 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 420 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data created according to the use of the device, etc.

[0077] Optionally, the present application also provides a storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the dynamic lesion tracking method provided by the above method embodiment.

[0078] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection created by this application.

Claims

1. A method for dynamic tracking of lesions, characterized in that: The steps include: Acquire any frame of ultrasound image containing the lesion; Two different network models were combined to localize the lesions; Extract lesion boundary pixels and track boundary pixels; Calculate the coordinates of the tracked boundary pixel points in the current frame ultrasound image; The calculated boundary pixel coordinates of the current frame ultrasound image are curve-fitted and displayed in the current frame ultrasound image.

2. The method for dynamic lesion tracking according to claim 1, characterized in that: The method of using two different network models to locate the lesion includes: Preprocessing ultrasound images containing lesions; The preprocessed ultrasound image is input into the trained YOLOv5 network model for rough positioning of the lesion; The rough positioning results output by the YOLOv5 network model are input into the U-Net network model to accurately locate the lesion.

3. The method for dynamic lesion tracking according to claim 2, characterized in that: The preprocessing of the ultrasound image containing the lesion includes: performing image denoising and contrast stretching on the ultrasound image in sequence.

4. The method for dynamic lesion tracking according to claim 3, characterized in that: The image denoising comprises: resetting the gray value of each pixel of the ultrasound image to the median value of all pixels in a fixed neighborhood; And / or, the contrast stretching includes: homogenizing the grayscale histogram of the ultrasound image in a grayscale range with similar grayscale values.

5. The method for dynamic lesion tracking according to claim 4, characterized in that: The fixed neighborhood is selected from any one of a 3*3 neighborhood, a 5*5 neighborhood and a 7*7 neighborhood.

6. The method for dynamic lesion tracking according to claim 1, characterized in that: The tracking boundary pixel points are the lesion boundary pixel points extracted by tracking using the LK optical flow pyramid algorithm. Tracking the lesion boundary pixel points extracted by using the LK optical flow pyramid algorithm includes: The extracted pixel points of the lesion boundary are used as reference points to be tracked, and the ultrasound image where the reference points are located is used as the reference image; The continuous frames of ultrasound images after the reference image are used as floating images; The LK optical flow pyramid algorithm is used to track the reference points on the floating image to obtain matching point pairs, where the matching point pairs are pixel points that match the reference points.

7. The method for dynamic lesion tracking according to claim 6, characterized in that: Tracking the extracted lesion boundary pixels using the LK optical flow pyramid algorithm also includes: The RANSAC algorithm is used to remove matching point pairs that fail to be tracked.

8. The method for dynamic lesion tracking according to claim 1, characterized in that: The binary image boundary extraction algorithm is used to extract the lesion boundary pixels.

9. A lesion dynamic tracking device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the method for dynamic tracking of lesions according to any one of claims 1 to 7 is implemented.

10. A computer storage medium having a program stored thereon, wherein when the program is executed by a processor, the program is used to implement the method for dynamic tracking of lesions according to any one of claims 1 to 7.

Images