Computer readable storage medium

By using a combination of cumulative sparse optical flow and convolutional neural network during the puncture process, the problem of inaccurate needle tip positioning during the puncture process is solved, accurate and real-time tracking of the needle tip is achieved, and the puncture effect is improved.

CN119055327APending Publication Date: 2024-12-03IMSONIC MEDICAL CHINA INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411105166.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-01-22
Filing Date
2024-08-13
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

During the puncture process, the puncture needle is prone to deviating from the ultrasound plane due to the low resolution of ultrasound image, severe spot noise and difficulty in maintaining stability during the puncture process, resulting in the puncture needle being prone to deviating from the ultrasound plane, making it difficult to accurately identify the needle tip, affecting the puncture effect, and it is especially difficult to track the movement of the needle tip in real time.

Method used

Using a computer-readable storage medium based on cumulative sparse optical flow, the cumulative sparse optical flow field in the needle tip candidate area is calculated by obtaining the ultrasonic image time series, and the needle tip positioning is achieved by using a convolutional neural network combined with grayscale map features to achieve needle tip positioning. Accurate and real-time tracking.

Benefits of technology

Improve the accuracy and sensitivity of needle tip recognition, can distinguish needle tip movement and interference based on multi-frame time information, reduce positioning errors, and does not rely on external equipment or needle body information, and is suitable for fuzzy or unclear needle body situations. .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119055327A_ABST
    Figure CN119055327A_ABST
Patent Text Reader

Abstract

The invention discloses a computer readable storage medium, and belongs to the technical field of target tracking. A computer program is stored on a computer readable storage medium, when the computer program is executed by a processor, the puncture needle tip positioning method based on accumulated sparse optical flow is realized, and the method comprises the following steps: acquiring a time sequence of an ultrasonic original image; determining a needle tip candidate region set; calculating an accumulated sparse optical flow field in the needle tip candidate area; and positioning the needle tip by using the convolutional neural network. Accurate and robust motion information is provided by accumulating the sparse optical flow field, and the optical flow field reflects the overall motion trend and condition of the scene around the needle tip. When analysis is carried out in a small area, due to the accumulative effect, single false needle tip movement caused by interference and image noise is not prone to being misjudged as needle tip related movement, and the accuracy of needle tip positioning is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims the priority of Chinese patent application No. 2024100871519 filed on January 22, 2024 with the State Intellectual Property Office of China, and the invention name “Puncture needle tip positioning method based on accumulated sparse optical flow”, the entire contents of which are incorporated by reference in this application. Technical Field

[0002] The present invention relates to the field of target tracking technology, and more specifically, to a computer-readable storage medium. Background Art

[0003] Ultrasound guidance plays an important role in puncture. However, current problems include low resolution of ultrasound images, severe speckle noise, and the difficulty in keeping the needle stable during the puncture process. The puncture needle easily deviates from the ultrasound plane, and the puncture needle appears blurred or even lost in the ultrasound image, making it difficult to accurately identify. This ambiguity makes it difficult for personnel to quickly and accurately locate the puncture needle, thus affecting the puncture effect.

[0004] The Chinese patent document with application number 2023116222716 discloses a method for locating the puncture needle body in an ultrasound image. This method mainly locates the entire puncture needle body. When the expected needle insertion position is unknown or the visibility of the puncture needle body is poor, the method in this patent can meet the need for needle body positioning. However, the positioning method in this solution makes it difficult to track the needle tip in real time and cannot improve the accuracy of needle tip recognition.

[0005] Therefore, developing an accurate and real-time method for locating the puncture needle tip in ultrasound images has important practical significance. Summary of the invention

[0006] In response to the above problems, the present invention provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, a method for locating the puncture needle tip based on accumulated sparse optical flow is implemented. The method realizes needle tip tracking, which not only improves the accuracy of needle tip identification, but also can track the needle tip in real time.

[0007] In order to achieve the above object, the technical solution provided by the present invention is:

[0008] The present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, a method for locating the tip of a puncture needle based on accumulated sparse optical flow is implemented. The method includes:

[0009] Acquire a time sequence of original ultrasound images, where the time sequence of original ultrasound images refers to an ordered set of ultrasound images arranged in chronological order and composed of a current frame image and a series of historical frame images;

[0010] Determine a needle tip candidate region set, wherein the needle tip candidate region set includes a plurality of potential regions where the needle tip appears in the original ultrasound image;

[0011] Calculate the accumulated sparse optical flow field in the needle tip candidate area, where the optical flow field is the motion field of the moving target in the original ultrasound image in the x and y directions, and the optical flow field is calculated by using a pyramid sparse optical flow algorithm on the current frame ultrasound image and the historical frame ultrasound image;

[0012] The needle tip is located by using a convolutional neural network. The method comprises: firstly, performing feature coupling on the accumulated sparse optical flow field and the grayscale image of the original ultrasound image in the needle tip candidate area; then taking the coupling feature as the input of the convolutional neural network classifier, and the classifier classifies each pixel point in the needle tip candidate area; setting the needle tip area according to the pixel point classification result; calculating the needle tip coordinates, and taking the obtained coupling feature in the needle tip area as the input of the convolutional neural network regressor to obtain the coordinate value N of the needle tip in the needle tip area. r (x, y), according to the relative position of the needle tip area in the original ultrasound image, the coordinate value of the needle tip in the original image is obtained.

[0013] The present invention also proposes a puncture needle tip positioning system based on accumulated sparse optical flow, comprising:

[0014] An acquisition unit, the acquisition unit is used to acquire a time sequence of an original ultrasound image, wherein the time sequence of the original ultrasound image refers to an ordered set of ultrasound images arranged in chronological order and composed of a current frame image and a series of historical frame images;

[0015] A determination unit, the determination unit is used to determine a needle tip candidate region set, wherein the needle tip candidate region set includes a plurality of potential regions where the needle tip appears in the original ultrasound image;

[0016] A calculation unit, which is used to calculate the accumulated sparse optical flow field in the needle tip candidate area, where the optical flow field is the motion field of the moving target in the original ultrasound image in the x and y directions, and the optical flow field is calculated by using a pyramid sparse optical flow algorithm on the current frame ultrasound image and the historical frame ultrasound image;

[0017] A positioning unit is used to locate the needle tip using a convolutional neural network. The method of locating the needle tip using a convolutional neural network includes: firstly, performing feature coupling on the accumulated sparse optical flow field and the grayscale image of the original ultrasound image in the needle tip candidate area; then, using the coupling feature as the input of the convolutional neural network classifier, and the classifier classifies each pixel point in the needle tip candidate area; setting the needle tip area according to the pixel point classification result; calculating the needle tip coordinates, and using the obtained coupling feature in the needle tip area as the input of the convolutional neural network regressor to obtain the coordinate value N of the needle tip in the needle tip area. r (x, y), according to the relative position of the needle tip area in the original ultrasound image, the coordinate value of the needle tip in the original image is obtained.

[0018] Furthermore, the method for obtaining the time series of the original ultrasound image is:

[0019] The time series V(x,y,t)∈R of the accumulated T frames of ultrasound raw images W×H×T , where x is the horizontal coordinate of the pixel point in the original ultrasound image, y is the vertical coordinate of the pixel point in the original ultrasound image, t is the frame number of the original ultrasound image, W is the width of the original ultrasound image in pixels, and H is the height of the original ultrasound image in pixels; V(:,:,T-1) represents the current frame ultrasound image, and V(:,:,0) represents the earliest historical frame ultrasound image in the time series of the original ultrasound image.

[0020] Furthermore, the step of determining a needle tip candidate region set includes:

[0021] Step 1: Calculate the differential image. Subtract the original ultrasound image of the historical frame from the original ultrasound image of the current frame to obtain the differential image, which is used to obtain the motion information in the current frame. The calculation process is as shown in formula (1):

[0022] D=V(:,:,T-1)-V(:,:,T-1-T n ) (1)

[0023] In formula (1), D is the difference image, T n is the frame number interval between the current frame ultrasound original image and the historical frame ultrasound original image;

[0024] Step 2: Binarize the difference image. The motion interference in the low-brightness area is removed by binarizing the difference image D. The binarization process is shown in formula (2):

[0025]

[0026] In formula (2), B is the binary image of the difference image, M is the grayscale average of D, and α is the control coefficient;

[0027] Step 3: Connected domain analysis: First, calculate the connected domain of the binary image B of the differential image, then select the N connected domains with the largest area, and extract the square area with a side length of L as the needle tip candidate area with the geometric center of the selected connected domain as the center. Finally, all the needle tip candidate areas are grouped into a set U N , U N The image includes multiple regions identified as potential needle tips in the original ultrasound image.

[0028] Furthermore, the step of calculating the accumulated sparse optical flow field in the tip candidate region includes:

[0029] Step 1: Arrange corner points. Set corner points evenly at intervals d in the original ultrasound image to form a sparse corner point matrix S∈N w×h , where w represents the number of corner points in a row of the original ultrasound image under the interval d configuration, and h represents the number of corner points in a column of the original ultrasound image under the interval d configuration;

[0030] Step 2: In the time series V(x, y, t) of the original ultrasound images with accumulated T frames, the t value is set to T-1;

[0031] Step 3: Set t to t-1, take the t-th frame ultrasound image V(:,:,t) and the current frame ultrasound image V(:,:,T-1) as the input of the pyramid sparse optical flow algorithm, and use the pyramid sparse optical flow algorithm on each corner point p(m,n) to obtain an optical flow vector ftp∈R 1×1×2 , where m is the horizontal coordinate of the corner point in the ultrasound image, and n is the vertical coordinate of the corner point in the ultrasound image;

[0032] Step 4: Execute step 3 until the value of t is equal to 0. When the value of t is equal to 0, T-1 sparse optical flow fields ftp∈R are obtained. M×N×2 , set the optical flow amplitude of non-corner positions in the sparse optical flow field to 0; add these T-1 sparse optical flow fields on the time axis to obtain the global cumulative sparse optical flow field F(x,y,c)∈R containing two directional components x(c=0) and y(c=1) M×N×2 , where c represents the channel dimension of the sparse optical flow field;

[0033] Step 5: Obtain the global accumulated sparse optical flow field in each tip candidate area. n ∈R L ×L×2 , the needle tip candidate region is the needle tip candidate region set U N The elements in .

[0034] Furthermore, the feature coupling of the accumulated sparse optical flow field in the needle tip candidate region and the grayscale image of the original ultrasound image includes: processing each needle tip candidate region in the needle tip candidate region set, and performing feature coupling on the accumulated sparse optical flow field F having two channel components x and y in each needle tip candidate region. n The single-channel ultrasound original image grayscale image is coupled to generate the three-channel coupling feature F of each needle tip candidate area. c ∈R L×L×3 ;

[0035] Furthermore, the coupling feature is used as the input of the convolutional neural network classifier, and the classifier classifies each pixel point in the needle tip candidate area, including: classifying each needle tip candidate area in the needle tip candidate area set to obtain the probability P (P ∈ [0, 1]) that each pixel point in the needle tip candidate area is a needle tip, and then obtaining the probability set of all pixel points in the needle tip candidate area of ​​the current frame ultrasound image P n (x,y), and calculate P n The average of the (x,y) probabilities Each candidate region of the needle tip will generate a probability mean

[0036] Furthermore, the step of setting the needle tip region according to the pixel classification result includes: firstly calculating the probability mean corresponding to each needle tip candidate region in the needle tip candidate region set Then, determine the largest value among all probability means when Greater than the probability threshold P t When , the needle tip candidate region corresponding to the maximum probability mean value is set as the needle tip region R n .

[0037] Compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0038] (1) The accumulated sparse optical flow field in the present invention provides accurate and robust motion information, and the optical flow field reflects the overall motion trend and situation of the scene around the needle tip. When analyzing in a small area, due to the cumulative effect, a single pseudo needle tip motion caused by interference and image noise is not easily misjudged as needle tip related motion, which greatly improves the accuracy of needle tip positioning.

[0039] (2) The accumulated sparse optical flow field of the present invention can provide time information from multiple frames. Compared with the existing needle tip positioning method that only uses single or two frames of image information, the needle tip positioning method disclosed in the present invention can more effectively distinguish needle tip motion from needle tip-like interference.

[0040] (3) Since the needle tip positioning method disclosed in the present invention spatially couples the accumulated sparse optical flow field and grayscale information representing the motion features, the resulting coupling features contain rich spatiotemporal information, which can more comprehensively capture the subtle changes in the needle tip motion and improve the sensitivity of needle tip positioning.

[0041] (4) The needle tip positioning method disclosed in the present invention does not rely on any external auxiliary equipment and only uses continuous multi-frame images as algorithm input, making it more convenient and practical.

[0042] (5) The needle tip positioning method disclosed in the present invention does not rely on the information of the puncture needle body. Even when the needle body is only partially visible or completely invisible, the method can still provide robust needle tip positioning.

[0043] (6) When it is known that there is a needle tip point in a candidate needle tip area, the needle tip positioning method disclosed in the present invention performs needle tip coordinate regression in the rectangular needle tip area, which can greatly reduce the positioning error and improve the accuracy of needle tip positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flow chart of a method for positioning the puncture needle tip;

[0045] Figure 2 A flowchart for determining a set of candidate regions for the needle tip;

[0046] Figure 3 Flowchart for accumulating sparse optical flow fields;

[0047] Figure 4 Schematic diagram of accumulating sparse optical flow fields;

[0048] Figure 5 A schematic diagram of the calculation steps for determining a set of candidate needle tip regions in a puncture needle tip positioning method;

[0049] Figure 6 This is a schematic diagram of accumulating sparse optical flow fields in the needle tip candidate area in the puncture needle tip positioning method;

[0050] Figure 7 It is a schematic diagram of the accumulated sparse optical flow field in the needle tip candidate area and the grayscale image of the ultrasound original image in the puncture needle tip positioning method;

[0051] Figure 8 It is a schematic diagram of the positioning result of the puncture needle tip in the current frame image in the puncture needle tip positioning method. DETAILED DESCRIPTION

[0052] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments.

[0053] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention, so they have no substantial technical significance. Any modification of the structure, change of the proportion relationship or adjustment of the size, without affecting the effects and purposes that can be achieved by the present invention, should still fall within the scope of the technical content disclosed by the present invention. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and so on quoted in this specification are only for the convenience of description, and are not used to limit the scope of implementation. The change or adjustment of their relative relationship should also be regarded as the scope of implementation of the present invention without substantially changing the technical content.

[0054] In one embodiment, if Figure 1 As shown, the puncture needle tip positioning method based on accumulated sparse optical flow provided in the embodiment of the present invention consists of four stages:

[0055] Stage 1: Acquire the time series of original ultrasound images.

[0056] The time series of the ultrasonic original images refers to an ordered set of ultrasonic images arranged in chronological order, consisting of a current frame image and a series of historical frame images.

[0057] Stage 2: Determine the set of candidate needle tip regions.

[0058] The needle tip candidate region set includes a plurality of potential regions where the needle tip appears in the original ultrasound image.

[0059] Stage 3: Calculate the accumulated sparse optical flow field in the tip candidate area.

[0060] The optical flow field is the motion field of the moving target in the original ultrasound image in the x and y directions. The optical flow field is calculated by using a pyramid sparse optical flow algorithm (BD Lucas and T. Kanade, "An iterative image registration technique with an application to stereovision," in Proc. Int. Joint. Conf. Artif Intell., pp. 674-679, Canada, 1981) on the current frame ultrasound image and the historical frame ultrasound image.

[0061] Stage 4: Use convolutional neural network to locate the needle tip.

[0062] In this embodiment, the accumulated sparse optical flow field provides accurate and robust motion information, and the optical flow field reflects the overall motion trend and situation of the scene around the needle tip. When analyzing in a small area, due to the cumulative effect, a single pseudo needle tip motion caused by interference and image noise is not easily misjudged as needle tip related motion, which greatly improves the accuracy of needle tip positioning.

[0063] In addition, the accumulated sparse optical flow field can provide temporal information from multiple frames. Compared with the existing needle tip localization method that only uses single-frame or two-frame image information, the puncture needle tip localization method based on accumulated sparse optical flow can more effectively distinguish needle tip motion and interference similar to the needle tip.

[0064] As a further improvement of the scheme of this embodiment, the method for obtaining the time series of the original ultrasound image in stage 1 is to accumulate the time series V(x, y, t)∈R of the original ultrasound image of T frames W×H×T , where x is the horizontal coordinate of the pixel point in the original ultrasound image, y is the vertical coordinate of the pixel point in the original ultrasound image, t is the frame number of the original ultrasound image, W is the width of the original ultrasound image in pixels, and H is the height of the original ultrasound image in pixels.

[0065] In this embodiment, the value of T is set to 20, V(:,:,19) represents the current frame ultrasound image, and V(:,:,0) represents the earliest historical frame ultrasound image in the ultrasound original image time series.

[0066] As a further improvement of the solution of this embodiment, stage 2 determines a set of candidate needle tip regions, such as Figure 2 As shown, the following steps are included:

[0067] Figure 5 (a) is an ultrasound image collected during the needle puncture process. Usually, the size of the original ultrasound image is much larger than the needle tip of the puncture needle. Narrowing the needle tip positioning range to a specific candidate area can reduce positioning errors and computational complexity.

[0068] Step 1: Calculate the difference image.

[0069] The difference image is obtained by subtracting the original ultrasound image of the historical frame from the original ultrasound image of the current frame, and is used to obtain the motion information in the current frame. The obtained difference image D is as follows: Figure 5 As shown in (b), the calculation process is as follows:

[0070] D=V(:,:,T-1)-V(:,:,T-1-T n ) (1)

[0071] In formula (1), T n It is the frame number interval between the current frame ultrasound original image and the historical frame ultrasound original image.

[0072] Step 2: Binarize the difference image.

[0073] By performing a binarization operation on the difference image D to remove motion interference in low-brightness areas, the binary image B of the difference image is obtained as follows: Figure 5 As shown in (c), the binarization process is as follows:

[0074]

[0075] In formula (2), M is the grayscale average value of D, and α is the control coefficient.

[0076] Step 3: Connected domain analysis.

[0077] First, the connected domain of the binary graph B of the differential image is calculated, and then the N connected domains with the largest area are selected. The larger the area of ​​the connected domain, the more significant the motion characteristics. Taking the geometric center of the selected connected domain as the center, a square area with a side length of L is extracted as the needle tip candidate area. Finally, all the needle tip candidate areas are grouped into a set U N , U N It includes multiple areas identified as potential needle tips in the original ultrasound image.

[0078] In this embodiment, when L is set to 80 pixels and N is set to 3, the three candidate needle tip regions are obtained as follows: Figure 5 As shown in (d), ROI1, ROI2 and ROI3, Figure 5 (d) shows the three candidate regions of the needle tip obtained based on the geometric centers of the three connected domains with the largest areas.

[0079] As a further improvement of the solution of this embodiment, stage 3 calculates the accumulated sparse optical flow field in the tip candidate area, such as Figure 3 The following steps are involved:

[0080] Step 1: Arrange the corner points.

[0081] Corner points are evenly set at intervals of d pixels in the original ultrasound image to form a sparse corner point matrix S∈N w ×h , where w represents the number of corner points in a row of the original ultrasound image with a configuration of d pixel points, and h represents the number of corner points in a column of the original ultrasound image with a configuration of d pixel points.

[0082] Step 2: In the time series V(x, y, t) of 20 frames of original ultrasound images acquired in stage 1, the t value is set to 19.

[0083] Step 3: Set t to t-1, use the t-th frame ultrasound image V(:,:,t) and the current frame ultrasound image V(:,:,19) as the input of the pyramid sparse optical flow algorithm, and use the pyramid sparse optical flow algorithm on each corner point p(m,n) to obtain an optical flow vector ftp∈R 1×1×2 , where m is the horizontal coordinate of the corner point in the ultrasound image, and n is the vertical coordinate of the corner point in the ultrasound image. The optical flow vector can reflect the tiny motion information in the ultrasound image.

[0084] Step 4: Execute step 3 until the value of t is equal to 0. When the value of t is equal to 0, 19 sparse optical flow fields ftp∈R are obtained. M×N×2 , set the optical flow amplitude of non-corner points in the sparse optical flow field to 0. Add these 19 sparse optical flow fields on the time axis to obtain the global cumulative sparse optical flow field F(x, y, c)∈R containing two directional components x(c=0) and y(c=1) M×N×2 , where c represents the channel dimension of the sparse optical flow field, and the accumulated sparse optical flow field is as follows Figure 4 As shown, the solid circle marked in the figure is the location of the needle tip.

[0085] Step 5: Obtain the cumulative sparse optical flow field of the global cumulative sparse optical flow field in each pinpoint candidate area. The cumulative sparse optical flow fields in the three pinpoint candidate areas are obtained as follows: Figure 6 As shown, Figure 6 (a) Figure 5 (d) The optical flow field in the ROI1 area, Figure 6 (b) Figure 5 (d) The optical flow field in the ROI2 area, Figure 6 (c) Figure 5 (d) Optical flow field in ROI3 area.

[0086] Will Figure 6 and Figure 5 By comparing the original ultrasound image and the differential image, it can be observed that the calculated optical flow field reflects the motion of the scene in the corresponding area.

[0087] As a further improvement of the solution of this embodiment, stage 4 uses a convolutional neural network to locate the needle tip, including the following steps:

[0088] Step 1: Feature coupling is performed between the accumulated sparse optical flow field in the needle tip candidate area and the grayscale image of the original ultrasound image.

[0089] Each needle tip candidate region in the needle tip candidate region set is processed, and the accumulated sparse optical flow field in each needle tip candidate region and the grayscale image of the original ultrasound image are coupled. Each needle tip candidate region will obtain an accumulated sparse optical flow field F with two channel components, x and y. nAnd the three-channel feature F of the single-channel grayscale image c ∈R L×L×3 like Figure 7 As shown, Figure 7 (a) Figure 5 (d) Schematic diagram of spatial coupling in ROI1 area. Figure 7 (b) Figure 5 (d) Schematic diagram of spatial coupling in ROI2 area. Figure 7 (c) Figure 5 (d) Schematic diagram of spatial coupling in ROI3 area.

[0090] The three-channel features fuse time and space information, which helps to calculate the motion trajectory of the needle tip in the time series and its accurate position in space.

[0091] Since the needle tip positioning method of this embodiment spatially couples the accumulated sparse optical flow field and grayscale information representing motion features, the resulting coupled features contain rich spatiotemporal information, can more comprehensively capture subtle changes in needle tip motion, and improve the sensitivity of needle tip positioning.

[0092] In another embodiment, in addition to optical flow features and grayscale features, other image features such as gradient, divergence, etc. are also input into the convolutional neural network for classification as features required for classification. The convolutional neural network classifier can be designed using the ResNet structure. In order to reduce the amount of calculation, the standard four residual layers are reduced to three. The first residual layer contains four convolutional layer sequences, the second residual layer contains four convolutional layer sequences, and the third residual layer contains two convolutional layer sequences. This configuration optimizes the computational efficiency by reducing the number of residual blocks and balances the model depth while retaining the advantages of the residual structure in ResNet.

[0093] Step 2: Use the coupling features as the input of the convolutional neural network classifier, and the classifier classifies each pixel in the candidate area of ​​the needle tip.

[0094] Classify each needle tip candidate region in the needle tip candidate region set to obtain the probability P (P ∈ [0, 1]) that each pixel point in the needle tip candidate region is a needle tip, and then obtain the probability set of all pixels in the needle tip candidate region of the current frame ultrasound image. P n (x,y), and calculate P n The average of the (x,y) probabilities Each candidate region of the needle tip will generate a probability mean

[0095] Step 3: Set the needle tip area according to the pixel classification result.

[0096] First, calculate the probability mean corresponding to each needle tip candidate area in the needle tip candidate area set Then, determine the largest value among all probability means when Greater than the probability threshold P t When , the needle tip candidate region corresponding to the maximum probability mean value is set as the needle tip region R n ,exist Figure 5 The needle tip candidate region ROI1 corresponding to the maximum probability mean value is set as the needle tip region R n In this embodiment, P t Set to 0.8.

[0097] Step 4: Calculate the needle tip coordinates.

[0098] The coupling features in the needle tip area obtained in step 1 are used as the input of the convolutional neural network regressor to obtain the coordinate value N of the needle tip in the needle tip area. r (x, y), according to the relative position of the needle tip area in the original ultrasound image, the coordinate value of the needle tip in the original image is obtained, and the positioning of the puncture needle tip in the ultrasound image is completed, such as Figure 8 As shown, Figure 8 The dot in the circular area marked in the middle is the location of the located needle tip.

[0099] When it is known that there is a needle tip point in a candidate needle tip area, the needle tip positioning method of this embodiment performs needle tip coordinate regression in the rectangular needle tip area, which can greatly reduce the positioning error and improve the accuracy of needle tip positioning.

[0100] The puncture needle tip positioning method based on accumulated sparse optical flow in this embodiment, on the one hand, does not rely on any external auxiliary equipment and only uses continuous multi-frame images as algorithm input, making it more convenient to use; on the other hand, it does not rely on the puncture needle body information. Even if the needle body is only partially visible or completely invisible, the method can still provide robust needle tip positioning.

[0101] The present invention and its embodiments are described schematically above, and the description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. Therefore, if a person skilled in the art is inspired by it and designs a structural method and an embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they shall all fall within the protection scope of the present invention.

Claims

1. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, a puncture needle tip positioning method based on accumulated sparse optical flow is implemented, and the method comprises: Acquire a time sequence of original ultrasound images, where the time sequence of original ultrasound images refers to an ordered set of ultrasound images arranged in chronological order and composed of a current frame image and a series of historical frame images; Determine a needle tip candidate region set, wherein the needle tip candidate region set includes a plurality of potential regions where the needle tip appears in the original ultrasound image; Calculate the accumulated sparse optical flow field in the needle tip candidate area, where the optical flow field is the motion field of the moving target in the original ultrasound image in the x and y directions, and the optical flow field is calculated by using a pyramid sparse optical flow algorithm on the current frame ultrasound image and the historical frame ultrasound image; The needle tip is located by using a convolutional neural network. The method comprises: firstly, performing feature coupling on the accumulated sparse optical flow field and the grayscale image of the original ultrasound image in the needle tip candidate area; then taking the coupling feature as the input of the convolutional neural network classifier, and the classifier classifies each pixel point in the needle tip candidate area; setting the needle tip area according to the pixel point classification result; calculating the needle tip coordinates, and taking the obtained coupling feature in the needle tip area as the input of the convolutional neural network regressor to obtain the coordinate value N of the needle tip in the needle tip area. r (x, y), according to the relative position of the needle tip area in the original ultrasound image, the coordinate value of the needle tip in the original image is obtained.

2. The computer-readable storage medium according to claim 1, It is characterized in that When the computer program is executed by a processor, the method for obtaining the time series of ultrasound original images is: The time series V(x,y,t)∈R of the accumulated T frames of ultrasound raw images W×H×T , where x is the horizontal coordinate of the pixel point in the original ultrasound image, y is the vertical coordinate of the pixel point in the original ultrasound image, t is the frame number of the original ultrasound image, W is the width of the original ultrasound image in pixels, and H is the height of the original ultrasound image in pixels; V(:,:,T-1) represents the current frame ultrasound image, and V(:,:,0) represents the earliest historical frame ultrasound image in the time series of the original ultrasound image.

3. The computer-readable storage medium according to claim 1, It is characterized in that When the computer program is executed by a processor, determining a needle tip candidate region set includes: Step 1: Calculate the differential image. Subtract the original ultrasound image of the historical frame from the original ultrasound image of the current frame to obtain the differential image, which is used to obtain the motion information in the current frame. The calculation process is as shown in formula (1): D=V(:,:,T-1)-V(:,:,T-1-T n ) (1) In formula (1), D is the difference image, T n is the frame number interval between the current frame ultrasound original image and the historical frame ultrasound original image; Step 2: Binarize the difference image. The motion interference in the low-brightness area is removed by binarizing the difference image D. The binarization process is shown in formula (2): In formula (2), B is the binary image of the difference image, M is the grayscale average of D, and α is the control coefficient; Step 3: Connected domain analysis: First, calculate the connected domain of the binary image B of the differential image, then select the N connected domains with the largest area, and extract the square area with a side length of L as the needle tip candidate area with the geometric center of the selected connected domain as the center. Finally, all the needle tip candidate areas are grouped into a set U N , U N The image includes multiple regions identified as potential needle tips in the original ultrasound image.

4. The computer-readable storage medium according to claim 1, It is characterized in that When the computer program is executed by the processor, the accumulated sparse optical flow field in the tip candidate region is calculated, including: Step 1: Arrange the corner points. Set the corner points evenly at intervals d in the original ultrasound image to form a sparse corner point matrix S∈N w×h , where w represents the number of corner points in a row of the original ultrasound image under the interval d configuration, and h represents the number of corner points in a column of the original ultrasound image under the interval d configuration; Step 2: In the time series V(x, y, t) of the original ultrasound images with accumulated T frames, the t value is set to T-1; Step 3: Set t to t-1, take the t-th frame ultrasound image V(:,:,t) and the current frame ultrasound image V(:,:,T-1) as the input of the pyramid sparse optical flow algorithm, and use the pyramid sparse optical flow algorithm on each corner point p(m,n) to obtain an optical flow vector ftp∈R 1×1×2 , where m is the horizontal coordinate of the corner point in the ultrasound image, and n is the vertical coordinate of the corner point in the ultrasound image; Step 4: Execute step 3 until the value of t is equal to 0. When the value of t is equal to 0, T-1 sparse optical flow fields ftp∈R are obtained. M ×N×2 , set the optical flow amplitude of non-corner positions in the sparse optical flow field to 0; add these T-1 sparse optical flow fields on the time axis to obtain the global cumulative sparse optical flow field F(x,y,c)∈R containing two directional components x(c=0) and y(c=1) M×N×2 , where c represents the channel dimension of the sparse optical flow field; Step 5: Obtain the global accumulated sparse optical flow field in each tip candidate area. n ∈R L×L×2 , the needle tip candidate region is the needle tip candidate region set U N The elements in .

5. The computer-readable storage medium according to claim 1, It is characterized in that When the computer program is executed by the processor, the accumulated sparse optical flow field in the needle tip candidate area and the grayscale image of the ultrasound original image are feature coupled, including: Each needle tip candidate region in the needle tip candidate region set is processed, and the accumulated sparse optical flow field F with two channel components x and y in each needle tip candidate region is calculated. n The single-channel ultrasound original image grayscale image is coupled to generate the three-channel coupling feature F of each needle tip candidate area. c ∈R L×L×3 .

6. The computer-readable storage medium according to claim 1, It is characterized in that When the computer program is executed by the processor, the coupling feature is used as the input of the convolutional neural network classifier, and the classifier classifies each pixel point in the needle tip candidate area, including: Classify each needle tip candidate region in the needle tip candidate region set to obtain the probability P (P ∈ [0, 1]) that each pixel point in the needle tip candidate region is a needle tip, and then obtain the probability set of all pixels in the needle tip candidate region of the current frame ultrasound image. P n (x,y), and calculate P n The average of the (x,y) probabilities Each candidate region of the needle tip will generate a probability mean 7. The computer-readable storage medium of claim 1, It is characterized in that When the computer program is executed by the processor, the needle tip area is set according to the pixel classification result, including: First, calculate the probability mean corresponding to each needle tip candidate area in the needle tip candidate area set Then, determine the largest value among all probability means when Greater than the probability threshold P t When , the needle tip candidate region corresponding to the maximum probability mean value is set as the needle tip region R n .