Puncture needle tip detection method and device in ultrasonic image and storage medium
By fusing the pin tip grayscale image and visual attention map into ultrasound images, and using Kalman filtering and deep convolutional neural network optimization models, the anti-interference and generalization problems of pin tip detection in ultrasound images are solved, achieving higher positioning accuracy and stability, and meeting the needs of clinical applications.
Patent Information
- Application Number
- CN202510036534.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-09
- Publication Date
- 2026-01-16
AI Technical Summary
Existing methods for detecting needle tips in ultrasound images are insufficient in terms of anti-interference ability, generalization ability, and accuracy when faced with dynamic interference, static noise, data scarcity, and inadequate utilization of prior information, making it difficult to meet clinical needs.
By extracting the pinpoint grayscale image from the ultrasound video stream and simultaneously acquiring the gaze positions of professional doctors to generate a visual attention map, the data is fused. A pinpoint detection model is then constructed using Kalman filtering and a deep convolutional neural network, and the loss function is optimized to achieve real-time pinpoint detection.
It improves the accuracy and stability of needle tip positioning, enhances anti-interference ability and generalization ability, and realizes automated real-time detection of the needle tip position, meeting the needs of clinical applications.
Smart Images
Figure CN121353151A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image localization technology, and in particular to a method, device and storage medium for detecting the tip of a puncture needle in an ultrasound image. Background Technology
[0002] Medical ultrasound imaging is an imaging technique that uses high-frequency sound waves to generate images of tissues and organs inside the body. It is widely used in many fields such as obstetrics, cardiology, urology, and interventional surgery, and plays an indispensable role, especially in procedures such as guided puncture, injection, and tissue biopsy.
[0003] In interventional procedures, such as ultrasound-guided puncture, injection, or biopsy, accurate needle tip positioning is crucial. Figure 1 The ultrasound images of thyroid biopsies shown rely on visual observation of the needle tip position, presenting doctors with challenges such as low image resolution, insufficient contrast, and difficulty in real-time dynamic monitoring. Traditional manual positioning methods have many shortcomings, especially in cases involving complex anatomical structures, changes in patient posture, or insufficient operator experience, which can easily lead to puncture failure, inaccurate positioning, or even complications.
[0004] Ultrasound image-based needle tip localization methods can be divided into two main categories: methods based on external navigation devices and image-based methods. The former uses external navigation devices such as optical or magnetic positioning to locate the needle tip in the image, while the latter achieves needle tip localization by analyzing and processing the ultrasound image itself.
[0005] Existing methods still have the following shortcomings, making it difficult to meet actual clinical needs:
[0006] (1) Clinical real human puncture data contains serious dynamic interference, and the existing methods have limited ability to resist dynamic interference and are insufficient in recognizing dynamic needle tips.
[0007] (2) Ultrasound images have a low signal-to-noise ratio and severe static interference due to the presence of a large amount of speckle noise. Existing methods lack specific optimization for this characteristic of ultrasound images and are insufficient in their ability to identify static needle tips.
[0008] (3) Due to reasons such as patient privacy protection, high cost of expert annotation and scarcity of disease data, there is a limited amount of real clinical puncture data that can be used to train deep neural networks, and the clinical generalization ability of existing methods is insufficient.
[0009] (4) Existing methods do not make full use of additional prior information, and there is still much room for improvement in their anti-interference capabilities.
[0010] Therefore, there is an urgent need for a method, device, and storage medium for detecting the tip of a puncture needle in ultrasound images, which can improve the accuracy and stability of needle tip positioning, enhance anti-interference ability and generalization ability, and thus better meet the needs of clinical applications. Summary of the Invention
[0011] The purpose of this invention is to provide a method, device, and storage medium for detecting the tip of a puncture needle in ultrasound images, aiming to solve the technical problems of poor anti-interference ability, generalization ability, and accuracy of traditional ultrasound image needle tip detection.
[0012] To achieve the above objectives, in a first aspect, the present invention provides a method for detecting the tip of a puncture needle in an ultrasound image, comprising:
[0013] S1. Sample Data Collection:
[0014] The grayscale image of the puncture needle tip is extracted from the ultrasound image video stream, and a visual attention map is generated by simultaneously collecting the gaze positions of the professional doctor's eyes.
[0015] S2, Data Preprocessing:
[0016] The pin-tip grayscale image and the visual attention map obtained in step S1 are fused and data augmentation processing is performed to initially obtain eye map dual-modal fusion data;
[0017] S3. Calculate the fusion weights for the eye map dual-modal fusion data based on Kalman filtering to obtain eye map fusion image data. As shown in the following formula:
[0018]
[0019] in:
[0020] The fused image at time t;
[0021] Kalman gain, used as fusion weights;
[0022] : Predicted pinpoint grayscale image;
[0023] : Current visual attention map;
[0024] S4. Construct a pinhead detection model based on a deep convolutional neural network, input the fused eye map image data into the pinhead detection model for model training and optimize the model's loss function to obtain an eye map dual-modal pinhead detection model.
[0025] S5. Real-time detection of the puncture needle tip is performed based on the eye diagram dual-modal needle tip detection model.
[0026] As a further improvement to the above scheme, in step S2, the data preprocessing steps include:
[0027] S21. First, scale the tip grayscale image of the ultrasound image to a preset size (preferably, the preset size is 640×640), and then use channel copying to expand the tip grayscale image into an RGB three-channel tip image.
[0028] S22. The RGB three-channel pinpoint image and the visual attention map of the eyeball are fused together;
[0029] S23. Then, random brightness perturbation data enhancement is performed on the fused image;
[0030] S23. Standardize the enhanced random brightness perturbation data; preferably, use the Z-score method for standardization.
[0031] As a further improvement to the above scheme, in step S23, when performing random brightness perturbation data enhancement on the fused image, the original pixel value of the fused image is multiplied by a random value in the range of [0.9, 1.1] to obtain random brightness perturbation enhancement data.
[0032] As a further improvement to the above scheme, the steps for obtaining eye map fusion image data include:
[0033] S31. Determine the state-space model:
[0034]
[0035] State variables The state variable represents the grayscale image information of the needle tip at time t in the current frame. The grayscale image information of the needle tip at time t-1 represents the current frame; the state transition matrix A is used to describe the linear evolution of the state;
[0036]
[0037] Observed variables This represents the visual attention map at time t in the current frame; the observation matrix H is used to map the state space to the observation space (identity matrix, assuming that the observations and states are directly related).
[0038] S32. Determine the prediction steps: at each ultrasound image video frame, predict the state based on the needle tip state of the puncture needle and visual attention observation data from the previous moment.
[0039] S33, Determine update steps
[0040] After acquiring new observation data (i.e., the visual attention map), the state estimate is updated based on the Kalman gain. This update step includes updating the Kalman gain, updating the state estimate, and updating the covariance matrix.
[0041] S34, Image Fusion
[0042] Repeat steps S32 and S33 to calculate the Kalman gain for each frame of the video, obtaining the pinpoint grayscale image and the image data after visual attention fusion for each frame, which is also the eye map fusion image data. .
[0043] As a further improvement to the above scheme, the prediction step includes:
[0044] S321. Prediction of the puncture needle tip state, based on the following formula:
[0045]
[0046] in: The predicted state estimate at time t (pinpoint grayscale image).
[0047] State estimation at time t-1;
[0048] : State transition matrix, describing the linear evolution of the state;
[0049] S322. Covariance prediction, based on the following formula:
[0050]
[0051] in: The prediction covariance matrix at time t represents the uncertainty of the state estimate;
[0052] The covariance of process noise represents the uncertainty of the model;
[0053] : is the covariance matrix of the previous time step, representing the uncertainty of the state estimate at the previous time step.
[0054] As a further improvement to the above scheme, the Kalman gain is updated according to the following formula:
[0055]
[0056] Kalman gain determines the fusion weights between predicted and observed values;
[0057] : Covariance of observation noise, representing the noise in the visual attention map;
[0058] : Observation matrix, which maps the state space to the observation space (identity matrix, assuming that observations and states are directly related);
[0059] As a further improvement to the above scheme, the state estimate is updated according to the following formula:
[0060]
[0061] in: The updated state estimate at time t is the fused pinpoint grayscale image at the current time.
[0062] : Observation at time t (i.e., the actual visual attention map obtained);
[0063] Predicted state estimation at time t;
[0064] As a further improvement to the above scheme, the covariance matrix is updated according to the following formula:
[0065]
[0066] in: The error covariance matrix is updated at time t, representing the uncertainty in the estimation of the system state.
[0067] Identity matrix, dimensions, and state vector same;
[0068] The predicted covariance matrix at time t, derived from the prediction step;
[0069] As a further improvement to the above scheme, in step S4, the needle tip detection model includes an input layer for receiving the eye map fusion image data, a feature extraction backbone network for extracting features from the eye map fusion image data, and a detection head for generating the final target detection result based on the image features extracted by the feature extraction backbone network.
[0070] The detection head includes three parallel dense prediction branches and a max pooling layer and three convolutional layers connected to each of the dense prediction branches.
[0071] The three parallel dense prediction branches are the target existence classification branch, the target category classification branch, and the bounding box regression branch. Each of the three parallel dense prediction branches generates the target probability, category probability, and bounding box coordinates respectively through one max pooling layer and three convolutional layers.
[0072] As a further improvement to the above scheme, in step S4, the loss function of the optimization model is as follows:
[0073]
[0074] in, For bounding box regression loss, For the existence classification loss of the target, Let L be the target category classification loss, IOU be the intersection-union ratio, Intersection(GT, b) be the area of the intersection between the ground truth bounding box and the predicted positive bounding box, and Union(GT, b) be the area of the intersection between the ground truth bounding box and the predicted positive bounding box. obj and L cls Both use the BCE loss function; y obj and y cls These represent the true label of target existence and the true label of target category corresponding to the detection results, respectively.
[0075] In a second aspect, the present invention also provides an apparatus comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to execute a method for detecting the tip of a puncture needle in an ultrasound image provided in the first aspect.
[0076] Thirdly, the present invention also provides a storage medium having a computer program stored thereon, wherein, when the program is running, it controls the device where the storage medium is located to execute a method for detecting the tip of a puncture needle in an ultrasound image provided in the first aspect.
[0077] Because the present invention adopts the above technical solutions, the beneficial effects of this application are as follows:
[0078] This invention provides a method for detecting the tip of a puncture needle in ultrasound images. First, a grayscale image of the puncture needle tip is extracted from the ultrasound video stream. Simultaneously, a visual attention map is generated by acquiring the gaze positions of a professional physician's eyes. The grayscale image of the needle tip and the visual attention map are then fused to form bimodal eye map fusion data. Next, fusion weights are calculated based on Kalman filtering to obtain fused eye map image data. Then, a needle tip detection model is constructed based on a deep convolutional neural network. The fused eye map image data is input into the needle tip detection model for training and the model's loss function is optimized to obtain an eye map bimodal needle tip detection model. Finally, real-time detection of the puncture needle tip is performed based on the eye map bimodal needle tip detection model. This setup, by integrating the eye attention map and the needle tip grayscale image, effectively solves the problem of dynamic interference in ultrasound images, thereby improving the accuracy and stability of puncture needle tip positioning. In this invention, not only is information from ultrasound images utilized, but also observation modal data (eye attention map) is combined, enhancing the anti-interference ability and generalization ability. In addition, this invention also uses prior data to construct a needle tip detection model based on a deep convolutional neural network, trains the needle tip detection model, and optimizes the model's loss function to obtain an optimized eye map bimodal needle tip detection model. Real-time puncture needle tip detection is performed based on the optimized eye map bimodal needle tip detection model, which not only improves the anti-interference ability of this detection method, but also enables automated real-time detection of the puncture needle tip position, thereby better meeting the needs of clinical applications. Attached Figure Description
[0079] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0080] Figure 1 This is a schematic diagram of an ultrasound image of a thyroid biopsy disclosed in this invention;
[0081] Figure 2 This is a schematic diagram illustrating the principle and flow of a method for detecting the tip of a puncture needle in an ultrasound image, as disclosed in this invention.
[0082] Figure 3 This is a schematic diagram of the eye diagram dual-modal pin tip detection model disclosed in this invention;
[0083] Figure 4 This is a schematic diagram comparing the detection results of a puncture needle tip detection method in ultrasound images disclosed in this invention with those of YOLOv8.
[0084] The realization of the objective, functional characteristics and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0086] It should be noted that all directional indicators (such as up, down, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.
[0087] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.
[0088] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0089] Example 1
[0090] See Figures 2-4 This invention provides a method for detecting the tip of a puncture needle in an ultrasound image, comprising:
[0091] S1. Sample Data Collection:
[0092] The grayscale image of the puncture needle tip is extracted from the ultrasound video stream, and a visual attention map is generated by simultaneously acquiring the gaze position of the professional doctor's eyes; these two images reflect the morphological characteristics of the needle tip and the gaze area of the human eye, respectively.
[0093] S2, Data Preprocessing:
[0094] The pin-tip grayscale image and the visual attention map obtained in step S1 are fused and data augmentation processing is performed to initially obtain eye map dual-modal fusion data;
[0095] S3. Calculate the fusion weights for the eye map dual-modal fusion data based on Kalman filtering to obtain eye map fusion image data. As shown in the following formula:
[0096]
[0097] in:
[0098] The fused image at time t;
[0099] Kalman gain, used as fusion weights;
[0100] : Predicted pinpoint grayscale image;
[0101] : Current visual attention map;
[0102] S4. Construct a pinpoint detection model based on a deep convolutional neural network. Input the fused eye map image data into the pinpoint detection model for model training and optimize the model's loss function to obtain a dual-modal eye map pinpoint detection model. Specifically, in this embodiment, see... Figure 2 and Figure 3 The input to the needle tip detection model is the fused eye map image data. The dimensions are 640×640×3 (height×width×number of channels).
[0103] S5. Real-time detection of the puncture needle tip based on the eye diagram dual-modal needle tip detection model;
[0104] This setup, by integrating the eye attention map and the needle tip grayscale image, effectively solves the problem of dynamic interference in ultrasound images, thereby improving the accuracy and stability of needle tip positioning. This invention not only utilizes information from ultrasound images but also combines observation modal data (eye attention map), enhancing its anti-interference and generalization capabilities. Furthermore, this invention utilizes prior data to construct a needle tip detection model based on a deep convolutional neural network, trains the model, and optimizes its loss function to obtain an optimized eye map bimodal needle tip detection model. Real-time needle tip detection based on this optimized model not only improves the anti-interference capability of the detection method but also enables automated real-time detection of the needle tip position, thus better meeting the needs of clinical applications.
[0105] In a preferred embodiment, step S2, the data preprocessing step includes:
[0106] S21. First, scale the tip grayscale image of the ultrasound image to a preset size (preferably, the preset size is 640×640), and then use channel copying to expand the tip grayscale image into an RGB three-channel tip image.
[0107] S22. The RGB three-channel pinpoint image and the visual attention map of the eyeball are fused together;
[0108] S23. Then, random brightness perturbation data enhancement is performed on the fused image. Specifically, when performing random brightness perturbation data enhancement on the fused image, the original pixel value of the fused image is multiplied by a random value in the range of [0.9, 1.1] to obtain random brightness perturbation enhancement data.
[0109] S23. Standardize the enhanced random brightness perturbation data; preferably, use the Z-score method for standardization.
[0110] In a preferred embodiment, in the ultrasound image puncture needle tip detection task, Kalman filtering is used to adjust the weights of the visual attention map and the data containing the needle tip image, thereby performing bimodal data fusion to improve the accuracy and robustness of the detection; specifically, the steps for obtaining the eye map fusion image data include:
[0111] S31. Determine the state-space model:
[0112]
[0113] State variables (state prediction of pinpoint grayscale image) The state variable represents the grayscale image information of the needle tip at time t in the current frame. The grayscale image information of the needle tip at time t-1 represents the current frame; the state transition matrix A is used to describe the linear evolution of the state;
[0114]
[0115] Observed variables (visual attention map) This represents the visual attention map at time t in the current frame; the observation matrix H is used to map the state space to the observation space (identity matrix, assuming that the observations and states are directly related).
[0116] S32. Determine the prediction steps: At each ultrasound image video frame, predict the state based on the needle tip state of the previous moment and visual attention observation data; assuming the state evolution in each frame is linear, the prediction steps include:
[0117] S321. Prediction of the puncture needle tip state, based on the following formula:
[0118]
[0119] in: The predicted state estimate at time t (pinpoint grayscale image).
[0120] State estimation at time t-1;
[0121] : State transition matrix, describing the linear evolution of the state;
[0122] S322. Covariance prediction, based on the following formula:
[0123]
[0124] in: The prediction covariance matrix at time t represents the uncertainty of the state estimate;
[0125] The covariance of process noise represents the uncertainty of the model;
[0126] : is the covariance matrix of the previous time step, representing the uncertainty of the state estimate at the previous time step;
[0127] S33, Determine update steps
[0128] After acquiring new observation data (i.e., the visual attention map), the state estimate is updated based on the Kalman gain. This update step includes updating the Kalman gain, updating the state estimate, and updating the covariance matrix.
[0129] Specifically, the Kalman gain is updated according to the following formula:
[0130]
[0131] Kalman gain determines the fusion weights between predicted and observed values;
[0132] : Covariance of observation noise, representing the noise in the visual attention map;
[0133] : Observation matrix, which maps the state space to the observation space (identity matrix, assuming that observations and states are directly related);
[0134] Update the state estimate according to the following formula:
[0135]
[0136] in: The updated state estimate at time t is the fused pinpoint grayscale image at the current time.
[0137] : Observation at time t (i.e., the actual visual attention map obtained);
[0138] Predicted state estimation at time t;
[0139] Update the covariance matrix according to the following formula:
[0140]
[0141] in: The error covariance matrix is updated at time t, representing the uncertainty in the estimation of the system state.
[0142] Identity matrix, dimensions, and state vector same;
[0143] The predicted covariance matrix at time t, derived from the prediction step;
[0144] S34, Image Fusion
[0145] Repeat steps S32 and S33 to calculate the Kalman gain for each frame of the video, obtaining the pinpoint grayscale image and the image data after visual attention fusion for each frame, which is also the eye map fusion image data. .
[0146] In a preferred embodiment, in step S4, the pinpoint detection model includes an input layer for receiving the eye map fusion image data, a feature extraction backbone network for extracting features from the eye map fusion image data, and a detection head for generating the final target detection result based on the image features extracted by the feature extraction backbone network.
[0147] Specifically, the structure of the feature extraction backbone network is as follows: Figure 3 As shown, it contains 5 stages. The first stage includes two convolutional layers, the second stage includes one max pooling layer and two convolutional layers, the third stage includes one max pooling layer and three convolutional layers, and the fourth and fifth stages each include one max pooling layer and three convolutional layers. The input and output of each stage are connected sequentially.
[0148] In this embodiment, the convolutional layer is used to extract local features of the image. The convolutional layer consists of three components: convolution, batch normalization (BN), and ReLU activation function. All five convolutional layers use a 3×3 kernel with a stride of 1 and padding of 1.
[0149] The convolution formula is shown below:
[0150]
[0151] in:
[0152] It is the value of the k-th channel at position (i, j) of the output feature map;
[0153] It is the pixel value of the input image at position (i+m, j+n);
[0154] It represents the weight of the convolution kernel at position (m, n) and the k-th channel;
[0155] It is a bias term;
[0156] To accelerate training and reduce internal covariate bias, the data for each mini-batch is normalized. The specific normalization formula is shown below:
[0157]
[0158]
[0159] in:
[0160] It is the input of the k-th sample. and These are the mean and variance of the small batch, respectively. It is a very small constant to prevent division by zero. and These are learnable scaling and translation parameters.
[0161] To enhance the expressive power of the model, a non-linear activation function, ReLU, is introduced and applied after the convolutional layer.
[0162] The pooling layer is responsible for downsampling to reduce the size of the feature map and reduce the amount of computation. The pooling layer is located after the convolutional layer. In this embodiment, the max pooling method is used with a 2×2 window size and a stride of 2.
[0163] The pooling layer formula is shown below:
[0164]
[0165] in:
[0166] O(i, j, k) is the value of the output feature map at position (i, j) and the kth channel.
[0167] I(i+m, j+n, k) is the value of the input feature map at position (i+m, j+n) and the kth channel.
[0168] K is the size of the pooling kernel;
[0169] The detection head includes classification and bounding box regression, specifically including three parallel dense prediction branches and a max pooling layer and three convolutional layers connected to each of the dense prediction branches.
[0170] The three parallel dense prediction branches are the target existence classification branch, the target category classification branch, and the bounding box regression branch. Each of the three parallel dense prediction branches passes through one max pooling layer and three convolutional layers to generate the target probability, category probability, and bounding box coordinates, respectively.
[0171] Specifically, the target existence classification branch is used to output the probability that a target exists at each position on the feature map, and the corresponding number of channels is 1;
[0172] The target category classification branch is used to output the probability that the target belongs to all predefined categories, which is used to determine the object category corresponding to each position on the feature map, and the number of output channels is the number of categories C;
[0173] The bounding box regression branch is used to output the coordinates of the bounding box containing the target (the center coordinates of the box plus the width and height), which is used to determine the specific location of the target; the number of output channels is 4;
[0174] It should be noted that after the detection head outputs the detection results, the detection results also need to be post-processed. Specifically, non-maximum suppression (NMS) is used for post-processing to filter out overlapping bounding boxes and retain the most likely detection results, thereby improving the accuracy of the sample data.
[0175] In a preferred embodiment, in step S4, the loss function of the optimization model is as follows:
[0176]
[0177] in, For bounding box regression loss, For the existence classification loss of the target, Let L be the target category classification loss, IOU be the intersection-union ratio, Intersection(GT, b) be the area of the intersection between the ground truth bounding box and the predicted positive bounding box, and Union(GT, b) be the area of the intersection between the ground truth bounding box and the predicted positive bounding box. obj and L cls Both use the BCE loss function; y obj and y cls Let y represent the true label of the target existence and the true label of the target category corresponding to the detection result, respectively. For the true category of a positive sample, we have y obj =1 and y cls =1, otherwise y obj =0 and y cls=0.
[0178] To verify the advancement of the detection method proposed in this invention, a performance evaluation was conducted on the ocular bimodal needle tip detection model obtained in this invention and the traditional YOLOv8 method, based on a dataset of 21,281 real clinical puncture images covering multiple sites including the thyroid, abdomen, breast, lymph nodes, and testis. The evaluation metric used was AP. 0.1:0.5 AP 0.2 AP 0.5 A comparison was made. As shown in Table 1, it can be seen that the method proposed in this invention has significant advantages.
[0179]
[0180] Combination Figure 4 As shown, the green box represents the detection result, the number in the box represents the detection probability, and the red box represents the true value. The method proposed in this invention has a lower false negative rate and false positive rate. Furthermore, the needle tip detection method provided by this invention can improve the accuracy and stability of needle tip positioning by integrating eye attention map and needle tip grayscale map.
[0181] Example 2:
[0182] The present invention also provides an apparatus comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to perform some or all of the steps in Embodiment 1;
[0183] A processor may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.
[0184] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.
[0185] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces processor waiting time, and thus improves system efficiency.
[0186] Example 3:
[0187] The present invention also provides a storage medium having a computer program stored thereon, wherein, when the program is executed, it controls the device where the storage medium is located to perform some or all of the steps in Embodiment 1.
[0188] The storage medium may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device. It is understood that the storage medium can be any machine-readable medium capable of storing program code, such as random access memory (RAM), magnetic disk, hard disk, solid-state disk (SSD), or non-volatile memory.
[0189] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or storage media. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0190] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. All equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.
Claims
1. An ultrasound image method for detecting a needle tip of a puncture needle, characterized by the steps of The method comprises the following steps: S1, sample data collection: extracting a needle tip grayscale image of the puncture needle from an ultrasound image video stream, and synchronously collecting a professional doctor's both-eye gaze position to generate a visual attention map; S2, data preprocessing: fusing the needle tip grayscale image and the visual attention map obtained in step S1, and performing data enhancement processing to preliminarily obtain eye map bimodal fusion data; S3, based on Kalman filtering, the eye diagram bimodal fusion data is fused weight calculation, obtain eye diagram fusion image data As shown in the following formula: ； wherein: : fusion image at time t; : Kalman gain as fusion weight; : predicted needle tip grayscale image; : current visual attention map; S4, constructing a needle tip detection model based on a deep convolutional neural network, the fused eye diagram fusion image data training the needle tip detection model and optimizing its loss function, to obtain an eye diagram bimodal needle tip detection model; S5, performing real-time detection of the puncture needle tip based on the eye map bimodal needle tip detection model.
2. The method of claim 1, wherein, In step S2, the step of data preprocessing comprises: S21, first scaling the needle tip grayscale image of the ultrasound image to a preset size, and then using a channel replication method to expand the needle tip grayscale image into an RGB three-channel needle tip image; S22, fusing the RGB three-channel needle tip image and the visual attention map of the eyeball; S23, then performing random brightness perturbation data enhancement on the fused image; S23, performing standardization processing on the enhanced random brightness perturbation data.
3. The method of claim 2, wherein, In step S23, when performing random brightness perturbation data enhancement on the fused image, a random value in the interval [0.9, 1.1] is multiplied by the original pixel value of the fused image to obtain random brightness perturbation enhancement data.
4. The method of claim 1-3, wherein, The step of obtaining eye map fusion image data comprises: S31, determining a state space model: ； state variable state variable state variable state transition matrix ； observed variables denotes the visual attention map at the current frame t; the observation matrix H is used to map the state space to the observation space; S32, determining a prediction step, at each ultrasound image video frame time, predicting the state according to the puncture needle tip state at the previous time and the visual attention observation data; S33, determining an update step After obtaining new observation data, updating the state estimation according to the Kalman gain, the update step comprises updating the Kalman gain, updating the state estimation and updating the covariance matrix: S34, image fusion Repeat step S32 and step S33, calculate the Kalman gain of each frame of the video, get the image data of each frame after the fusion of the stylus gray scale image and the visual attention, that is, obtain the eye diagram fusion image data .
5. The method of claim 4, wherein, The prediction step comprises: S321, puncture needle tip state prediction, which is predicted according to the following formula: ； wherein: is the predicted state estimate (needle gray scale image) at time t; is the state estimate at time t - 1; : state transition matrix, describing the linear evolution of the states; S322, covariance prediction, which is predicted according to the following formula: ； wherein: : prediction covariance matrix at time t, representing the uncertainty of the state estimate; : Covariance of process noise, representing uncertainty of the model; : Covariance matrix of the previous time step, representing the uncertainty of the state estimate of the previous time step.
6. The method of claim 4, wherein, The Kalman gain is updated according to the following formula: ； : Kalman gain, determines the fusion weight of the predicted value and the observed value; : Covariance of observation noise, representing noise in the visual attention map; : observation matrix, mapping state space to observation space (identity matrix, assuming observations are directly related to states); The state estimation is updated according to the following formula: ； wherein: : updated state estimation at time t, i.e. the fused needle grey scale image at the current time instant; : observed value at time t (i.e. the actual obtained visual attention map); : predicted state estimate at time t; The covariance matrix is updated according to the following formula: ； wherein: : update the error covariance matrix at time t, which represents the uncertainty of the system state estimation; : identity matrix, dimension same as state vector : same as : prediction covariance matrix at time t, from the prediction step.
7. The method of claim 1-3, wherein, In step S4, the needle tip detection model comprises an input layer for receiving the eye map fusion image data, a feature extraction backbone network for extracting features of the eye map fusion image data, and a detection head for generating a final target detection result based on the feature extraction backbone network; The detection head comprises three parallel dense prediction branches, and one maximum pooling layer and three convolution layers corresponding to each of the dense prediction branches; The three parallel dense prediction branches are target existence classification branch, target category classification branch and boundary box regression branch respectively, and each of the three parallel dense prediction branches passes through one maximum pooling layer and three convolution layers to correspondingly generate target probability, category probability and boundary box coordinates.
8. The method of claim 7, wherein, In step S4, the loss function of the model is as follows: ； wherein, is a bounding box regression loss, is a target existence classification loss, is a target class classification loss, IOU is an intersection over union, Intersection(GT, b) represents an area of intersection calculated from a real bounding box and a positive sample predicted box, and Union(GT, b) represents an area of intersection calculated from a real bounding box and a positive sample predicted box; L obj and L cls both use a BCE loss function; y obj and y cls respectively represent a target existence real label and a target class real label corresponding to a detection result.
9. An apparatus, comprising: The device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method for detecting a puncture needle tip in an ultrasound image according to any one of claims 1-8.
10. A storage medium, characterized by A storage medium storing a computer program, wherein the computer program controls a device in which the storage medium is located to perform the method of claim 1-8 when the computer program is running.