Handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance

Abstract

A handheld ultrasonic probe detection and attribute recognition method based on multi-scale attention and hand guidance, collects process images of an operator holding an ultrasonic probe to examine a patient, and pre-processes the process images as input images; a handheld target detection model detects the input images, outputs probe categories, boundary box coordinates and detection confidence; and an attribute recognition model recognizes attributes of the input image probes, and outputs direction attributes and position attributes of the probes. The hand detection module locks the hand region in the handheld target detection model, and the position attention module establishes a spatial correlation between the hand features and the probe image features, significantly improving the probe category recognition accuracy, the boundary box positioning accuracy and the detection confidence. The attribute recognition model captures the interaction between the probe image features and the target region features through a cross-attention mechanism, dynamically adjusts the channel weight of the fused features, enhances the key channel information, and accurately identifies the direction attributes and the position attributes of the probes.

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method for detection and attribute recognition of a handheld ultrasound probe based on multi-scale attention and hand guidance. Background Technology

[0002] Ultrasound examination, as a non-invasive, real-time, and portable medical imaging technology, has become a core tool for clinical diagnosis and emergency assessment due to its advantages of no radiation risk, repeatability, and dynamic observation. It plays an irreplaceable role in the rapid screening of acute and critical illnesses such as intracranial hypertension, pneumothorax, cardiac tamponade, and hemorrhagic shock. Bedside ultrasound can complete multi-site assessments in a short time, providing crucial information for emergency decisions and significantly shortening the treatment time window. In resource-constrained scenarios such as primary healthcare and field rescue, portable ultrasound devices have become essential tools for quickly assessing patient conditions, and their application value continues to increase with the development of smart healthcare.

[0003] However, the accuracy of ultrasound examinations is highly dependent on the operator's professional skills and experience. Strict operating procedures exist for ultrasound examinations of different diseases; details such as probe placement, angle, and pressure directly determine image quality. Operational deviations may lead to unclear or misdiagnosed key signs. Without standardized guidance, non-professionals (such as emergency personnel or the general public) often fail to perform the examination due to non-standard operation, thus failing to realize the true value of ultrasound equipment.

[0004] The existing technology system has significant shortcomings in supporting non-professionals to use ultrasound equipment independently: traditional operation guidelines are mostly presented in text or static diagrams, which cannot meet the dynamic adjustment needs in real-time operation, and the control of key parameters lacks quantitative standards; although mainstream auxiliary tools can identify the probe, they are difficult to analyze its spatial relationship with the human body, and operation deviations cannot be corrected in time; the development of dedicated systems is lagging behind, lacking both operation constraint models for specific diseases and multimodal analysis capabilities that associate hand movements, probe posture and anatomical structures, resulting in a significant reduction in the reliability of ultrasound applications in non-professional scenarios.

[0005] This technological bottleneck directly limits the widespread availability of ultrasound equipment. In primary healthcare and home settings, the high operational threshold prevents many portable ultrasound devices from being fully utilized, often delaying the initial screening of acute and critical illnesses. Therefore, developing intelligent technologies that can sense probe status in real time, quantify operational compliance, and provide dynamic guidance can not only fill the technological gap for non-professionals to use ultrasound equipment independently, but also promote the transformation of ultrasound examinations from specialized to public accessibility. This has significant practical implications for improving self-rescue and mutual-rescue capabilities in acute and critical illnesses and expanding the coverage of medical services.

[0006] Patent document CN118039130A discloses an intelligent ultrasound-assisted diagnostic method and system integrating ChatGPT. This method first activates the ultrasound software system and AI-assisted detection system, establishing a connection with ChatGPT. Then, the ultrasound software system automatically collects the patient's medical records, physical examination information, and doctor's operational steps, transmitting this data to the AI-assisted detection system. Subsequently, the AI-assisted detection system integrates multimodal information, adds limiting keywords, inputs it into ChatGPT for analysis, and outputs the patient's condition, diagnostic assistance results, and related data. Finally, based on the doctor's feedback on the chosen diagnostic plan, the diagnostic strategy is continuously fine-tuned to improve accuracy. However, this method has significant limitations. The essence of this technique is to provide diagnostic references for professional doctors with ultrasound operation skills. Its core lies solely in text information integration and diagnostic suggestion generation. It cannot perceive the correctness of the probe position and angle in real time, nor can it provide dynamic adjustment prompts. It completely fails to provide operational guidance for non-professionals, cannot help them complete standardized examinations, and the operational threshold problem remains unresolved. Furthermore, due to the lack of a dedicated visual perception module for medical scenarios, it has poor adaptability to complex clinical scenarios, and its overall auxiliary effect is limited to the level of diagnostic suggestions, making it difficult to truly lower the professional threshold of ultrasound examination.

[0007] Patent document CN119318504A discloses a control system and medium for a handheld intelligent ultrasound diagnostic device, relating to the field of medical imaging diagnostic technology. This method constructs an intelligent control model, taking the initial parameters of the ultrasound probe, the relative coordinates of the patient's position, and the patient's physiological characteristics as inputs, and outputting a combination of probe parameters that matches the optimal image quality, attempting to solve the problem of inconsistent diagnosis caused by manual adjustment. However, this technology has significant limitations. Its essence revolves around the intelligent optimization of device parameters, focusing on the indirect guarantee of diagnostic image quality. The core relies solely on model-controlled probe parameters, failing to perceive in real-time whether the probe's tilt angle and rotation amplitude are standardized, nor can it judge the rationality of the probe's contact with the human body. It completely lacks dynamic guidance for the operational behavior itself, failing to help non-professionals complete ultrasound examination procedures correctly, and the issue of operational standardization remains unresolved. Furthermore, due to the lack of a dedicated visual perception module for direct recognition of probe posture and human interaction scenarios, its adaptability to complex clinical operation scenarios, such as different lighting conditions and partial occlusion, is poor. The overall auxiliary effect is limited to the parameter control level, making it difficult to truly lower the practical threshold for ultrasound examinations and failing to meet the needs of non-professionals or novices for standardized operation. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a lightweight and highly accurate handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance.

[0009] The technical solution adopted to solve the above-mentioned technical problems is: a handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance, including the following steps:

[0010] Step 1. Capture images of the operator holding an ultrasound probe to examine patients with different diseases using a camera. The images should include the hand, the ultrasound probe, and the target area of ​​the human body. The captured images should be preprocessed by cleaning, Gaussian filtering for noise reduction, and size normalization, and used as input images.

[0011] Step 2. Use a handheld target detection model to detect the input image with a handheld ultrasound probe, and output the probe category, bounding box coordinates, and detection confidence score;

[0012] The handheld target detection model includes a hand detection module, a first feature extraction module, an attention-based feature fusion module, and a detection head module;

[0013] The hand detection module is used to detect the hand region in the input image and generate hand heatmap features, which are then output to the attention-based feature fusion module.

[0014] The first feature extraction module is used to extract features related to the handheld ultrasound probe from the input image and generate probe image features to be output to the attention-based feature fusion module;

[0015] The attention-based feature fusion module is used to adjust and fuse the channel dimensions of the input hand heatmap features and probe image features to be consistent, thereby enhancing the expressive power of the probe image features and outputting the fused features to the detection head module.

[0016] The detection head module is used to perform target detection on the fused features and output the probe category, bounding box coordinates and detection confidence.

[0017] Step 3. Use the attribute recognition model to identify the probe attributes of the input image and output the probe's orientation and position attributes;

[0018] The attribute recognition model includes a target area image acquisition module, a second feature extraction module, a cross attention module, a channel attention module, and an attribute output module;

[0019] The target region image acquisition module is used to detect the target region from the input image and crop out the target region image.

[0020] The second feature extraction module is used to extract the features of the target area image and the features of the probe area image cropped from the bounding box output by the handheld target detection model, respectively, to generate the target area image features and the probe image features;

[0021] The cross-attention module fuses probe image features and target area image features through a cross-attention mechanism, captures the interaction between the two, and outputs fused features.

[0022] The channel attention module dynamically adjusts the channel weights of the fused features, enhances key channel information, and outputs the enhanced features.

[0023] The attribute output module is used to output the orientation and position attributes of the probe based on the enhanced features.

[0024] As a preferred technical solution, the handheld target detection model is as follows: a YOLOv5-Hand pre-trained model is added in parallel to the input layer of the YOLOv5 base model, and a position attention module is inserted between the feature fusion layer of the neck network and the detection head; the YOLOv5-Hand pre-trained model serves as the hand detection module, and the position attention module is used to establish the spatial correlation between hand image features and probe image features, adaptively weight probe image features according to the hand position, suppress background interference, and enhance the feature representation of the probe area near the hand.

[0025] As a preferred technical solution, the position attention module includes a first 1×1 convolutional layer, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a spatial correlation generation unit, and a Softmax normalization unit. The first 1×1 convolutional layer is used to adjust the hand heatmap features output by the hand detection module to the same channel dimension as the probe image features, thereby obtaining standardized hand image features. The second 1×1 convolutional layer is used to adapt the probe image feature dimensions output by the first feature extraction module, thereby obtaining standardized probe image features. The spatial correlation generation unit calculates the spatial correlation weight mapping between the standardized hand image features and the standardized probe image features through matrix multiplication. in This is the query matrix for feature mapping of hand images. Let T be the key matrix for feature mapping of the probe image, and T be the matrix transpose operation;

[0026] The Softmax normalization unit maps spatial correlation weights. Normalization is performed to obtain the attention weight A in the 0-1 interval, thereby achieving adaptive weighting of probe image features;

[0027] The third 1×1 convolutional layer is used to perform residual fusion of the weighted probe image features and the original probe image features, and output the enhanced probe fusion features, which strengthens the feature expression of the probe area near the hand and suppresses background interference.

[0028] As a preferred technical solution, the target area image acquisition module of the attribute recognition model is composed of a pre-trained YOLOv8 base model or a YOLOv8-face model.

[0029] As a preferred technical solution, the second feature extraction module of the attribute recognition model is composed of a ResNet50 backbone network.

[0030] As a preferred technical solution, the specific operation of the cross-attention module of the attribute recognition model to output fused features is as follows:

[0031] The probe image features are mapped to a query matrix via 1×1 convolution. The features of the inspected target area image are mapped to a key matrix. The sum matrix V is used to calculate the interaction weights between the probe and target features. The probe image features containing human body correlation information are obtained by fusion. , These are features of the probe image.

[0032] The beneficial effects of this invention are as follows:

[0033] This invention locks the hand region by setting an independent hand detection module in the handheld target detection model, and integrates a position attention module to establish a spatial correlation between hand features and probe image features. Based on the hand position, it adaptively weights probe image features and suppresses background interference, thereby enhancing the feature expression of the probe region near the hand. This effectively solves the problem of high probe detection difficulty caused by factors such as changes in lighting and local occlusion in clinical scenarios. Compared with traditional detection models, it significantly improves the accuracy of probe category recognition, bounding box positioning accuracy, and detection confidence, and achieves accurate detection of ultrasound probes in complex scenarios.

[0034] This invention achieves precise analysis of the spatial association between the probe and the target area of ​​the human body. The attribute recognition model integrates the cross-attention module and the channel attention module. First, the cross-attention mechanism captures the interaction between the probe image features and the target area features. Then, the channel attention module dynamically adjusts the channel weights of the fused features and enhances key channel information. This enables precise identification of the probe's directional and positional attributes, and completes the quantitative analysis of the probe's operating status. It solves the technical shortcoming of existing technologies that can only identify the probe entity but cannot analyze its spatial association with the human body, providing an objective and quantitative core basis for judging the standardization of ultrasound operations.

[0035] This invention outputs the detection results, direction, and position attributes of the ultrasound probe in real time, providing real-time dynamic guidance for the operator's ultrasound operation. It promptly detects operational deviations such as probe placement and angle, and provides a reference for adjustment. This overcomes the shortcomings of traditional text and static graphic operation guides, which cannot adapt to the needs of real-time dynamic adjustment. It enables non-professionals such as emergency personnel, primary healthcare workers, and the general public to complete standardized ultrasound examinations, solving the problem of ultrasound examination failure in non-professional scenarios.

[0036] The network structure of this invention is lightweight and can be adapted to the hardware computing power of portable ultrasound devices. It can be quickly deployed in resource-constrained scenarios such as bedside and field rescue, meeting the real-time requirements for rapid screening of acute and critical illnesses. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance of the present invention.

[0038] Figure 2 This is a schematic diagram of the handheld target detection model of the present invention.

[0039] Figure 3 This is a schematic diagram of the attribute recognition model of the present invention. Detailed Implementation

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0041] exist Figure 1 The present embodiment of this method for detection and attribute recognition of a handheld ultrasound probe based on multi-scale attention and hand guidance includes the following steps:

[0042] Step 1. Capture images of the operator holding an ultrasound probe to examine patients with different diseases using a camera. The images should include the hand, the ultrasound probe, and the target area of ​​the human body. The captured images should be preprocessed by cleaning, Gaussian filtering for noise reduction, and size normalization, and used as input images.

[0043] Step 2. Use a handheld target detection model to detect the input image with a handheld ultrasound probe, and output the probe type, bounding box coordinates, and detection confidence score.

[0044] exist Figure 2In this model, the handheld target detection model is: a YOLOv5-Hand pre-trained model is added in parallel to the input layer of the YOLOv5 base model, and a position attention module is inserted between the feature fusion layer of the neck network and the detection head. The YOLOv5-Hand pre-trained model serves as the hand detection module, detecting the hand region in the input image and generating hand heatmap features at scales of 80×80×3 and 40×40×3. The YOLOv5 backbone network, as the first feature extraction module, extracts features related to the handheld ultrasound probe from the input image, generating probe image features at scales of 80×80 and 40×40. The feature fusion layer of the YOLOv5 neck network and the position attention module constitute an attention-based feature fusion module, used to adjust the channel dimensions of the input hand heatmap features and probe image features to be consistent and then fuse them. During the fusion process, a spatial relationship between hand image features and probe image features is established. The probe image features are adaptively weighted according to the hand position to suppress background interference and enhance the expressive power of the probe image features, resulting in fused features. The fused features are input to the detection head module composed of YOLOv5 detection heads, which performs target detection on the fused features. The classification branch outputs the probe category and confidence score, and the regression branch outputs the bounding box coordinates.

[0045] The position attention module includes a first 1×1 convolutional layer, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a spatial correlation generation unit, and a Softmax normalization unit. The first 1×1 convolutional layer adjusts the hand heatmap features output by the hand detection module to match the channel dimensions of the probe image features, obtaining standardized hand image features. The second 1×1 convolutional layer adapts the probe image feature dimensions output by the first feature extraction module to obtain standardized probe image features. The spatial correlation generation unit calculates the spatial correlation weight mapping between the standardized hand image features and the standardized probe image features through matrix multiplication. in This is the query matrix for feature mapping of hand images. The key matrix is ​​the feature mapping of the probe image, and T is the matrix transpose; the Softmax normalization unit maps spatial correlation weights. Normalization is performed to obtain the attention weight A in the range of 0 to 1, which realizes adaptive weighting of probe image features; the third 1×1 convolutional layer is used to perform residual fusion of the weighted probe image features and the original probe image features, and output the enhanced probe fusion features, which strengthens the feature expression of the probe area near the hand and suppresses background interference.

[0046] Step 3. Identify probe attributes using the attribute recognition model, and output the probe's orientation and position attributes;

[0047] The attribute recognition model includes a target region image acquisition module, a second feature extraction module, a cross-attention module, a channel attention module, and an attribute output module, such as... Figure 3 .

[0048] The target region image acquisition module consists of a pre-trained YOLOv8 base model or a YOLOv8-face model, used to detect the target region from the input image and crop out the target region image. The pre-trained YOLOv8 base model is suitable for acquiring target region images in non-facial human body areas, while the YOLOv8-face model is suitable for acquiring target region images in facial areas.

[0049] The second feature extraction module is composed of a ResNet50 backbone network. The target branch of the second feature extraction module extracts the features of the target area image under test, and the probe branch extracts the features of the probe area image cropped based on the bounding box output by the handheld target detection model, thereby generating the features of the target area image and the probe image.

[0050] The cross-attention module fuses probe image features and target area image features through a cross-attention mechanism, captures the interaction between the two, and outputs fused features. Specifically, the operation is as follows:

[0051] The probe image features are mapped to a query matrix via 1×1 convolution. The features of the inspected target area image are mapped to a key matrix K and a value matrix V. The interaction weights between the probe and the target area features are calculated. Where T is the matrix transpose operation, the fusion yields probe image features containing human body correlation information. , These are features of the probe image.

[0052] The channel attention module is the SEblock module, which dynamically adjusts the channel weights of the fused features, enhances key channel information, and outputs enhanced features.

[0053] The attribute output module consists of two 1×1 convolutional layers, a fully connected layer, and a sigmoid activation function. The enhanced features undergo dimensionality reduction through two 1×1 convolutional layers. The processed features are then input into the fully connected layer, and the sigmoid activation function is used to output the classification probabilities of the orientation and position attributes.

[0054] experiment

[0055] To verify the beneficial effects of the present invention, the inventors conducted performance tests on the same generalization test set using the method of the present invention and the comparison algorithm YOLOv5s. The specific results are as follows:

[0056]

[0057] mAP (0.5 to 0.95) is the average accuracy of the cross-union ratio in the range of 0.5 to 0.95.

[0058] In the probe detection task, the mAP (0.5 to 0.95) of this invention is improved by 0.01 compared with the basic model, proving that the hand guidance and position attention mechanism can effectively enhance the probe feature expression and improve the positioning accuracy in complex scenarios. In the attribute recognition task, the average accuracy is improved by 0.009, reflecting that the cross attention and channel attention fusion strategy can more accurately capture the relative relationship between the probe and the human body, providing a reliable basis for judging the operation specifications.

Claims

1. A method for detection and attribute recognition of a handheld ultrasound probe based on multi-scale attention and hand guidance, characterized in that, Includes the following steps: Step 1. Capture images of the operator holding an ultrasound probe to examine patients with different diseases using a camera. The images should include the hand, the ultrasound probe, and the target area of ​​the human body. The captured images should be preprocessed by cleaning, Gaussian filtering for noise reduction, and size normalization, and used as input images. Step 2. Use a handheld target detection model to detect the input image with a handheld ultrasound probe, and output the probe category, bounding box coordinates, and detection confidence score; The handheld target detection model includes a hand detection module, a first feature extraction module, an attention-based feature fusion module, and a detection head module; The hand detection module is used to detect the hand region in the input image and generate hand heatmap features, which are then output to the attention-based feature fusion module. The first feature extraction module is used to extract features related to the handheld ultrasound probe from the input image and generate probe image features to be output to the attention-based feature fusion module; The attention-based feature fusion module is used to adjust and fuse the channel dimensions of the input hand heatmap features and probe image features to be consistent, thereby enhancing the expressive power of the probe image features and outputting the fused features to the detection head module. The detection head module is used to perform target detection on the fused features and output the probe category, bounding box coordinates and detection confidence. Step 3. Use the attribute recognition model to identify the probe attributes of the input image and output the probe's orientation and position attributes; The attribute recognition model includes a target area image acquisition module, a second feature extraction module, a cross attention module, a channel attention module, and an attribute output module; The target region image acquisition module is used to detect the target region from the input image and crop out the target region image. The second feature extraction module is used to extract the features of the target area image and the features of the probe area image cropped from the bounding box output by the handheld target detection model, respectively, to generate the target area image features and the probe image features; The cross-attention module fuses probe image features and target area image features through a cross-attention mechanism, captures the interaction between the two, and outputs fused features. The channel attention module dynamically adjusts the channel weights of the fused features, enhances key channel information, and outputs the enhanced features. The attribute output module is used to output the orientation and position attributes of the probe based on the enhanced features.

2. The handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance according to claim 1, characterized in that, The handheld target detection model is as follows: a YOLOv5-Hand pre-trained model is added in parallel to the input layer of the YOLOv5 base model, and a position attention module is inserted between the feature fusion layer of the neck network and the detection head; the YOLOv5-Hand pre-trained model serves as the hand detection module, and the position attention module is used to establish the spatial correlation between hand image features and probe image features, adaptively weight probe image features according to the hand position, suppress background interference, and enhance the feature representation of the probe area near the hand.

3. The handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance according to claim 2, characterized in that, The position attention module includes a first 1×1 convolutional layer, a second 1×1 convolutional layer, a third 1×1 convolutional layer, a spatial correlation generation unit, and a Softmax normalization unit. The first 1×1 convolutional layer adjusts the hand heatmap features output by the hand detection module to match the channel dimension of the probe image features, obtaining standardized hand image features. The second 1×1 convolutional layer adapts the probe image feature dimension output by the first feature extraction module to obtain standardized probe image features. The spatial correlation generation unit calculates the spatial correlation weight mapping between the standardized hand image features and the standardized probe image features through matrix multiplication. in This is the query matrix for feature mapping of hand images. Let T be the key matrix for feature mapping of the probe image, and T be the matrix transpose operation; The Softmax normalization unit maps spatial correlation weights. Normalization is performed to obtain the attention weight A in the interval of 0 to 1, thereby achieving adaptive weighting of probe image features; The third 1×1 convolutional layer is used to perform residual fusion of the weighted probe image features and the original probe image features, and output the enhanced probe fusion features, which strengthens the feature expression of the probe area near the hand and suppresses background interference.

4. The handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance according to claim 1, characterized in that, The target area image acquisition module of the attribute recognition model consists of a pre-trained YOLOv8 base model or a YOLOv8-face model.

5. The handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance according to claim 1, characterized in that, The second feature extraction module of the attribute recognition model is composed of a ResNet50 backbone network.

6. The handheld ultrasound probe detection and attribute recognition method based on multi-scale attention and hand guidance according to claim 1, characterized in that, The specific operation of the cross-attention module of the attribute recognition model to output fused features is as follows: The probe image features are mapped to a query matrix via 1×1 convolution. The features of the inspected target area image are mapped to a key matrix. The sum matrix V is used to calculate the interaction weights between the probe and target features. The probe image features containing human body correlation information are obtained by fusion. , These are features of the probe image.

Images