Breast ultrasound video HER2 expression state recognition method based on space-time feature interaction

Abstract

This invention belongs to the technical field of breast cancer molecular subtyping, and more specifically, relates to a method for identifying HER2 expression status in breast ultrasound videos based on spatiotemporal feature interaction. The method includes: using raw breast ultrasound videos as input data and performing multi-level data preprocessing operations; then performing data balancing on the training data to obtain a standardized training set; sequentially constructing a feature module, a multi-stage feature fusion module, and a classification module based on the UniFormerV2 network, and training the resulting model; and inputting the test set into the model to obtain the final prediction result of HER2 expression status. This invention solves the problems of existing HER2 detection methods being invasive and unable to dynamically monitor, and existing deep learning methods based on static images ignoring the spatiotemporal dynamic information in ultrasound videos, leading to insufficient accuracy in predicting HER2 expression status, especially the HER2-Low subtype.

Description

Technical Field

[0001] This invention belongs to the technical field of molecular subtyping of breast cancer, and more specifically, relates to a method for identifying the HER2 expression status of breast ultrasound video based on spatiotemporal feature interaction. Background Technology

[0002] This invention relates to the field of molecular subtyping technology for breast cancer, and in particular to a method for predicting the expression status of human epidermal growth factor receptor 2 (HER2) using breast ultrasound video data. This method achieves non-invasive and accurate prediction of HER2 expression status by analyzing the spatiotemporal dynamic characteristics of ultrasound video, providing auxiliary diagnostic evidence for individualized treatment of breast cancer patients.

[0003] A deep learning method based on static ultrasound images: Elias, SG, et al. "Imaging features of HER2 overexpression in breast cancer: a systematic review and meta-analysis." Cancer epidemiology, biomarkers & prevention 23.8 (2014): 1464-1483. This method selects a single frame or a few representative static images from dynamic videos acquired by breast ultrasound examinations, extracts image features using a convolutional neural network, and finally outputs the predicted HER2 status (binary classification of HER2 positive or negative) by a fully connected layer classifier. Xiong et al. proposed the ConvoHER2 transfer learning model based on the Inception V3 framework. On a dataset containing 4873 pathological images, the accuracy rate for H&E staining images reached 85%, and the accuracy rate for IHC staining images reached 88%.

[0004] In summary, such methods are still limited to static image analysis. First, they lose the rich temporal information contained in ultrasound examinations, simplifying ultrasound videos into static image processing. In clinical practice, ultrasound physicians comprehensively judge the nature of lesions by observing the morphological changes of lesions in continuous frames, their relative movement with surrounding tissues, and the dynamic evolution of blood flow signals. Simplifying videos into static images completely ignores crucial dynamic information. Secondly, it's difficult to capture multi-scale dynamic features related to HER2 expression status. Key information requires models to capture subtle changes within the local spatiotemporal neighborhood while also integrating long-range spatiotemporal dependencies to understand complete dynamic patterns. Existing methods are limited by local time windows, making it difficult to simultaneously address the modeling needs of local details and global dynamics. Furthermore, there's a lack of effective handling of inherent image quality issues in ultrasound videos. Ultrasound video acquisition involves inter-frame jitter, speckle noise, and insufficient contrast, all of which severely affect the model's perception of lesion dynamic changes. Existing static image analysis methods fail to effectively address these issues. Finally, it's limited to binary classification tasks, making it difficult to meet clinical needs. With the clinical application of novel antibody-drug conjugates, accurate identification of HER2-Low subtypes has become an urgent clinical requirement. Existing methods struggle to distinguish the more ambiguous and heterogeneous HER2-Low subtypes. Summary of the Invention

[0005] The present invention aims to overcome at least one of the defects of the prior art and provide a method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction. This method addresses the problem of failing to utilize the temporal dynamic information in ultrasound video to simplify ultrasound video into static image processing, ignoring key temporal information such as lesion morphology changes and blood flow dynamics contained in the video sequence, and being limited to binary classification tasks, which makes it difficult to meet the clinical needs for accurate identification of HER2-Low subtypes.

[0006] The detailed technical solution of this invention is as follows:

[0007] A method for identifying HER2 expression status in breast ultrasound videos based on spatiotemporal feature interaction, the method comprising:

[0008] S1. Using the original breast ultrasound video as input data, firstly, the original input data is subjected to multi-level data preprocessing operations to obtain preprocessed video images, then divided into training set and test set, and data balancing processing is performed on the training set to obtain a standardized training set.

[0009] S2. Construct a feature module, a multi-stage feature fusion module, and a classification module based on the UniFormerV2 network in sequence; the feature module based on the UniFormerV2 network extracts features, the multi-stage feature fusion module fuses the extracted features, and the classification module obtains the probability distribution of each category based on the fused features.

[0010] S3. Construct the standardized training set data into tensor form and input it into the constructed model to obtain the trained model;

[0011] S4. Input the test set into the trained model to obtain the final prediction result of HER2 expression status.

[0012] Furthermore, the multi-level data preprocessing includes: inter-frame registration processing, noise reduction processing, and contrast enhancement processing;

[0013] Inter-frame registration processing: Using the original breast ultrasound video as input, inter-frame registration is performed using the optical flow method. The regularization parameter α is set to 0.1. The first frame is used as the reference frame, and all frame images are registered with the first frame in sequence. By calculating the pixel-level motion vectors of adjacent frames and correcting them in reverse, the spatial alignment between the reference frame and subsequent frames is achieved, and the registered image L is obtained.

[0014] Denoising: Using the registered image as input, a nonlocal mean algorithm is employed with a search window size of 11×11 and a similarity window size of 5×5. The filtering parameter h is set to denoise each frame of the registered image, resulting in the denoised image. ;

[0015] Contrast enhancement: using the denoised image As input, a contrast-limited adaptive histogram equalization is used to divide the image into 8×8 blocks, set a contrast limit value, and then perform grayscale normalization to linearly map all pixel values ​​to the [0,1] interval to obtain the enhanced image.

[0016] Furthermore, the data balancing process specifically includes: the training set is undersampled using stratified random sampling, and the HER2-Positive and HER2-Low categories are undersampled to the same number of training samples as the HER2-Zero category.

[0017] Furthermore, the preprocessed video images are constructed into tensor form, with T frames and an image size of H*W. and Let C represent the image height and width, respectively, with C channels, and the final tensor dimension is (1,240,224,224,3).

[0018] Furthermore, the construction of the feature module based on the UniFormerV2 network specifically includes:

[0019] Convolutional mapping layer: Performs preliminary spatial feature extraction on the input tensor. Each input frame first passes through a convolutional mapping layer to obtain an initial feature representation, denoted as . It is then divided into non-overlapping blocks and processed through N stages, each stage including a Local UniBlock module and a Global UniBlock module. l The stage index is the first one. l layer;

[0020] Local UniBlock module: Introduces a local multi-head relative self-attention mechanism into shallow networks, which limits attention computation to a local spatiotemporal neighborhood and is used to capture fine-grained dynamic features.

[0021] For those from the Layer input features Processing is performed to obtain the first... l Feature map of layer :

[0022] (1)

[0023] In equation (1), For the first l The first-level feature map output from layer 1 is used as the input feature of the current layer; As a local multi-head relative self-attention module, attention computation is limited to the local spatiotemporal neighborhood, which is used to capture subtle local dynamic changes in the feature map; The global multi-head relative self-attention module establishes spatial context dependencies through a global attention mechanism, enabling each location to perceive global spatial information; FFN is a feedforward network consisting of two fully connected layers, used to perform nonlinear transformations and dimensional adjustments on the fused spatiotemporal features; For the extracted first l Level 1 feature map; "For residual connection" The input features are added to the features transformed by the module to promote gradient flow and preserve the original information.

[0024] The Global UniBlock module introduces learnable positional encoding and context-aware attention mechanisms into deep networks to establish global spatiotemporal dependencies. This is used to model long-range dynamic patterns such as blood perfusion time-series features. l Level 1 Feature Map Processing is performed to obtain the first... l Second-level feature map of the layer ;

[0025] (2)

[0026] In Equation (2), DPE is a learnable location coding module used to inject spatiotemporal location information into the feature map, enabling the model to perceive the relationship between different frames and spatial locations. For context-aware multi-head relative self-attention modules, spatiotemporal dependencies are modeled globally, and long-range contextual information is captured by dynamically fusing content similarity and relative position bias. The output after processing by the global dependency modeling module l Second-level feature map; For residual connection The input features are added to the features transformed by the module to promote gradient flow and preserve the original information.

[0027] Furthermore, the construction of the multi-stage feature fusion module specifically includes: weighted fusion of all secondary feature maps to finally generate a unified fused feature representation;

[0028] Global features of all stages The input is aggregated into a multi-stage fusion module, where... To represent the total number of stages: first, through deconvolution operation... The feature maps at different depths are resolution aligned, and then a set of learnable stage weights are used. The aligned features are weighted and summed to generate a unified fused feature representation. :

[0029] (3)

[0030] Finally, the fusion features are obtained. Then, input it into the classification module.

[0031] Furthermore, the classification construction module specifically includes: compressing the spatiotemporal dimension of the fused features to extract a fixed-length feature vector, and performing linear mapping and normalization processing to finally predict the probability distribution of each category;

[0032] First, a global space-time pooling operation (GSTP) is used to... The vector is compressed into a fixed-length feature vector; this vector is then linearly transformed by a classification head, which consists of a weight matrix M, and M is a learnable parameter matrix; the output dimension is the number of categories, i.e., 3 categories, corresponding to HER2-Zero, HER2-Low, and HER2-Positive.

[0033] Finally, after normalization using the Softmax function, the probability distributions for each category are obtained. That is, the final prediction result;

[0034] (4)

[0035] In another aspect of the invention, an electronic device is also provided, comprising:

[0036] At least one processor; and

[0037] The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the breast ultrasound video HER2 expression state recognition method based on spatiotemporal feature interaction as described above.

[0038] In another aspect of the invention, a computer-readable storage medium is also provided, which stores executable instructions that, when executed, cause the machine to perform the breast ultrasound video HER2 expression status recognition method based on spatiotemporal feature interaction as described above.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] (1) The HER2 expression status identification method based on spatiotemporal feature interaction in breast ultrasound video provided by this invention is the first to use breast ultrasound video as the analysis object. By mining the spatiotemporal dynamic information contained in the video sequence, it achieves non-invasive and accurate prediction of HER2 expression status. At the same time, it is the first to combine ultrasound video analysis with the HER2 three-class classification task, and performs refined prediction for the three classes: HER2-Zero, HER2-Low, and HER2-Positive. In order to address the class imbalance problem in the three-class classification task, this invention designs a hierarchical random undersampling strategy in the data preprocessing stage and introduces a class weight mechanism in the loss function, which effectively alleviates the problem of insufficient minority class samples and significantly improves the identification ability of HER2-Low subtype.

[0041] (2) The method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction provided by this invention constructs a local-global progressive spatiotemporal feature interaction and fusion mechanism: In the shallow network, Local UniBlock limits attention calculation to the local spatiotemporal neighborhood, accurately capturing fine-grained dynamic features such as minute inter-frame deformations of lesions; In the deep network, Global UniBlock establishes global spatiotemporal dependencies through learnable positional encoding and context-aware attention, effectively modeling long-range dynamic patterns such as blood perfusion time-series features; Finally, through a multi-stage feature fusion module, learnable weights are used to adaptively integrate spatiotemporal features at different levels, achieving collaborative modeling of local details and global semantics. This mechanism breaks through the limitations of existing methods in spatiotemporal multi-scale feature modeling.

[0042] (3) The method for identifying the expression state of breast ultrasound video based on spatiotemporal feature interaction provided by the present invention designs a multi-level preprocessing process for the characteristics of breast ultrasound video, including: data balancing processing: combining hierarchical random undersampling and weighted cross-entropy loss function to alleviate the class imbalance problem in the HER2 three-class classification task and improve the recognition accuracy of minority class (HER2-Zero) and boundary class (HER2-Low); inter-frame registration: using optical flow to eliminate inter-frame jitter; non-local mean denoising: suppressing speckle noise while retaining edge information; CLAHE contrast enhancement: adaptively improving local contrast; in summary, the four work together to form a complete data purification link, providing high-quality input for subsequent spatiotemporal feature extraction. Attached Figure Description

[0043] Figure 1 This is a flowchart of the framework for predicting HER2 expression status in breast ultrasound video based on spatiotemporal interactive feature fusion, as proposed in this invention.

[0044] Figure 2 This is a schematic diagram of the UniFormerV2 model architecture in Embodiment 1 of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention is described.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should also be understood that when the terms "comprising" and / or "including" are used in this specification, they specify the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0048] In the case of no conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0049] Embodiment 1

[0050] Refer Figure 1 , this embodiment provides a method for identifying the HER2 expression status of breast ultrasound videos based on spatio-temporal feature interaction, and the method includes:

[0051] S1. Taking the original breast ultrasound video as input data, first performing multi-level data preprocessing operations on the input data to obtain preprocessed video images. Subsequently, dividing the preprocessed video images into a training set and a test set, and performing data balancing processing on the training set to obtain a standardized training set.

[0052] Furthermore, the multi-level data preprocessing includes: inter-frame registration processing, denoising processing, and contrast enhancement processing;

[0053] Inter-frame registration processing: Taking the original breast ultrasound video as input, using the optical flow method for inter-frame registration to eliminate non-pathological displacements caused by breathing and probe movement. The regularization parameter α is taken as 0.1, and all 240 frame images are sequentially registered with the first frame: By calculating the pixel-level motion vectors of adjacent frames and performing inverse correction, the spatial alignment of the reference frame, that is, the first frame, and the subsequent frames is achieved, and the registered image L is obtained.

[0054] Preferably, the regularization parameter α is crucial here. It controls the trade-off between data matching degree and motion field smoothness, achieving inter-frame alignment and theoretically providing a stable data basis for subsequent analysis. However, the calculation accuracy of the registration process itself, the ability to handle non-rigid deformations, and the possible geometric distortions introduced are potential problems that need to be continuously concerned. The improvement of image availability and the gain of feature extraction accuracy ultimately need to be strictly verified by the performance of downstream tasks. The core of the optical flow method lies in solving an optimization problem, that is, its objective function.

[0055] (5)

[0056] In formula (5), The objective function for optical flow optimization consists of two terms: the first term measures the degree of matching of pixel intensity between two frames of images after deformation based on the optical flow field; the second term constrains the smoothness of the optical flow field itself to avoid unreasonable and drastic motion changes. and The original breast ultrasound video single frame image is in and Gradient of direction; The time gradient refers to the brightness change between consecutive frames in the original breast ultrasound video. and for and The directional optical flow component is used to characterize the pixel motion vector between adjacent frames in the original breast ultrasound video; and represent and The spatial gradient reflects the smoothness of the optical flow field; This is a regularization parameter used to balance the data fidelity and smoothing terms of optical flow; Represents the image domain; and Let L be an integral infinitesimal element. After obtaining the optical flow field by solving the above objective function, the original image frame is inversely corrected to output the registered image L.

[0057] Denoising: Using the registered image as input, a nonlocal mean algorithm is employed with a search window size of 11×11, a similarity window size of 5×5, and a filtering parameter h of 10. Denoising is then applied to each frame of the registered image to obtain the denoised image. .

[0058] Preferably, noise reduction is a necessary step in ultrasound video preprocessing. The speckle noise and random noise in the original image can obscure the fine structure of tissues, especially blurring the boundaries of lesion areas. Therefore, an algorithm is needed that can smooth noise while preserving edge information effectively.

[0059] The nonlocal mean algorithm searches for blocks similar to its local neighborhood window across the entire image for a target pixel. Each found similar block is assigned a weight based on its similarity to the target block; the higher the similarity, the greater the weight. The denoised grayscale value of the target pixel is obtained by weighted averaging of the grayscale values ​​of the center pixels of all these similar blocks. The calculation formula is as follows:

[0060] (6)

[0061] In equation (6), The location of the target pixel to be denoised; For other pixels within the search domain Ω; L represents the denoised image; L represents each frame after inter-frame registration. The search range is used to define the set of pixels that participate in the calculation of the current pixel's denoising value; (a, b) is the weight function; Normalization factor; and These are the neighborhoods centered on the pixel to be denoised and the pixels within the search domain, respectively. Parameters used to control the noise reduction intensity.

[0062] Contrast enhancement: using the denoised image As input, a contrast-limited adaptive histogram equalization is used to divide the image into 8×8 blocks with a contrast limit of 2.0. Then, grayscale normalization is performed to linearly map all pixel values ​​to the [0,1] interval, resulting in the enhanced image, i.e., the preprocessed video image.

[0063] Preferably, insufficient contrast in ultrasound images is a common technical limitation. Low-contrast environments result in subtle grayscale differences between lesions and surrounding normal glands or adipose tissue, leading to blurred boundaries. This directly impacts the accuracy of key region identification and segmentation in subsequent automated analysis. Therefore, improving image contrast is a crucial preliminary step. Global histogram equalization methods are generally not adopted because they excessively amplify noise and may destroy local details. This study selects a contrast-limited adaptive histogram equalization algorithm.

[0064] The core of the adaptive histogram equalization algorithm lies in its locality and constraint. It divides the image into several consecutive rectangular context regions and performs histogram equalization independently within each region. This design allows it to adapt to differences in brightness distribution across different parts of the image, selectively enhancing local contrast. The key is the "contrast-constrained" mechanism: by pre-setting a clipping limit, the number of pixels at a single gray level in the local histogram is restricted, thus preventing artificial noise or excessive smoothing in local areas due to over-enhancement. After processing by the adaptive histogram equalization algorithm, the edges and internal textures of lesion areas are usually more visually prominent. However, due to differences in variables such as device gain and probe pressure, the overall brightness and contrast baseline may still vary between different frames and even between different cases. The calculation formula is as follows:

[0065]

[0066] In equation (6), The image to be enhanced after denoising. It is a grayscale index; The image after denoising processing The original grayscale value within a local area; G is the number of grayscale levels; This represents the enhanced image output after contrast enhancement transformation; grayscale value within a local area The probability distribution; For corresponding gray levels Contrast adjustment factor.

[0067] Furthermore, the data balancing process specifically includes: stratified random undersampling of the training set in this case, undersampling the HER2-Positive and HER2-Low categories to the same number of training samples as the HER2-Zero category. This balances the sample sizes of the three categories, providing a balanced data foundation for model training.

[0068] S2. Construct a feature module, a multi-stage feature fusion module, and a classification module based on the UniFormerV2 network in sequence. The feature module based on the UniFormerV2 network extracts features, the multi-stage feature fusion module fuses the extracted features, and the classification module obtains the probability distribution of each category based on the fused features. The overall structure is as follows: Figure 2 As shown.

[0069] Specifically, the construction of the feature module, multi-stage feature fusion module, and classification module based on the UniFormerV2 network includes:

[0070] S21. The preprocessed data is input into the feature extraction stage for further processing. The feature extraction stage uses the UniFormerV2 network structure to perform feature modeling on the input video and obtain the corresponding feature representation.

[0071] Preferably, the feature modules based on the UniFormerV2 network include:

[0072] Convolutional mapping layer: Performs preliminary spatial feature extraction on the input tensor. Each input frame first passes through a convolutional mapping layer to obtain an initial feature representation, denoted as . It is then divided into non-overlapping blocks, which are processed sequentially by the LocalUniBlock module and the Global UniBlock module;

[0073] Local UniBlock is designed to extract fine local spatiotemporal features, and is particularly suitable for ultrasound video data. For data from... Layer input features Local UniBlock needs to extract subtle but critical dynamic patterns from ultrasound videos while ensuring computational efficiency. The amplitude of tissue movement in ultrasound videos is often small and is often masked by noise, so the sensitivity to local temporal changes is crucial.

[0074] Local UniBlock module: Introduces a local multi-head relative self-attention mechanism in shallow networks, which limits attention calculation to the local spatiotemporal neighborhood and is used to capture fine-grained dynamic features such as small inter-frame deformations of lesions.

[0075] For those from the Layer input features Processing is performed to obtain l L-layer feature map :

[0076] (1)

[0077] In equation (1), For the first l The feature map output from layer 1 is used as the input feature of the current layer. As a local multi-head relative self-attention module, attention computation is limited to the local spatiotemporal neighborhood, which is used to capture subtle local dynamic changes in the feature map; The global multi-head relative self-attention module establishes spatial context dependencies through a global attention mechanism, enabling each location to perceive global spatial information; FFN is a feedforward network consisting of two fully connected layers, used to perform nonlinear transformations and dimensional adjustments on the fused spatiotemporal features; The output after processing by the local feature extraction module is the first... l Layer feature map; "For residual connections, the input features are added to the features after module transformation to promote gradient flow and preserve the original information.

[0078] The Global UniBlock module introduces learnable positional encoding and context-aware attention mechanisms into deep networks to establish global spatiotemporal dependencies. This is used to model long-range dynamic patterns such as blood perfusion time-series features, and to process the feature maps of layer l. Processing is performed to obtain l Second-level feature map of the layer That is, global features;

[0079] (2)

[0080] In Equation (2), DPE is a learnable location coding module used to inject spatiotemporal location information into the feature map, enabling the model to perceive the relationship between different frames and spatial locations. For context-aware multi-head relative self-attention modules, spatiotemporal dependencies are modeled globally, and long-range contextual information is captured by dynamically fusing content similarity and relative position bias. This is the l-th layer feature map output after processing by the global dependency modeling module; For residual connections, the input features are added to the transformed features of the modules to facilitate gradient flow and preserve the original information.

[0081] The core task of Global UniBlock is to establish long-range dependencies within a global spatiotemporal scope. In breast ultrasound videos, dynamic patterns related to HER2 expression status are not limited to local regions. The temporal characteristics of blood perfusion in the tumor region often involve signal changes across multiple cardiac cycles. Such features require the model to capture global spatiotemporal context. Based on this need, Global UniBlock introduces a learnable location encoding module. This differs from the fixed positional encoding in traditional Transformers. The parameters of the DPE are dynamically updated during training. This design stems from the fact that the actual semantic meaning in breast ultrasound videos is not absolute. Changes in probe angle or scanning depth can lead to different anatomical structures corresponding to the same coordinates. The learnable encoding mechanism allows the model to adaptively adjust its positional representation during training, flexibly aligning spatiotemporal relationships across different video samples. Furthermore, a context-aware attention module is also required. Unlike traditional self-attention mechanisms, When calculating weights, not only the content similarity between the query and the key is considered, but the relative positional bias modulated by the current input context is also dynamically fused. This means that the generation process of the attention matrix is ​​more sensitive to the input content itself. After the above processing, the initial integration from local details to global patterns has been completed at the semantic level. The information carried by each spatiotemporal location not only includes its own visual content, but also implies its relative role in the global dynamic context.

[0082] This framework, based on the Transformer, combines the advantages of convolutional neural networks in local feature extraction with the Transformer's ability to model global spatiotemporal dependencies, thereby enabling multi-level feature analysis of ultrasound videos. UniFormerV2 reduces inter-frame redundancy by introducing local multi-head relative self-attention in shallow networks; and captures a wider range of spatiotemporal dependencies through global multi-head relative self-attention in deep networks, thus achieving efficient video feature modeling.

[0083] S22. Construct a multi-stage feature fusion module to perform weighted fusion of all secondary feature maps and finally generate a unified fused feature representation;

[0084] To effectively integrate the hierarchical representations captured at different stages of the model, global features from all stages are needed. The input is aggregated into a specially designed multi-stage fusion module, where... This represents the total number of stages, and the process begins with a deconvolution operation. Resolution alignment is performed on feature maps of different depths to ensure consistency in spatial dimensions. Then, a set of learnable stage weights is used. The aligned features are weighted and summed to generate a unified fused feature representation. The mathematical expression for calculating this fusion feature is as follows:

[0085] (3)

[0086] Finally, the fusion features are obtained. Then, input it into the classification module.

[0087] S23. Input the fused features into the classification module, compress the spatiotemporal dimension of the fused features to extract a fixed-length feature vector, and perform linear mapping and normalization processing to finally predict the probability distribution of each category.

[0088] First, a global spatiotemporal pooling operation (GSTP) is employed, performing average pooling in the time dimension and max pooling in the spatial dimension. The vector is compressed into a fixed-length feature vector. This vector is then linearly transformed by a classification head, which consists of a weight matrix M. M is a learnable parameter matrix, initially set to random values ​​and optimized during training. Its output dimension is the number of classes (3 classes in this study, corresponding to HER2-Zero, HER2-Low, and HER2-Positive). Finally, the vector is normalized using the Softmax function to obtain the probability distribution for each class. That is, the final prediction result;

[0089] (4)

[0090] The output probabilities are: HER2-Zero class 0.02, HER2-Low class 0.93, HER2-Positive class 0.05;

[0091] The model predicted that the patient's HER2 expression status was HER2-Low, which was completely consistent with the results of the gold standard pathological test.

[0092] S3. Construct the standardized training set data into tensor form and input it into the constructed model to obtain the trained model;

[0093] Preferably, the preprocessed video image is constructed into a tensor form with frame number T=240, image size H=224, W=224, number of channels C=3, and the final tensor dimension is (1,240,224,224,3). The input tensor is processed layer by layer through convolutional mapping layer, multiple Local UniBlock modules and Global UniBlock module.

[0094] The preprocessed input video is represented as a shape of The tensor, in which It's the frame rate. and These represent the image height and width, respectively. It refers to the number of channels.

[0095] S4. Input the test set into the trained model to obtain the final prediction result of HER2 expression status.

[0096] This study systematically evaluated the TransHER2 framework on a private breast ultrasound video dataset. To verify its effectiveness, six representative models were selected for horizontal comparison. These six models cover several mainstream technical routes: ResNet50, as a representative of traditional two-dimensional convolutional neural networks

[104] , is used to extract static features of single-frame images; 3DCNN directly captures the spatiotemporal information in video segments through three-dimensional convolutional kernels

[105] ; VIVIT segments the video and inputs it into the Transformer encoder, which belongs to the video analysis method based on Transformer; Video Swin Transformer introduces a shift window mechanism to enhance the spatiotemporal feature modeling capability; in addition, it also includes two classic machine learning models, support vector machine and decision tree, whose input features are based on manually extracted temporal descriptors of ultrasound videos. All models adopt the same preprocessing process, including video frame sampling, size normalization, pixel value standardization and other steps, and are trained in the same hardware environment to ensure the fairness of the experimental setup.

[0097] The experimental results are summarized in Table 1. This table lists the comprehensive performance evaluations of each model using metrics such as Accuracy, Recall, Precision, and F1-Score. This evaluation ensures the accuracy and reliability of the models in classifying HER2 expression status from breast ultrasound videos. Data shows that TransHER2 performed best across all key metrics, achieving an F1 score of 0.9658. This advantage is significant compared to the other six models. The F1 scores for ResNet50 are 0.8721, 3DCNN 0.8913, VIVIT 0.9245, Video Swin Transformer 0.9382, Support Vector Machine 0.8034, and Decision Tree 0.7765. TransHER2's F1 score is approximately 2.8 percentage points higher than the second-best performing Video Swin Transformer. This gap demonstrates that TransHER2 has a certain advantage in capturing discriminative spatiotemporal features in ultrasound videos, thereby improving the accuracy of HER2 expression status classification. Ultrasound videos contain dynamic blood flow information and tissue elasticity changes in the tumor region, and this temporal information may have supplementary value for judging HER2 expression. Specific details of key indicators such as Precision, Recall, F1-score, and Accuracy for each model across the three HER2 expression status categories (HER2-Zero, HER2-Low, and HER2-Positive) are shown in the table below.

[0098] Table 1. Comparison results with other models

[0099]

[0100] Table 2. Comparison of details of other models

[0101]

[0102] Ultimately, this embodiment uses three patients as specific test subjects, as described below:

[0103] Patient 1: HER2-Low expression case.

[0104] (1) Input data: A female patient, aged 52 years, preoperative breast ultrasound video, video duration 8 seconds, a total of 240 frames of images were acquired, each frame image size is 640×480 pixels, three-channel RGB format. After the operation, the patient was independently detected by immunohistochemistry and fluorescence in situ hybridization by three senior pathologists, and the HER2 expression status was jointly confirmed to be HER2-Low.

[0105] (2) Pretreatment process:

[0106] Data balancing: The training set described in this case was undersampled using stratified random sampling, undersampling the HER2-Positive and HER2-Low categories to the same 27 cases as the HER2-Zero category.

[0107] Inter-frame registration: Optical flow was used for registration, with a regularization parameter α of 0.1. The 240 frames were sequentially registered with the first frame to eliminate jitter caused by breathing and probe movement.

[0108] Denoising: The nonlocal mean algorithm is used, the search window size is set to 11×11, the similarity window size is set to 5×5, the filtering parameter h is set to 10, and the denoising is performed on each frame of the registered image.

[0109] Contrast enhancement: Adaptive histogram equalization with limited contrast is used to divide the image into 8×8 blocks, with the contrast limit set to 2.0. Then, grayscale normalization is performed to linearly map all pixel values ​​to the [0,1] interval.

[0110] (3) Model processing:

[0111] Input tensor construction: The preprocessed video images are constructed into tensor form with frame number T=240, image size H=224, W=224, number of channels C=3, and the final tensor dimension is (1,240,224,224,3).

[0112] Feature extraction and fusion: Forward propagation is performed using the TransHER2 network. In the local feature extraction module, a local temporal multi-head relative self-attention mechanism focuses on subtle changes in tumor morphology between consecutive frames; in the global dependency modeling module, a context-aware attention mechanism captures the overall tumor deformation pattern across multiple cardiac cycles; in the multi-stage feature fusion module, the output features from the four stages are weighted and fused, with learnable weights c1 to c4 trained to 0.23, 0.31, 0.26, and 0.20, respectively.

[0113] Classification output: The fused features are input into the classification head after global spatiotemporal pooling. The Softmax output probabilities are: 0.02 for HER2-Zero class, 0.93 for HER2-Low class, and 0.05 for HER2-Positive class.

[0114] (4) Output results: The model predicted that the patient's HER2 expression status was HER2-Low, which was completely consistent with the pathological gold standard test results.

[0115] Patient 2: HER2-Positive expression case;

[0116] (1) Input data: A female patient, aged 45 years, preoperative breast ultrasound video, video duration 6 seconds, a total of 180 frames were acquired, each frame image size is 640×480 pixels, three-channel RGB format. The patient was confirmed to have HER2-Positive expression by postoperative pathological examination.

[0117] (2) Pretreatment process:

[0118] Data balancing: Similar to Example 1, a stratified random undersampling strategy is used to balance the training set.

[0119] Inter-frame registration: Optical flow is used for registration, with a regularization parameter α of 0.1. The 180 frames are registered with the first frame in sequence.

[0120] Noise reduction: The nonlocal mean algorithm is used, with a search window size of 11×11, a similarity window size of 5×5, and a filtering parameter h of 10.

[0121] Contrast enhancement: Adaptive histogram equalization with limited contrast is adopted, with a block size of 8×8, a contrast limit of 2.0, and grayscale normalization is performed.

[0122] (3) Model processing:

[0123] Input tensor construction: Construct the input tensor with frame number T=180, image size H=224, W=224, number of channels C=3, and tensor dimensions (1,180,224,224,3).

[0124] Feature extraction and fusion: Forward propagation was performed using the TransHER2 network. The network captured a clear pattern of enhanced blood flow signals in the tumor region across multiple cardiac cycles in the case video, and the global dependency modeling module effectively encoded this long-range dynamic feature.

[0125] Classification output: Softmax output probabilities are: HER2-Zero class 0.01, HER2-Low class 0.12, HER2-Positive class 0.87.

[0126] (4) Output results: The model predicts that the patient’s HER2 expression status is HER2-Positive, which is consistent with the pathological gold standard detection results.

[0127] Patient 3: HER2-Zero expression case

[0128] (1) Input data: A female patient, aged 58 years, preoperative breast ultrasound video, video duration 7 seconds, a total of 210 frames were acquired, each frame image size is 640×480 pixels, three-channel RGB format. The patient was confirmed to have HER2-Zero expression by postoperative pathological examination.

[0129] (2) Pretreatment process:

[0130] Data balancing: Similar to Example 1, a stratified random undersampling strategy is used to balance the training set.

[0131] Inter-frame registration: Optical flow was used for registration, with a regularization parameter α of 0.1. The 210 frames were sequentially registered with the first frame.

[0132] Noise reduction: The nonlocal mean algorithm is used, with a search window size of 11×11, a similarity window size of 5×5, and a filtering parameter h of 10.

[0133] Contrast enhancement: Adaptive histogram equalization with limited contrast is adopted, with a block size of 8×8, a contrast limit of 2.0, and grayscale normalization is performed.

[0134] (3) Model processing:

[0135] Input tensor construction: Construct the input tensor with frame number T=210, image size H=224, W=224, number of channels C=3, and tensor dimensions (1,210,224,224,3).

[0136] Feature extraction and fusion: Forward propagation was performed using the TransHER2 network. In this case video, the tumor morphology changed little between frames, and the blood flow signal was not obvious. The local feature extraction module captured limited dynamic changes, and the long-range features output by the global dependency modeling module also showed low activation levels.

[0137] Classification output: Softmax output probabilities are: HER2-Zero class 0.89, HER2-Low class 0.07, HER2-Positive class 0.04.

[0138] (4) Output results: The model predicts that the patient’s HER2 expression status is HER2-Zero, which is consistent with the pathological gold standard detection results.

[0139] Example 2

[0140] This embodiment also provides an electronic device, including:

[0141] At least one processor; and

[0142] The memory stores instructions that, when executed by the at least one processor, cause the at least one processor to perform the wire foreign object detection method based on the improved YOLOv11 as described above.

[0143] In this embodiment, the electronic device may include, but is not limited to: personal computer, server computer, workstation, desktop computer, laptop computer, notebook computer, mobile computing device, smartphone, tablet computer, cellular phone, personal digital assistant (PDA), handheld device, messaging device, wearable computing device, consumer electronic device, etc.

[0144] Example 3

[0145] This embodiment also provides a machine-readable storage medium storing executable instructions that, when executed, cause the machine to perform the wire foreign object detection method based on the improved YOLOv11 as described above.

[0146] Specifically, a system or apparatus equipped with a readable storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer or processor of the system or apparatus can read and execute the instructions stored in the readable storage medium.

[0147] In this case, the program code read from the readable medium itself can perform the functions of any of the above embodiments, and therefore the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of this specification.

[0148] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer or the cloud via a communication network.

[0149] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0150] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0151] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0152] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0153] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction, characterized in that, The method includes: S1. Using the original breast ultrasound video as input data, perform multi-level data preprocessing operations on the input data to obtain preprocessed video images, then divide them into training set and test set, and perform data balancing processing on the training set to obtain a standardized training set. S2. Construct a feature module, a multi-stage feature fusion module, and a classification module based on the UniFormerV2 network in sequence; the feature module based on the UniFormerV2 network extracts features, the multi-stage feature fusion module fuses the extracted features, and the classification module obtains the probability distribution of each category based on the fused features. The construction of feature modules based on the UniFormerV2 network specifically includes: Convolutional mapping layer: This layer performs preliminary spatial feature extraction on the input tensor. Each input frame first passes through a convolutional mapping layer to obtain an initial feature representation, denoted as . F 0 It is then divided into non-overlapping blocks and processed through N stages, each stage including a Local UniBlock module and a Global UniBlock module. l The stage index is the first one. l layer; Local UniBlock module: Introduces a local multi-head relative self-attention mechanism into shallow networks, which limits attention computation to a local spatiotemporal neighborhood and is used to capture fine-grained dynamic features. For those from ( l -1) Input features of layer Processing is performed to obtain the first... l Feature map of layer : （1） In equation (1), For the first l The first-level feature map output from layer 1 is used as the input feature map for the current layer, and ; This is a local multi-head relative self-attention module; The first is a global multi-head relative self-attention module; the second is a feedforward network consisting of two fully connected layers, used to perform nonlinear transformation and dimensionality adjustment on the fused spatiotemporal features. For the extracted first l Level 1 feature map; "For residual connection" ; The Global UniBlock module introduces learnable positional encoding and context-aware attention mechanisms into deep networks to establish global spatiotemporal dependencies. This is used to model long-range dynamic patterns such as blood perfusion time-series features. l Level 1 Feature Map Processing is performed to obtain the first... l Second-level feature map of the layer : （2） In equation (2), DPE is a learnable position coding module; A context-aware multi-head relative self-attention module; The output after processing by the global dependency modeling module l Second-level feature map; For residual connection ; S3. Construct the standardized training set data into tensor form and input it into the constructed model to obtain the trained model; S4. Input the test set into the trained model to obtain the final prediction result of HER2 expression status.

2. The method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction according to claim 1, characterized in that, The multi-level data preprocessing includes: inter-frame registration processing, noise reduction processing, and contrast enhancement processing; Inter-frame registration processing: Using the original breast ultrasound video as input, inter-frame registration is performed using the optical flow method. The regularization parameter α is set to 0.

1. The first frame is used as the reference frame, and all frame images are registered with the first frame in sequence. By calculating the pixel-level motion vectors of adjacent frames and correcting them in reverse, the spatial alignment between the reference frame and subsequent frames is achieved, and the registered image L is obtained. Denoising: Using the registered image as input, a nonlocal mean algorithm is employed with a search window size of 11×11 and a similarity window size of 5×5. The filtering parameter h is set to denoise each frame of the registered image, resulting in the denoised image. ; Contrast enhancement: using the denoised image As input, a contrast-limited adaptive histogram equalization is used to divide the image into 8×8 blocks, set a contrast limit value, and then perform grayscale normalization to linearly map all pixel values ​​to the [0,1] interval to obtain the enhanced image.

3. The method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction according to claim 2, characterized in that, The training set employs stratified random undersampling, undersampling the HER2-Positive and HER2-Low categories to the same number of training samples as the HER2-Zero category.

4. The method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction according to claim 1 or 3, characterized in that, The preprocessed video images are constructed into tensor form, with T frames and H*W image size. and Let C represent the image height and width, respectively, with C channels, and the final tensor dimension is (1,240,224,224,3).

5. The method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction according to claim 1, characterized in that, The construction of the multi-stage feature fusion module specifically includes: weighted fusion of all secondary feature maps to generate a unified fused feature representation; Global features of all stages Input is aggregated into a multi-stage fusion module: First, deconvolution is performed. The feature maps at different depths are resolution aligned, and then a set of learnable stage weights are used. The aligned features are weighted and summed to generate a unified fused feature representation. : （3） Finally, the fusion features are obtained. Then, input it into the classification module.

6. The method for identifying HER2 expression status in breast ultrasound video based on spatiotemporal feature interaction according to claim 5, characterized in that, The construction of the classification module specifically includes: compressing the spatiotemporal dimension of the fused features to extract a fixed-length feature vector, and performing linear mapping and normalization processing to finally predict the probability distribution of each category; First, a global space-time pooling operation (GSTP) is used to... The vector is compressed into a fixed-length feature vector. This vector is then linearly transformed by a classification head, which consists of a weight matrix M, where M is a learnable parameter matrix. The output dimension is the number of categories, i.e., 3 categories, corresponding to HER2-Zero, HER2-Low, and HER2-Positive. Finally, after normalization using the Softmax function, the probability distributions for each category are obtained. That is, the final prediction result; （4）。 7. An electronic device, characterized in that, The electronic device includes: processor; A memory on which computer programs that can run on the processor are stored; When the computer program is executed by the processor, it implements the steps of the breast ultrasound video HER2 expression state recognition method based on spatiotemporal feature interaction as described in any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images