A breast ultrasound video lesion detection method fusing spatio-temporal context information

Abstract

This invention discloses a method for detecting breast lesions in breast ultrasound video by incorporating spatiotemporal contextual information. Addressing the shortcomings of existing methods such as insufficient spatial representation, weak temporal correlation modeling, and high computational cost, this invention constructs a detection network comprising a backbone network, a single-frame cross-scale feature fusion module, a detection head, a cross-batch feature aggregation module, and a multi-frame temporal feature fusion module. The video is divided into consecutive multi-frame temporal segments, and shallow and deep features are enhanced through frequency-domain aware cross-scale fusion to improve spatial representation. A candidate feature memory stores high-quality candidate features, and cross-batch feature aggregation absorbs historical temporal information with low computational overhead. The multi-frame temporal feature fusion module then adaptively weights classification and regression features to highlight key temporal clues. This invention effectively improves the accuracy of breast ultrasound video lesion detection and grading without significantly increasing computational cost, and possesses good real-time performance and clinical application value.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing and computer vision technology, specifically to a method for detecting breast lesions using ultrasound video that integrates spatiotemporal contextual information. Background Technology

[0002] Breast cancer is one of the leading malignant tumors threatening women's health. Early detection and accurate grading are crucial for improving cure rates and reducing mortality. Ultrasound imaging, with its advantages of being radiation-free, real-time, and relatively inexpensive, has become a widely used imaging technique for breast lesion screening and diagnosis. Compared to static ultrasound images, breast ultrasound videos simultaneously contain information on spatial structure and temporal evolution, providing physicians with more comprehensive evidence and decision support in clinical scenarios such as early tumor diagnosis, risk screening and prevention, individualized treatment planning, and efficacy evaluation and follow-up.

[0003] However, existing deep learning-based methods for detecting breast lesions in ultrasound videos still have the following problems: First, most studies only focus on lesion localization and simplified benign / malignant binary classification in static ultrasound images, failing to fully consider the standardized BIRADS grading required in clinical practice, making it difficult to provide patients with more refined risk stratification and personalized treatment plans; Second, in the BIRADS grading scenario, the appearance of lesions and the tissue background often exhibit high similarity and highly dense texture structures. Traditional feature aggregation methods often use simple upsampling of deep features and pixel-by-pixel addition with shallow features, which easily leads to boundary blurring and loss of detail, resulting in insufficient detection and localization accuracy of suspicious lesions; Third, if the temporal correlation in dynamic videos is not effectively modeled, the spatiotemporal context information is difficult to fully utilize. Existing video target detection methods such as FGFA, SELSA, and MEGA... Introducing multi-frame information often leads to a significant increase in computational complexity, and directly splicing multiple batches of features introduces redundancy and noise, which weakens detection performance. Fourth, while existing detection algorithms for ultrasound videos such as CVA-Net, STNet, and UltraDet have partially alleviated the above problems, there is still a clear trade-off between generalization ability on the BIRADS graded video dataset, efficiency of temporal feature utilization, and computational overhead.

[0004] In summary, how to effectively integrate spatiotemporal context information in breast ultrasound video scenarios, and significantly improve the accuracy of lesion detection and BIRADS grading while controlling computational costs, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method for detecting breast lesions using ultrasound video that integrates spatiotemporal contextual information, in order to solve the problems of insufficient spatial representation, weak temporal correlation modeling, and excessively high computational cost in existing techniques for detecting breast lesions using ultrasound video.

[0006] To achieve the above objectives, the technical solution provided by this invention is: a method for detecting breast lesions using ultrasound video that integrates spatiotemporal context information, comprising the following steps: S1: Construct a detection network, including a backbone network, a single-frame cross-scale feature fusion module, a detection head, a cross-batch feature aggregation module, and a multi-frame temporal feature fusion module; train the detection network using labeled breast ultrasound videos until convergence; S2: The breast ultrasound video is divided into consecutive multi-frame temporal segments and input into the backbone network. The output is high-resolution shallow features and low-resolution deep features. The single-frame cross-scale feature fusion module constructs low-pass and high-pass filters based on the shallow features. It upsamples and smooths the deep features and enhances the edges of the shallow features, respectively. The two filters are added together to obtain the cross-scale fused features, which are input into the detection head and output classification feature map, regression feature map and bounding box prediction. S3: Set up a candidate feature memory to store high-quality candidate features; extract candidate features, their confidence scores, and classification scores from the classification feature map of the current batch; select several high-quality candidate features based on the combined value of the confidence scores and classification scores; read historical high-quality candidate features from the candidate feature memory, input them and the high-quality candidate features of the current batch into the cross-batch feature aggregation module, aggregate them using a multi-head attention mechanism, add them to the high-quality candidate features of the current batch, update the candidate feature memory with the added high-quality candidate features, and use the added features as the cross-batch enhanced classification features for this frame; S4: Input the cross-batch enhanced classification features and corresponding regression features of multiple frames within the same time segment into the multi-frame time feature fusion module; This module linearly maps the classification and regression features of each frame to obtain the query and key, calculates the attention weights and weights them with confidence scores; The weighted features of the classification and regression branches are added together and concatenated with the value matrix, and then linearly projected to obtain the cross-frame enhanced classification features; S5: Use the cross-frame enhanced classification features as the final classification basis, output the final lesion category for each frame, and use the bounding box prediction as the final position to obtain the lesion category and position for each frame of the video.

[0007] To optimize the above technical solution, the specific measures also include: In step S1, the total loss function used to train the detection network is:

[0008] in, For regression loss; Loss of confidence in the target; This is the initial classification loss; Enhance the classification loss corresponding to the classification features across frames; and These are the weighting coefficients.

[0009] Furthermore, in step S2, the single-frame cross-scale feature fusion module inputs a low-pass filter. right Upsampling results and high-pass filter right The high-frequency enhancement results, output The expression is:

[0010] in, Main backbone network Shallow layer features; Main backbone network Deep layer features This is a cross-scale fusion feature.

[0011] Further, in step S2, the construction of the low-pass filter based on shallow features specifically includes: Shallow features of the input Perform 3×3 convolution dimensionality reduction to obtain spatially varied filter weights. ; Using pixel reverse washing The height and width are halved, the channels are enlarged, and the channels are split into four groups, each group being a spatial variation filter. , For each group of filter weights Normalization is performed to obtain a low-pass filter. ; Utilizing each set of low-pass filters for deep features Weighted filtering is performed to obtain the corresponding low-pass features. Then, each group of low-pass features is rearranged through pixel shuffling to form a feature that is upsampled twice as the output of the dynamic low-pass filter.

[0012] In step S2, the high-pass filter is constructed based on shallow features, specifically including: processing the input shallow features... 3×3 convolution dimensionality reduction is performed to obtain spatially varying filter weights, which are then normalized and inverted to generate a high-pass filter. This high-pass filter is then used to refine shallow features. Convolution is performed to enhance high-frequency edge components, resulting in a high-pass filter output.

[0013] Step S3 is as follows: The confidence score and classification score are multiplied together via broadcasting to obtain a comprehensive probability score, which is then set to a preset threshold. Perform masking filtering; based on the comprehensive probability scores, use the TopK function to select the top K high-quality candidate features of the current batch from both the masked candidate features and the high-quality candidate features stored in the candidate feature memory. and the top K historical high-quality candidate features ;Will and After linear layer mapping, the aggregated features are obtained by inputting the cross-batch feature aggregation module; Furthermore, the aggregated features are combined with high-quality candidate features from the current batch. The elements are added one by one to obtain cross-batch enhanced classification features, and the candidate feature memory is updated with these cross-batch enhanced classification features. The original historical high-quality candidate features are replaced with the high-quality candidate features of the current batch.

[0014] In step S3, the capacity of the candidate feature memory is set to... ,in When updating the candidate feature memory, only high-quality features from the current batch are written, while low-quality features are discarded, ensuring that the current batch can utilize at most the previous batches. Batch time sequence characteristics information.

[0015] Step S4 specifically includes: The same time segment Frame, number The cross-batch enhanced classification features of the frame are denoted as The corresponding regression feature is denoted as ,in Enhance classification features across batches for each frame. With regression features The query vector is obtained through linear mapping respectively. , and key , The classification attention weight matrix is ​​calculated by scaling the dot product attention. and regression attention weight matrix ; Obtain the confidence score for each candidate feature, including the classification score and the IoU score, and construct the classification weight matrix through a broadcast mechanism. and regression weight matrix The confidence score is multiplied element-wise by the attention weight, and then the weighted attention distribution is obtained by applying softmax. Furthermore, the weighted features of the classification weight matrix are... Weighted characteristics of the regression weight matrix After addition and the shared value matrix By concatenating the series, the fusion features are obtained. :

[0016] For the value matrix Calculate the similarity matrix after layer normalization. Select those with similarity exceeding the threshold The reference terms are average pooled, and the average pooled features are then compared with... The data are concatenated and then linearly projected to obtain cross-frame enhanced classification features.

[0017] The present invention also proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a breast ultrasound video lesion detection method that integrates spatiotemporal context information as described above.

[0018] The present invention also proposes a computer-readable storage medium storing a computer program that enables a computer to execute a breast ultrasound video lesion detection method that integrates spatiotemporal context information as described above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention designs a frequency-domain-aware single-frame cross-scale feature aggregation module. By using dynamic low-pass and dynamic high-pass filters to coordinate high-level semantic information and low-level contour information in the frequency domain, it overcomes the boundary blurring and detail loss problems caused by simple upsampling and pixel-by-pixel addition in traditional cross-scale aggregation, and significantly improves the spatial localization accuracy and edge clarity of lesions in a single-frame image.

[0020] This invention proposes a cross-batch feature aggregation module based on a candidate feature memory. By storing and filtering high-quality candidate features, the current batch can make full use of the temporal information of historical batches, which significantly reduces computational complexity, avoids redundancy and noise accumulation, and maintains better computational efficiency while enhancing the ability to model in the time dimension.

[0021] This invention designs a multi-frame feature aggregation module based on a self-attention mechanism. By jointly utilizing the feature representations of classification and regression branches and the weight matrix guided by confidence, it adaptively weights cross-frame temporal information, which can effectively highlight key temporal clues related to the current task and significantly improve the robustness and reliability of lesion category discrimination.

[0022] The overall framework of this invention achieves detection accuracy surpassing mainstream video object detection methods on both publicly available breast ultrasound video datasets and a self-built BIRADS grading video dataset, while simultaneously reducing the detection time per frame compared to existing typical methods. Experiments demonstrate that this invention significantly improves lesion detection and BIRADS grading performance in breast ultrasound videos while maintaining real-time inference capabilities, fully reflecting its technological advancement and practical value. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating the overall process of a breast ultrasound video lesion detection method that integrates spatiotemporal context information, as proposed in this invention. Figure 2 This is a structural diagram of the frequency domain-aware single-frame cross-scale feature aggregation module FD-SCFAM proposed in this invention; Figure 3 This is a flowchart illustrating the structure and operation of the dynamic low-pass filter proposed in this invention. Figure 4 This is a flowchart illustrating the structure and operation of the dynamic high-pass filter proposed in this invention. Figure 5 This is a schematic diagram of the cross-batch feature aggregation module CB-FAM proposed in this invention; Figure 6 This is a schematic diagram of the multi-frame feature aggregation module MF-FAM proposed in this invention; Figure 7 This is a visual comparison of the results of breast ultrasound video lesion detection according to the present invention.

[0024] Figure 8 This is a structural diagram of the backbone network AWFD-FastNet in an embodiment of the present invention. Detailed Implementation

[0025] The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0026] This invention provides a method for detecting breast lesions using ultrasound video that integrates spatiotemporal contextual information. In some embodiments, such as... Figure 1 As shown, it includes the following steps: S1: Construct a detection network, including a backbone network, a single-frame cross-scale feature fusion module, a detection head, a cross-batch feature aggregation module, and a multi-frame temporal feature fusion module; train the detection network using labeled breast ultrasound videos until convergence; S2: The breast ultrasound video is divided into consecutive multi-frame temporal segments and input into the backbone network. The output is high-resolution shallow features and low-resolution deep features. The single-frame cross-scale feature fusion module constructs low-pass and high-pass filters based on the shallow features. It upsamples and smooths the deep features and enhances the edges of the shallow features, respectively. The two filters are added together to obtain the cross-scale fused features, which are input into the detection head and output classification feature map, regression feature map and bounding box prediction. S3: Set up a candidate feature memory to store high-quality candidate features; extract candidate features, their confidence scores, and classification scores from the classification feature map of the current batch; select several high-quality candidate features based on the combined value of the confidence scores and classification scores; read historical high-quality candidate features from the candidate feature memory, input them and the high-quality candidate features of the current batch into the cross-batch feature aggregation module, aggregate them using a multi-head attention mechanism, add them to the high-quality candidate features of the current batch, update the candidate feature memory with the added high-quality candidate features, and use the added features as the cross-batch enhanced classification features for this frame; S4: Input the cross-batch enhanced classification features and corresponding regression features of multiple frames within the same time segment into the multi-frame time feature fusion module; This module linearly maps the classification and regression features of each frame to obtain the query and key, calculates the attention weights and weights them with confidence scores; The weighted features of the classification and regression branches are added together and concatenated with the value matrix, and then linearly projected to obtain the cross-frame enhanced classification features; S5: Use the cross-frame enhanced classification features as the final classification basis, output the final lesion category for each frame, and use the bounding box prediction as the final position to obtain the lesion category and position for each frame of the video.

[0027] In some implementations, the backbone network AWFD-FastNet replaces the traditional backbone network CSPDarknet-53 with the lightweight general-purpose network FasterNet, effectively improving feature modeling efficiency and performance. Simultaneously, an Adaptive Wavelet Feature Down-sampling (AWFD) module is integrated to replace the traditional downsampling layer. The AWFD module uses discrete wavelet transform to decompose feature map data, extracting low-frequency and high-frequency components, and processes them in parallel using a low-frequency self-attention module (LFSA) and a residual convolution module, effectively improving the network's feature extraction capability for lesion regions. FBlock consists of partial convolutions (PConv) and pointwise convolutions (PW-Conv), with a batch normalization (BN) layer activated by ReLU added between the two pointwise convolutional layers, maintaining feature diversity and reducing computational latency. The structure is as follows: Figure 8 As shown.

[0028] Clinically acquired breast ultrasound videos were divided into fixed-length time segments according to time sequence. Preprocessing: grayscale normalization is performed on each frame of the image to normalize the pixel values ​​to the [0,1] range; data enhancement operations such as random horizontal flipping, random cropping and color dithering are used to uniformly scale each frame of the image to a resolution of 640×640.

[0029] Each frame in the time sequence is sequentially fed into the backbone network AWFD-FastNet, which outputs multi-scale features that include high-resolution shallow features and low-resolution deep features.

[0030] A frequency-domain-aware single-frame cross-scale feature aggregation module, FD-SCFAM, is used to aggregate multi-scale features. Frequency-domain operations are used to balance the semantic information of deep features with the contour information of shallow features before inputting them into a low-pass filter. right Upsampling results and high-pass filter right The high-frequency enhancement results, output The expression is:

[0031]

[0032] in, Main backbone network Shallow layer features; Main backbone network Deep layer features This is the upsampled result of the low-pass filter output; This represents the high-frequency enhancement result of the high-pass filter output; This is a cross-scale fusion feature.

[0033] In some implementations, the low-pass filter (DLPF) structure is as follows: Figure 3 As shown, this aims to effectively smooth deep features to alleviate feature inconsistency. Depthwise separable convolutions are used for shallow features. Channel dimensionality reduction is performed, compressing the number of channels from C to K²×G (K is the filter kernel size, G is the number of groups), and batch normalization is used for stable training. Softmax is used to normalize the filter weights at each spatial location to ensure that the sum of the weights is 1, specifically: Shallow features of the input Perform 3×3 convolution dimensionality reduction to obtain spatially varied filter weights. , The expression is:

[0034] in, Represents the real number field; Indicates the kernel size of the low-pass filter; W represents the height (spatial dimension) of the shallow feature map; W represents the width (spatial dimension) of the shallow feature map.

[0035] The filter weights are adjusted by using pixel unshuffle. The shape was reshaped, reducing the height and width by half (from 2H×2W to H×W), and the channels were enlarged by a factor of four; the channels were split into four groups, each with a spatially varied sub-filter weight. , ∈ ,in, Sub-filter grouping index For each group of filter weights After normalization, a smooth low-pass filter is finally generated. The expression is:

[0036] in,( i , j ) is the central spatial location; Indicates center ( i , j Spatial location outside; Represented in pixels ( i , j ) as the center Neighborhood spatial location set (total) (Number of positions) are used for local weighted convolution. After pixel inverse shuffling and shape reshaping, the filter weights... Each location contains One filter.

[0037] In some implementations, subpixel convolutional upsampling techniques are used for deep features. ∈ Perform upsampling; calculate the low-pass features generated by the low-pass filter for each group, as shown in the following expression:

[0038] Four sets of features after low-pass filtering Rearranged to form features that are upsampled twice as much. ,Right now:

[0039] PixelShuffle represents the pixel shuffling (subpixel convolution upsampling) operation; This represents the features obtained from the first low-pass filter. This represents the features obtained from the second low-pass filter; This represents the features obtained from the third low-pass filter. This represents the features obtained from the fourth group of low-pass filtering; In some implementations, a high-pass filter (DHPF) extracts and amplifies high-frequency components from shallow features, preserves key edge details, and combines them with deeper features during feature fusion to form a more refined edge representation, as shown in the structure... Figure 4 As shown, it consists of convolutional layers, normalization layers, and filter inversion operations. The specific construction process is as follows: Shallow features of the input By performing convolutional dimensionality reduction, spatially varying filter weights are obtained, i.e.:

[0040] right Normalization and filter inversion are performed to obtain a high-pass filter; the high-pass filter is then used to analyze shallow features. Spatially varied high-pass filtering amplifies and preserves high-frequency components, resulting in enhanced shallow features. ; In some implementations, and The upsampling results are added at the pixel level to obtain the final output of FD-SCFAM, which is the cross-scale fusion feature, such as... Figure 2 As shown.

[0041] The cross-scale fused features are input into the detection head to generate classification feature maps and regression feature maps for all frames in the current time segment, and to obtain initial class predictions and bounding box predictions.

[0042] The cross-batch feature aggregation module CB-FAM in this invention achieves a balance between computational cost and performance, and its structure is as follows: Figure 5 As shown, the core idea is to construct a candidate feature memory (CFMB), which enables the current batch of frames to fully utilize the high-quality features of previous batches of frames through a cross-batch information storage and fusion mechanism.

[0043] In some implementations, candidate features are extracted from the classification feature map of the current batch, and their confidence scores and classification scores are multiplied together using a broadcast method to obtain a comprehensive probability score. Features that meet the criteria are then filtered using a mask. The formula for calculating the mask is:

[0044] in, Represents the mask function; Represents the overall probability score; The preset threshold for the mask is set to 0.75.

[0045] Based on the comprehensive probability scores, the TopK function is used to select the top K high-quality candidate features of the current batch from the candidate features after masking and from the high-quality candidate features stored in the candidate feature memory. and the top K historical high-quality candidate features The expression is:

[0046]

[0047] in, It is a feature selection function. These are high-quality candidate features for the current batch; These are high-quality candidate features from historical periods; , It is the number of features in the candidate region. It is the feature dimension.

[0048] Will After linear layer mapping, the aggregated features are obtained by inputting the cross-batch feature aggregation module. The expression is:

[0049] in, , and There are three linear layers. , This is expressed as the number of candidate boxes per frame; Indicates the number of related frames.

[0050] Combine the aggregated features with the high-quality candidate features of the current batch. Element-wise addition yields cross-batch enhanced classification features:

[0051] The candidate feature memory is updated using the cross-batch enhanced classification feature, replacing the original historical high-quality candidate features with the current batch's high-quality candidate features, so as to facilitate the fusion of the next batch.

[0052] In some implementations, the capacity of the candidate feature memory is set to... ,in When updating the candidate feature memory, only high-quality features from the current batch are written, while low-quality features are discarded, ensuring that the current batch can utilize at most the previous batches. Batch time sequence characteristics information.

[0053] In some implementations, the multi-frame temporal feature fusion module MF-FAM adaptively fuses features from multiple frames along the temporal dimension and jointly utilizes feature representations from the classification and localization branches to enhance the understanding of cross-frame context by classification features, thereby more effectively guiding subsequent classification predictions. The structure is as follows: Figure 6 As shown, the specific process is as follows: The candidate feature set obtained by the cross-batch feature aggregation module CB-FAM is: , No. The classification branch features and regression branch features of the frame are denoted as follows:

[0054] The same time segment Frame, number The cross-batch enhanced classification features of the frame are denoted as , The corresponding regression feature is denoted as , ;in Enhance classification features across batches for each frame. With regression features The query vector is obtained through linear mapping respectively. , and key , The classification attention weight matrix is ​​calculated by scaling the dot product attention. and regression attention weight matrix ,in, ( ) denotes the linear projection operator.

[0055] Obtain the confidence score for each candidate feature. :

[0056] in, This represents the confidence prediction for a candidate feature, including the classification score and the IoU score. .

[0057] In some implementations, the classification weight matrix is ​​constructed using a broadcast mechanism. and regression weight matrix :

[0058] The confidence score is multiplied element-wise by the attention weight, and MatMul is used to... and Multiply, then pass through softmax Obtain the weighted attention distribution; weight the features of the classification weight matrix. Weighted characteristics of the regression weight matrix After addition and the shared value matrix By concatenating the series, the fusion features are obtained. :

[0059]

[0060] Log-value matrix Calculate the similarity matrix after layer normalization. Select those with similarity exceeding the threshold The reference terms are average pooled, and the average pooled features are then compared with... Series:

[0061] Cross-frame enhanced classification features are obtained through linear projection:

[0062] in, Represents the Hadamard (element-wise) product; ( ) indicates layer normalization.

[0063] In some implementations, the total loss function used to train the detection network is:

[0064] in, For regression loss; Loss of confidence in the target; This is the initial classification loss; To enhance the classification loss corresponding to the classification features across frames, the regression loss adopts the IoU loss function, and all other losses adopt the BCE loss function; and All weighting coefficients were set to 2. The NMS threshold for the final detection stage was set to 0.45. Example 1 This invention is trained and inferred using a Python 3.8 language environment, a PyTorch 1.7 neural network framework, and a 64-bit Ubuntu 22 operating system. The hardware platform is a server equipped with an Intel i9-10980XE processor, 64GB of RAM, and three NVIDIA GTX 3060 graphics cards. Basic object detection employs a detector based on the AWFD-FastNet backbone, with AdamW as the optimizer, a batch size of 16, an initial learning rate of 0.001, and 200 training epochs. The localization loss is CIoULoss, and the classification loss is Binary Cross-Entropy (BCE). In the ultrasound video enhancement stage, the proposed model was trained for 14 epochs on a breast ultrasound video dataset, with a default batch size of 16. The optimizer used was SGD, with a learning rate of 0.0025, weight decay of 0.0005, and a momentum coefficient of 0.9.

[0065] Table 1. Comparison of experimental results with other algorithms (CVA-BUS)

[0066] Table 2. Comparison results with other algorithms (BIRADS)

[0067] As shown in Tables 1 and 2, multiple comparative and ablation experiments on the publicly available CVA-BUS dataset and the self-built BIRADS video dataset demonstrate that the method proposed in this invention can significantly improve the lesion detection accuracy in breast ultrasound videos while maintaining real-time inference capabilities. On the CVA-BUS dataset, the AP50 reaches 82.3%, a significant improvement compared to mainstream video object detection methods and basic lightweight detectors. On the self-built BIRADS video dataset, the mAP50 reaches 67.6%, a 12.4% improvement over MEGA and an 11.2% improvement over the baseline using only a single-frame path. Furthermore, the detection time for a single frame image is only 17.3 ms, a 7.6 ms reduction compared to UltraDet, fully validating the effectiveness and advancement of this invention in the BIRADS graded video scenario.

[0068] like Figure 7 As shown, using Ground Truth as the real data annotation, compared with the Baseline, the present invention obtains more comprehensive detection results in terms of annotation content and has a higher overall confidence level, indicating that effective modeling of continuous information can improve the reliability of breast ultrasound video lesion detection.

[0069] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A method for detecting breast lesions using ultrasound video that integrates spatiotemporal contextual information, characterized in that, Includes the following steps: S1: Construct a detection network, including a backbone network, a single-frame cross-scale feature fusion module, a detection head, a cross-batch feature aggregation module, and a multi-frame temporal feature fusion module; train the detection network using labeled breast ultrasound videos until convergence; S2: The breast ultrasound video is divided into consecutive multi-frame temporal segments and input into the backbone network. The output is high-resolution shallow features and low-resolution deep features. The single-frame cross-scale feature fusion module constructs low-pass and high-pass filters based on the shallow features. It upsamples and smooths the deep features and enhances the edges of the shallow features, respectively. The two filters are added together to obtain the cross-scale fused features, which are input into the detection head and output classification feature map, regression feature map and bounding box prediction. S3: Set up a candidate feature memory to store high-quality candidate features; extract candidate features, their confidence scores, and classification scores from the classification feature map of the current batch; select several high-quality candidate features based on the combined value of the confidence scores and classification scores; Read historical high-quality candidate features from the candidate feature memory, input them and the high-quality candidate features of the current batch into the cross-batch feature aggregation module, use a multi-head attention mechanism to aggregate them, add them to the high-quality candidate features of the current batch, update the candidate feature memory with the added high-quality candidate features, and use the added features as the cross-batch enhanced classification features of the frame. S4: Input the cross-batch enhanced classification features and corresponding regression features of multiple frames within the same time segment into the multi-frame time feature fusion module; This module linearly maps the classification and regression features of each frame to obtain the query and key, calculates the attention weights and weights them with confidence scores; it then adds the weighted features of the classification and regression branches, concatenates them with the value matrix, and obtains the cross-frame enhanced classification features through linear projection. S5: Use the cross-frame enhanced classification features as the final classification basis, output the final lesion category for each frame, and use the bounding box prediction as the final position to obtain the lesion category and position for each frame of the video.

2. The method for detecting breast lesions using ultrasound video fusion with spatiotemporal context information according to claim 1, characterized in that: In step S2, the single-frame cross-scale feature fusion module is input to a low-pass filter. right Upsampling results and high-pass filter right The high-frequency enhancement results, output The expression is: in, Main backbone network Shallow layer features; Main backbone network Deep layer features This is a cross-scale fusion feature.

3. The method for detecting breast lesions using ultrasound video fusion with spatiotemporal context information according to claim 1, characterized in that: In step S2, constructing a low-pass filter based on shallow features specifically includes: Shallow features of the input Perform 3×3 convolution dimensionality reduction to obtain spatially varied filter weights. ; Using pixel reverse washing The height and width are halved, the channels are enlarged, and the channels are split into four groups, each group being a spatial variation filter. , For each group of filter weights Normalization is performed to obtain a low-pass filter. ; Utilizing each set of low-pass filters for deep features Weighted filtering is performed to obtain the corresponding low-pass features. Then, each group of low-pass features is rearranged through pixel shuffling to form a feature that is upsampled twice as the output of the dynamic low-pass filter.

4. The method for detecting breast lesions using ultrasound video fusion with spatiotemporal context information according to claim 1, characterized in that: In step S2, the high-pass filter is constructed based on shallow features, specifically including: processing the input shallow features... 3×3 convolution dimensionality reduction is performed to obtain spatially varying filter weights, which are then normalized and inverted to generate a high-pass filter. This high-pass filter is then used to refine shallow features. Convolution is performed to enhance high-frequency edge components, resulting in a high-pass filter output.

5. The method for detecting breast lesions using ultrasound video fusion with spatiotemporal context information according to claim 1, characterized in that: Step S3 is as follows: The confidence score and classification score are multiplied together via broadcasting to obtain a comprehensive probability score, which is then set to a preset threshold. Perform masking filtering; based on the comprehensive probability scores, use the TopK function to select the top K high-quality candidate features of the current batch from both the masked candidate features and the high-quality candidate features stored in the candidate feature memory. and the top K historical high-quality candidate features ;Will and After linear layer mapping, the aggregated features are obtained by inputting the cross-batch feature aggregation module; Combine the aggregated features with the high-quality candidate features of the current batch. The elements are added one by one to obtain cross-batch enhanced classification features, and the candidate feature memory is updated with these cross-batch enhanced classification features. The original historical high-quality candidate features are replaced with the high-quality candidate features of the current batch.

6. The method for detecting breast lesions using ultrasound video fusion with spatiotemporal context information according to claim 1, characterized in that: In step S3, the capacity of the candidate feature memory is set to... ,in When updating the candidate feature memory, only high-quality features from the current batch are written, while low-quality features are discarded, ensuring that the current batch can utilize at most the previous batches. Batch time sequence characteristics information.

7. The method for detecting breast lesions using ultrasound video fusion with spatiotemporal context information according to claim 1, characterized in that: Step S4 specifically includes: The same time segment Frame, number The cross-batch enhanced classification features of the frame are denoted as The corresponding regression feature is denoted as ,in Enhance classification features across batches for each frame. With regression features The query vector is obtained through linear mapping respectively. , and key , The classification attention weight matrix is ​​calculated by scaling the dot product attention. and regression attention weight matrix ; Obtain the confidence score for each candidate feature, including the classification score and the IoU score, and construct the classification weight matrix through a broadcast mechanism. and regression weight matrix The confidence score is multiplied element-wise by the attention weight, and then the weighted attention distribution is obtained by applying softmax. Weighted features of the classification weight matrix Weighted characteristics of the regression weight matrix After addition and the shared value matrix By concatenating the series, the fusion features are obtained. : For the value matrix Calculate the similarity matrix after layer normalization. Select those with similarity exceeding the threshold The reference terms are average pooled, and the average pooled features are then compared with... The data are concatenated and then linearly projected to obtain cross-frame enhanced classification features.

8. A method for detecting breast lesions using ultrasound video that integrates spatiotemporal contextual information, characterized in that: In step S1, the total loss function used to train the detection network is... for: in, For regression loss; Loss of confidence in the target; This is the initial classification loss; Enhance the classification loss corresponding to the classification features across frames; and These are the weighting coefficients.

9. An electronic device, characterized in that, include: The invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements a method for detecting breast lesions using ultrasound video that integrates spatiotemporal context information as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program causes the computer to execute a breast ultrasound video lesion detection method that integrates spatiotemporal context information as described in any one of claims 1 to 7.

Images