Sparse memory temporal fusion method and device for ultrasound image lesion detection

Abstract

The application discloses a sparse memory time sequence fusion method for lesion detection of an ultrasonic image, and comprises the following steps: acquiring a continuous dynamic ultrasonic video frame sequence, wherein the video frame sequence comprises a plurality of ultrasonic images; extracting global anatomical features and local lesion features of each ultrasonic image; performing feature fusion projection on the global anatomical features and the local lesion features to generate current frame fusion features of a current frame ultrasonic image; evaluating the current frame fusion features to obtain an evaluation result, and updating a sparse memory feature library according to the evaluation result; and performing dynamic fusion on the current frame fusion features and historical memory features in the sparse memory feature library to obtain spatio-temporal joint enhanced features. The method can improve the stability and precision of ultrasonic video stream detection of parts such as a breast, an axillary lymph node, an abdomen and a neck.

Description

Technical Field

[0001] This invention relates to the field of ultrasound image processing technology, and in particular to a sparse memory temporal fusion method, apparatus, storage medium, and computer device for lesion detection in ultrasound images. Background Technology

[0002] In real-time breast ultrasound video stream lesion detection scenarios, traditional static image detection models only perform independent reasoning on single-frame ultrasound images, completely ignoring the inter-frame temporal correlation information of the ultrasound scan video. This results in problems such as feature redundancy, temporal discontinuity, missed detection of small lesions, and severe inter-frame detection jitter. Clinical breast ultrasound scans are a continuous dynamic video acquisition process. Adjacent frames have a high degree of spatial correlation and temporal continuity. The location, morphology, and texture features of lesions change slowly in consecutive frames. Independent detection of a single frame will lose the inter-frame contextual temporal features, making it impossible to use the information from previous and subsequent frames to assist in the identification of lesions in the current frame. This can easily lead to missed detection of transient microcalcifications and transient occult lesions. Meanwhile, traditional temporal fusion methods often use full-frame feature stacking fusion, which has the drawbacks of large feature parameters, a lot of redundant information, and slow inference speed, making it unsuitable for the low-latency requirements of real-time ultrasound scanning. Moreover, existing temporal fusion mechanisms lack the ability to differentiate between long and short distance features, cannot take into account both the semantics of the global scanning scene and the detailed features of local lesions, and do not have a dedicated memory update and sparse filtering mechanism. Redundant temporal features continuously interfere with model inference, resulting in poor stability and low accuracy of video stream detection. Summary of the Invention

[0003] To address the aforementioned issues, this application proposes a sparse memory temporal fusion method, apparatus, storage medium, and computer equipment for ultrasound image lesion detection, which can improve the stability and accuracy of ultrasound video stream detection in areas such as breast, axillary lymph nodes, abdomen, and neck.

[0004] This application proposes a sparse memory temporal fusion method for lesion detection in ultrasound images, including:

[0005] Acquire a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images;

[0006] Extract the global anatomical features and local lesion features of each frame of the ultrasound image;

[0007] The global anatomical features and the local lesion features are fused and projected to generate the current frame fusion features of the current frame ultrasound image;

[0008] The fusion features of the current frame are evaluated to obtain the evaluation results, and the sparse memory feature library is updated based on the evaluation results.

[0009] The current frame fusion feature is dynamically fused with the historical memory feature in the sparse memory feature library to obtain the spatiotemporal joint enhancement feature.

[0010] In some embodiments, the step of extracting global anatomical features and local lesion features of each frame of the ultrasound image includes:

[0011] Global mean pooling is performed on the current frame ultrasound image to reduce and compress the spatial dimension, thereby obtaining the global anatomical features;

[0012] The current frame ultrasound image is masked to obtain the region of interest (ROI) features of the lesion. The ROI features are then subjected to local adaptive average pooling to obtain the local lesion features.

[0013] In some embodiments, feature fusion projection of the global anatomical features and the local lesion features based on a dual-branch compression mechanism includes:

[0014] Calculate the feature energy difference between the global anatomical features and the local lesion features, and obtain the dynamic fusion weight coefficient based on the feature energy difference;

[0015] Based on the dynamic fusion weight coefficients, the global anatomical features and the local lesion features are interactively fused to obtain the current frame fusion features.

[0016] In some embodiments, the current frame fusion features are evaluated to obtain an evaluation result, and then updated based on the evaluation result, including:

[0017] The fused features of the current frame are evaluated to obtain a feature importance score;

[0018] When the feature importance score is greater than the dynamic score threshold, the current fused feature is saved into the sparse memory feature library and the sparse memory feature library is updated.

[0019] In some embodiments, the fused features of the current frame are evaluated based on a frame feature importance scoring model to obtain a feature importance score. The frame feature importance scoring model is as follows:

[0020] ,

[0021] in, For the first Importance score of the current frame fused features For the first Frame number Channel feature map; This is a saliency mask for lesions, with lesion areas assigned a value of 1 and pure background areas assigned a value of 0. These represent the number of feature channels, the height of the feature map, and the width, respectively.

[0022] In some embodiments, the current frame fusion feature is dynamically fused with the historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhanced features, including:

[0023] Calculate the attention score between the current frame fusion feature and the historical memory feature in the sparse memory feature library;

[0024] Based on the attention score, the historical memory features are dynamically retrieved to obtain the historical context features with the highest confidence that are fused with the current frame features;

[0025] The retrieved historical context features are fused with the current frame fusion features to obtain spatiotemporal joint enhanced features.

[0026] In some embodiments, it also includes:

[0027] The spatiotemporal joint enhancement features are input into a pre-trained lesion detection model to obtain lesion detection results.

[0028] In some embodiments, a sparse memory temporal fusion apparatus for lesion detection in ultrasound images is also provided, comprising:

[0029] A video frame acquisition module is used to acquire a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images;

[0030] The feature extraction module is used to extract the global anatomical features and local lesion features of each frame of the ultrasound image;

[0031] The feature fusion module is used to perform feature fusion projection of the global anatomical features and the local lesion features to generate the current frame fusion feature of the current frame ultrasound image;

[0032] The feature update module is used to evaluate the fused features of the current frame, obtain the evaluation result, and update the sparse memory feature library according to the evaluation result;

[0033] The feature enhancement module is used to dynamically fuse the current frame fused features with the historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhanced features.

[0034] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described above.

[0035] In some embodiments, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the methods described above.

[0036] The method described in the above embodiments acquires a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images; extracts global anatomical features and local lesion features from each ultrasound image frame; fuses and projects the global anatomical features and local lesion features to generate current frame fusion features for the current ultrasound image; evaluates the current frame fusion features to obtain evaluation results, and updates the sparse memory feature library based on the evaluation results; and dynamically fuses the current frame fusion features with historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhancement features. This method completely solves the technical problems of missing temporal information, feature redundancy, inter-frame jitter, and missed detection of small lesions in traditional breast ultrasound video stream detection. It also enables on-demand retention of temporal features, significantly reduces model computation, ensures low-latency inference requirements for real-time ultrasound scanning, and improves the efficiency and accuracy of ultrasound scanning of the breast, axillary lymph nodes, abdomen, and neck.

[0037] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0038] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0039] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0040] Figure 1 This is a flowchart of the sparse memory temporal fusion method for lesion detection in ultrasound images provided in the embodiments of this application;

[0041] Figure 2 This is a schematic diagram showing the results of the sparse memory temporal fusion method for detecting lesions in ultrasound images provided in the embodiments of this application.

[0042] Figure 3 This is a structural block diagram of the sparse memory temporal fusion device for detecting lesions in ultrasound images provided in the embodiments of this application. Detailed Implementation

[0043] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0044] Existing technologies cannot achieve sparse, efficient, accurate, and stable temporal feature fusion to address the temporal characteristics and real-time requirements of dynamic video stream detection of breast ultrasound.

[0045] To solve the above problems, refer to Figure 1 This application proposes a sparse memory temporal fusion method for lesion detection in ultrasound images, comprising:

[0046] Step 101: Obtain a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images.

[0047] In this embodiment, an ultrasound probe is used to continuously acquire and arrange 2D / 3D ultrasound images in chronological order to form a continuous dynamic video (usually stored as a multi-frame DICOM or AVI file) to capture organ changes (such as changes in organs in the breast, axillary lymph nodes, abdomen, neck, etc.).

[0048] Step 102: Extract the global anatomical features and local lesion features of each frame of the ultrasound image.

[0049] In this embodiment, a global-local dual-branch parallel feature extraction and compression structure is adopted to extract the global anatomical features and local lesion features of each frame of the ultrasound image. This can be specifically adapted to the multi-scale feature requirements of breast ultrasound images and solve the defect that a single branch cannot take into account both global anatomical features and local lesion details.

[0050] In the global-local dual-branch parallel feature extraction and compression structure, the global branch is responsible for extracting the global anatomical features of each frame of breast ultrasound image, capturing the overall glandular layering, overall lesion distribution, and global background structure information. Through global mean pooling and dimensionality compression strategies, redundant global semantics are eliminated, while preserving the prior features of the overall breast anatomy, providing global spatial constraints for temporal fusion. The local branch is responsible for focusing on local lesion areas in each frame of breast ultrasound image, extracting fine-grained features such as breast microcalcifications, spiculated edges, and local echo differences. Through local adaptive sampling compression, core lesion details are preserved, while invalid details such as local speckle noise and scanning artifacts are filtered out.

[0051] Step 103: Perform feature fusion projection on the global anatomical features and the local lesion features to generate the current frame fusion features of the current frame ultrasound image.

[0052] In this embodiment, a feature fusion projection mechanism is adopted to aggregate global anatomical features and local lesion features, and to fuse the current frame features of the current frame ultrasound image, thereby solving the problem that a single feature dimension cannot fully represent the ultrasound of organs such as the breast.

[0053] Step 104: Evaluate the fusion features of the current frame to obtain the evaluation results, and update the sparse memory feature library based on the evaluation results.

[0054] In ultrasound video streams, a large number of frames may contain meaningless background (such as subcutaneous fat or muscle layers) or noise (such as speckle noise or probe artifacts). If all frames are stored without filtering, the temporal feature memory will be filled with redundant information, and the inference speed will decrease.

[0055] In this embodiment, based on the sparse memory mechanism of the spatial perceiver, and considering the high inter-frame redundancy and sparse effective lesion features of breast ultrasound video streams, the importance of the fused features of the current frame is scored. This involves evaluating the proportion of effective lesion information in each ultrasound image frame to determine whether the fused features of that frame are worth storing in the temporal feature memory. The temporal feature memory stores the effective fused features of each ultrasound image frame after scoring and filtering.

[0056] The scoring mechanism distinguishes between valid and invalid frames: a high score indicates that the frame contains key anatomical features such as breast lesions or microcalcifications and must be retained; a low score indicates that the frame is pure background or noise and should be discarded. This approach, which only performs complex temporal fusion calculations on high-scoring frames, significantly reduces the computational load of video stream processing, optimizes computational resources, and ensures real-time performance (<20ms latency).

[0057] Step 105: Dynamically fuse the current frame fusion feature with the historical memory feature in the sparse memory feature library to obtain spatiotemporal joint enhancement features.

[0058] In this embodiment, an adaptive gating interaction strategy is employed to dynamically fuse the current frame fusion features with the historical memory features in the sparse memory feature library, resulting in spatiotemporal joint enhanced features. The historical memory features are the effective fusion features retained after scoring and filtering of previous frame ultrasound images stored in the above steps.

[0059] The adaptive gating interaction strategy, employing learnable gating coefficients and a cross-attention mechanism, first uses the current frame's fused features as the query and historical memory features from a sparse memory bank as the key and value, calculating an attention score to dynamically retrieve historical memory information related to the current anatomical structure. Then, a gating function dynamically calculates the confidence weights between the current frame's fused features and historical memory features. If the current frame's ultrasound image is clear (high energy), the gating coefficients favor the current frame's fused features; if the current frame's ultrasound image is blurred due to acoustic occlusion or probe movement, the gating coefficients automatically increase the weight of historical memory features, achieving automatic completion of historical memory information and obtaining spatiotemporally enhanced features.

[0060] This adaptive strategy significantly improves the smoothness and anti-interference ability of video stream detection while ensuring real-time performance, thus ensuring the efficiency and stability of target locking during ultrasonic scanning.

[0061] The method described in the above embodiments acquires a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images; extracts global anatomical features and local lesion features from each ultrasound image frame; fuses and projects the global anatomical features and local lesion features to generate current frame fusion features for the current ultrasound image; evaluates the current frame fusion features to obtain evaluation results, and updates the sparse memory feature library based on the evaluation results; and dynamically fuses the current frame fusion features with historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhancement features. This method completely solves the technical problems of missing temporal information, feature redundancy, inter-frame jitter, and missed detection of small lesions in traditional breast ultrasound video stream detection. It also enables on-demand retention of temporal features, significantly reduces model computation, ensures low-latency inference requirements for real-time ultrasound scanning, and improves the efficiency and accuracy of ultrasound scanning of the breast, axillary lymph nodes, abdomen, and neck.

[0062] In some embodiments, global anatomical features and local lesion features of each frame of the ultrasound image are extracted, including:

[0063] (1) Perform global mean pooling on the current frame ultrasound image to reduce the spatial dimension and compress it to obtain global anatomical features;

[0064] In this embodiment, the global branch performs dimensionality reduction and compression on the current frame ultrasound image using global mean pooling, preserving the overall anatomical semantics of the breast. The compression formula is as follows:

[0065] ,

[0066] In the formula, Features of the current frame ultrasound image This is the compressed global anatomical feature vector. Let H be the feature vector of the current frame ultrasound image at spatial location (h, w). H and W are the height and width of the current frame ultrasound image, respectively.

[0067] (2) Mask the current frame ultrasound image to obtain the region of interest (ROI) features of the lesion, and perform local adaptive average pooling on the ROI features to obtain local lesion features.

[0068] In this embodiment, lesion saliency masking is first used. The current frame of ultrasound image is masked to remove the background area and retain only the region of interest (ROI) features. Then, adaptive local average pooling is performed on the masked ROI features to extract local lesion features that focus on the core details of the lesion.

[0069] Local branch adaptive focusing lesion region sampling compression preserves subtle lesion features; the compression formula is:

[0070] ,

[0071] in, This represents the compressed feature vector of the local lesion. This represents the current frame of the ultrasound image; Element-by-element multiplication; This is a saliency mask used to highlight the lesion area and filter background noise; To achieve adaptive local average pooling, background noise is filtered out while preserving lesion details.

[0072] In some embodiments, feature fusion projection of the global anatomical features and the local lesion features includes:

[0073] (1) Calculate the feature energy difference between global anatomical features and local lesion features, and obtain the dynamic fusion weight coefficient based on the feature energy difference. :

[0074] ,

[0075] ,

[0076] In the formula, This represents the dynamic fusion weight coefficient, with a value range of... , This represents the local feature energy, such as the feature energy of a local lesion region in a breast ultrasound image, which is the local lesion feature vector. The L2 norm, i.e. It focuses on characterizing the details of lesions, including fine-grained pathological features such as microcalcifications, marginal spiculations, and local shadows; This represents the global anatomical feature energy, such as the global anatomical feature energy of a whole frame of breast ultrasound image, which is the global anatomical feature vector. The L2 norm, i.e. It mainly represents background information such as the overall anatomical structure of the breast, organ outline, and global grayscale distribution.

[0077] (2) Based on the dynamic fusion weight coefficients, the interaction between global anatomical features and local lesion features is completed to obtain the current frame fusion features (memory tensor):

[0078] ,

[0079] The core logic of this adaptive fusion mechanism is that when the energy of local lesion features is higher, the weighting coefficient is increased. Approaching a value of 1, it automatically increases the weight of local lesion features, maximizing the preservation of fine-grained pathological information such as microcalcifications and subtle lesion edges, thus adapting to scenarios involving the detection of small lesions; however, when the overall anatomical background is complex and there is significant tissue interference, it is suitable for such scenarios. The increased weighting automatically reduces local weighting and increases global weighting, relying on global anatomical features to constrain lesion location, ensuring that lesion positioning does not deviate from the boundaries of breast organs, effectively improving detection stability and accuracy in complex scenarios.

[0080] In some embodiments, the current frame fusion features are evaluated to obtain an evaluation result, and then updated based on the evaluation result, including:

[0081] (1) Evaluate the fusion features of the current frame to obtain a feature importance score.

[0082] In this embodiment, the fused features of the current frame are scored based on a frame feature importance scoring model to quantify the proportion of effective lesion information in a single frame ultrasound image. The formula for the scoring model is as follows:

[0083] ,

[0084] in, For the first Importance score of the current frame fused features For the first Frame number Channel feature map; This is a saliency mask for lesions, with lesion areas assigned a value of 1 and pure background areas assigned a value of 0. These represent the number of feature channels, the height of the feature map, and the width. A higher score indicates that the frame contains more effective information, such as breast lesions and microcalcifications.

[0085] (2) When the feature importance score is greater than the dynamic score threshold, the current fused feature is saved into the sparse memory feature library and the sparse memory feature library is updated.

[0086] After scoring, feature selection is completed based on the set dynamic score threshold (default τ=0.5 for breast ultrasound): If If the fusion feature of the frame is greater than or equal to τ, then the fused feature of that frame is determined to be a valid feature and included in the sparse memory feature library; if If the value is less than τ, it is considered a redundant background feature, and an attenuation coefficient is introduced. =0.1, which greatly weakens its interference and does not include it in the memory bank.

[0087] ,

[0088] In the formula, For sparse memory feature database, This is the redundancy feature attenuation coefficient, with a default value of 0.1, which significantly weakens the interference of invalid time-series features in the pure background. A sliding window update mechanism is also employed, with a window length... Frame, iteratively update historical memory features:

[0089] ,

[0090] in, The memory retention coefficient has a default value of 0.7, which takes into account both historical effective temporal features and current frame fusion features. This represents the effective fusion features selected at the current moment; This indicates the updated state of the sparse memory feature library; This represents the state of the sparse memory feature library at the previous time step.

[0091] The above method addresses the characteristics of high inter-frame redundancy and sparse effective disease features in breast ultrasound video streams. It enables selective memorization, scoring, and dynamic updating of temporal features, and completes the retention of effective features and the removal of redundant features through a standardized mathematical model.

[0092] In some embodiments, the current frame fusion feature is dynamically fused with the historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhanced features, including:

[0093] (1) Calculate the attention score between the current frame fusion feature and the historical memory feature in the sparse memory feature library.

[0094] Fuse features of the current frame A query vector is generated through a linear projection layer. Historical tokens in the sparse memory are then used to generate key and value vectors through linear projection. The attention score is calculated as follows:

[0095] ,

[0096] In the formula, Indicates features fused from the current frame Through a learnable linear projection matrix The generated query vector, i.e. This represents "what historical information you want to find in the current frame". This represents each historical memory token in the sparse memory bank through a learnable linear projection matrix. The generated key vector, i.e. It represents "what is included in historical memory". This represents each historical memory feature in the sparse memory bank through a learnable linear projection matrix. The generated value vector, i.e. It represents "the actual characteristic values ​​stored in historical memory". Represents the key vector The dimension (usually 64 or 128), divided by This is to prevent the dot product result from becoming too large, causing the Softmax function to enter the gradient saturation region (i.e., the gradient vanishing problem), and to ensure stable training. This represents the matrix multiplication (dot product similarity) between the query vector and the key vector. The result is an attention score matrix that reflects the relevance of the current frame to each historical memory frame. The higher the score, the more similar the current frame is to the anatomical structure of that historical frame. This means normalizing the attention score matrix to a probability distribution function, ensuring that the sum of the weights of all historical memories is 1.

[0097] (2) Based on the attention score, the historical memory features are dynamically retrieved to obtain the historical context features with the highest confidence in the fused features of the current frame.

[0098] The confidence weights of the current frame's fused features and historical memory features are dynamically calculated using gating coefficients. The formula for calculating the gating coefficients is as follows:

[0099] ,

[0100] In the formula, This represents the gating coefficient, and its value range is... Scalar value. The closer it is to 1, the more the model trusts the fused features of the current frame; The closer it is to 0, the more the model relies on historical memory features. The Sigmoid activation function maps any real number to... Interval. Represents the learnable gated weight matrix, with dimension . ,in is the dimension of the feature energy vector. This matrix is ​​automatically optimized during training through backpropagation. This represents the concatenation vector, which combines the feature energy of the current frame. Energy of historical memory characteristics They are pieced together along the dimensional direction. Feature energy. Defined as the L2 norm of the eigenvectors, i.e.: The higher the energy, the clearer and more information-rich the image frame. This represents a learnable gating bias vector with dimension 1, used as a baseline value for adjusting the gating coefficients.

[0101] If the current frame ultrasound image is clear ( (High energy), gating coefficient If the inference is biased towards 1, then it mainly relies on the features of the current frame for reasoning; if the current frame ultrasound image is blurred due to acoustic shadowing or probe movement ( (Low energy), gating coefficient The model automatically approaches 0, increases the weight of historical memory, and achieves automatic completion of historical memory features.

[0102] (3) The retrieved historical context features are fused with the current frame fusion features to obtain spatiotemporal joint enhancement features.

[0103] Fusion, also known as residual connection or stitching, enables a "fill-in-the-blank" function: if the ultrasound image in the current frame is blurred due to sound occlusion or probe movement, clear features will be automatically extracted from historical memory features and added to the current frame for dynamic fusion, resulting in spatiotemporal joint enhanced features, thereby significantly improving the smoothness and anti-interference ability of video stream detection.

[0104] Furthermore, in some embodiments, spatiotemporal joint enhancement features are input into a pre-trained lesion detection model to obtain lesion detection results.

[0105] In this embodiment, the fused spatiotemporal joint enhancement features are input into the detection head to complete the real-time detection, localization, and classification output of lesions. For example, taking a continuous dynamic video stream frame sequence of breast ultrasound as an example, the sparse memory temporal fusion method for lesion detection in ultrasound images according to this application embodiment yields the following breast ultrasound detection results: Figure 2As shown, before using this method, the inter-frame detection jitter was significant, resulting in large confidence fluctuations and inaccurate detection. After using this method, the detection effect is as follows: Figure 2 As shown in (c), the stable positions between frames are consistent, and the confidence level is stable, indicating accurate detection results. The method adopted in this application embodiment aligns with the clinical characteristics of breast ultrasound video detection, effectively solving the problems of feature redundancy, temporal memory confusion, and lack of distinction between minor and major features in traditional temporal models. The final model exhibits smooth and stable inter-frame detection results in dynamic breast ultrasound video stream detection, effectively reducing the false positive rate of microcalcifications and occult lesions, as well as the false positive rate in complex backgrounds. This significantly improves the accuracy and robustness of real-time breast ultrasound screening and can be widely adapted to clinical dynamic breast ultrasound scanning and real-time terminal detection deployment scenarios.

[0106] In some embodiments, such as Figure 3 As shown, a sparse memory temporal fusion device for lesion detection in ultrasound images is also proposed, comprising:

[0107] The video frame acquisition module 301 is used to acquire a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images.

[0108] The feature extraction module 302 is used to extract the global anatomical features and local lesion features of each frame of the ultrasound image;

[0109] Feature fusion module 303 is used to perform feature fusion projection of the global anatomical features and the local lesion features to generate the current frame fusion features of the current frame ultrasound image;

[0110] The feature update module 304 is used to evaluate the fused features of the current frame, obtain the evaluation result, and update the sparse memory feature library according to the evaluation result.

[0111] The feature enhancement module 305 is used to dynamically fuse the current frame fused features with the historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhanced features.

[0112] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in the above embodiments.

[0113] In some embodiments, a computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the above embodiments.

[0114] The principles by which the above-mentioned devices, media, and related equipment in the embodiments of the present invention solve the problem are similar to those of the aforementioned methods. Therefore, their implementation can refer to the implementation of the aforementioned methods, and repeated details will not be repeated.

[0115] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0116] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of equivalents of this invention, this invention is also intended to include these modifications and variations.

Claims

1. A sparse memory temporal fusion method for lesion detection in ultrasound images, characterized in that, include: Acquire a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images; Extract the global anatomical features and local lesion features of each frame of the ultrasound image; The global anatomical features and the local lesion features are fused and projected to generate the current frame fusion features of the current frame ultrasound image; The fusion features of the current frame are evaluated to obtain the evaluation results, and the sparse memory feature library is updated based on the evaluation results. The current frame fusion feature is dynamically fused with the historical memory feature in the sparse memory feature library to obtain the spatiotemporal joint enhancement feature.

2. The method as described in claim 1, characterized in that, The steps for extracting global anatomical features and local lesion features from each frame of the ultrasound image include: Global mean pooling is performed on the current frame ultrasound image to reduce and compress the spatial dimension, thereby obtaining the global anatomical features; The current frame ultrasound image is masked to obtain the region of interest (ROI) features of the lesion. The ROI features are then subjected to local adaptive average pooling to obtain the local lesion features.

3. The method as described in claim 1, characterized in that, Based on a dual-branch compression mechanism, the global anatomical features and the local lesion features are fused and projected, including: Calculate the feature energy difference between the global anatomical features and the local lesion features, and obtain the dynamic fusion weight coefficient based on the feature energy difference; Based on the dynamic fusion weight coefficients, the global anatomical features and the local lesion features are interactively fused to obtain the current frame fusion features.

4. The method as described in claim 1, characterized in that, The current frame fusion features are evaluated to obtain an evaluation result, and then updated based on the evaluation result, including: The fused features of the current frame are evaluated to obtain a feature importance score; When the feature importance score is greater than the dynamic score threshold, the current fused feature is saved into the sparse memory feature library and the sparse memory feature library is updated.

5. The method as described in claim 4, characterized in that, The fused features of the current frame are evaluated based on a frame feature importance scoring model to obtain a feature importance score. The frame feature importance scoring model is as follows: ， in, For the first Importance score of the fused features of the current frame. For the first Frame number Channel feature map; This is a saliency mask for lesions, with lesion areas assigned a value of 1 and pure background areas assigned a value of 0. These represent the number of feature channels, the height of the feature map, and the width, respectively.

6. The method as described in claim 1, characterized in that, The current frame fusion features are dynamically fused with the historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhanced features, including: Calculate the attention score between the current frame fusion feature and the historical memory feature in the sparse memory feature library; Based on the attention score, the historical memory features are dynamically retrieved to obtain the historical context features with the highest confidence that are fused with the current frame features; The retrieved historical context features are fused with the current frame fusion features to obtain spatiotemporal joint enhanced features.

7. The method as described in claim 1, characterized in that, Also includes: The spatiotemporal joint enhancement features are input into a pre-trained lesion detection model to obtain lesion detection results.

8. A sparse memory temporal fusion device for lesion detection in ultrasound images, characterized in that, include: A video frame acquisition module is used to acquire a continuous dynamic ultrasound video frame sequence, wherein the video frame sequence contains multiple ultrasound images; The feature extraction module is used to extract the global anatomical features and local lesion features of each frame of the ultrasound image; The feature fusion module is used to perform feature fusion projection of the global anatomical features and the local lesion features to generate the current frame fusion feature of the current frame ultrasound image; The feature update module is used to evaluate the fused features of the current frame, obtain the evaluation result, and update the sparse memory feature library according to the evaluation result; The feature enhancement module is used to dynamically fuse the current frame fused features with the historical memory features in the sparse memory feature library to obtain spatiotemporal joint enhanced features.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

Images