Breast ultrasound video processing method, device, equipment, medium and product

Abstract

The application provides a breast ultrasound video processing method, device, equipment, medium and product, relates to the technical field of medical auxiliary diagnosis, and the method comprises the following steps: inputting a breast ultrasound video into a lesion analysis model to obtain a lesion benign and malignant probability and a key frame index; the model comprises a hierarchical mixed expert layer, which is used for independently extracting static lesion features of each frame, dynamic time sequence features of the breast ultrasound video and general features of all frames respectively; the model further comprises an output module, which obtains the lesion benign and malignant probability based on a global representation of the breast ultrasound video, and determines the key frame index based on an importance score of each frame; the global representation is determined based on the static lesion features, the dynamic time sequence features and the general features, and the importance score is determined based on static routing weights and dynamic routing weights. The application can synchronously output the lesion benign and malignant probability and the key frame index, can provide assistance for breast lesion benign and malignant diagnosis and key frame selection of an ultrasound doctor, and has application potential for improving clinical efficiency.

Description

Technical Field

[0001] This application relates to the field of medical auxiliary diagnostic technology, and more specifically, to a method, apparatus, equipment, medium, and product for processing breast ultrasound video. Background Technology

[0002] Early and accurate identification and timely treatment of breast cancer help improve patient prognosis. Breast ultrasound has advantages such as being non-invasive, real-time, and cost-effective, and is therefore widely used in the early screening and diagnosis of breast cancer. However, the interpretation of breast ultrasound images largely depends on the clinical experience of the sonographer, especially for cases with speckle noise, blurred lesion boundaries, and limited spatial resolution. This leads to problems such as significant subjective differences, low diagnostic efficiency, and a shortage of experienced sonographers. Early AI-assisted breast ultrasound diagnostic methods mostly relied on static ultrasound images, typically extracting static lesion features from keyframes annotated by the physician. These methods do not fully utilize the dynamic temporal features between consecutive video frames during dynamic scanning. To utilize the temporal information in breast ultrasound videos, existing technologies have proposed various methods for keyframe selection and lesion classification based on breast ultrasound videos. For example, keyframes are extracted at fixed one-second intervals, and then the lesion classification results are output; or keyframe sequences are selected based on the lesion area estimated by a segmentation model, followed by multi-view feature fusion, and then the lesion classification results are output.

[0003] While the methods described above can process breast ultrasound videos for lesion analysis, most employ a two-stage framework of "selecting keyframes first, then classifying lesions as benign or malignant," with the keyframe selection and lesion classification stages relatively independent. On one hand, using fixed time intervals for keyframe selection may not adequately adapt to different scanning speeds and lesion appearance times. Furthermore, selecting keyframes based on lesion segmentation results makes the selection dependent on the accuracy of the upstream segmentation model. When lesion segmentation results are inaccurate, especially when the segmentation result of the frame with the largest lesion area deviates, the keyframe sequence selected using that frame as an anchor point may also shift, affecting the downstream classification of lesions as benign or malignant. Thus, keyframe selection errors propagate downstream, limiting overall analytical performance. On the other hand, this two-stage processing method differs somewhat from the clinical workflow where sonographers complete lesion benign / malignant diagnosis and select keyframes as diagnostic criteria during dynamic scanning.

[0004] Breast ultrasound video diagnosis requires models to simultaneously model static lesion features and dynamic temporal features of continuous video frames. Traditional hybrid expert networks typically employ a globally shared expert pool, lacking clear division of labor among experts. This can easily lead to overlapping expert functions and confusion or even conflicting knowledge, limiting the model's feature representation and pattern discrimination capabilities. Furthermore, existing breast ultrasound video diagnosis methods primarily output lesion benign / malignant classification results, rarely simultaneously outputting keyframe indices related to the classification results during the same processing. This lack of interpretability makes it difficult to provide intuitive diagnostic evidence, limiting the model's application and promotion in clinical practice. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a method, apparatus, device, medium, and product for processing breast ultrasound videos. It designs a hierarchical, hybrid expert-led collaborative modeling approach to static lesion features, dynamic temporal features, and general features in breast ultrasound videos. This improves the model's feature representation capabilities and its ability to distinguish between benign and malignant lesions. Furthermore, within a unified, single-stage breast ultrasound video processing framework, it simultaneously outputs the probability of benign or malignant lesions and keyframe indexes. This approach more closely aligns with the clinical workflow of ultrasound physicians who diagnose benign or malignant lesions and select keyframes as diagnostic criteria during dynamic scanning.

[0006] In a first aspect, this application provides a method for processing breast ultrasound video, including: Obtain breast ultrasound video; The breast ultrasound video is input into the lesion analysis model to obtain the lesion analysis results output by the lesion analysis model. The lesion analysis results include the probability of benign or malignant lesions and the keyframe index of the breast ultrasound video. The lesion analysis model includes a hierarchical hybrid expert layer, which comprises: A single-frame static expert is used to extract static lesion features from each frame of the breast ultrasound video. A temporal dynamics expert is used to extract the dynamic temporal features of the breast ultrasound video based on the context feature block formed by the current frame and the adjacent frames of the current frame. Shared experts are used to extract common features from all frames in the breast ultrasound video; The lesion analysis model further includes an output module, used to obtain the benign or malignant probability of the lesion based on the global representation of the breast ultrasound video, and to determine the keyframe index based on the importance score of each frame in the breast ultrasound video; the global representation is determined based on the static lesion features, the dynamic temporal features and the general features, and the importance score is determined based on the static routing weight of the single-frame static expert and the dynamic routing weight of the temporal dynamic expert.

[0007] In some embodiments, the lesion analysis model includes a hierarchical hybrid expert Transformer encoder, the hierarchical hybrid expert Transformer encoder including the hierarchical hybrid expert layer, and the lesion analysis model further includes an input module, the input module being used for: The breast ultrasound video is segmented into continuous video frames; Extract the spatial features of each frame in the continuous video frames; Add positional coding to the spatial features of each frame to obtain the initial feature sequence of the consecutive video frames; A category token is inserted before the sequence header of the initial feature sequence to obtain the feature sequence of the consecutive video frames; The feature sequence is input into the hierarchical hybrid expert Transformer encoder.

[0008] In some embodiments, the lesion analysis model further includes: A static routing network is used to activate a target single-frame static expert based on the features of the current frame and generate the static routing weights of the target single-frame static expert. A dynamic routing network is used to activate a target temporal dynamic expert based on the context feature block and generate the dynamic routing weights of the target temporal dynamic expert.

[0009] In some embodiments, the hierarchical hybrid expert Transformer encoder obtains the static lesion features, the dynamic temporal features, and the general features based on the feature sequence, and obtains the video feature representation of the breast ultrasound video based on the static lesion features, the dynamic temporal features, and the general features, and inputs the video feature representation to the output module; The output module includes a multi-layer fully connected classification head, and the process of obtaining the benign or malignant probability of the lesion based on the global representation of the breast ultrasound video includes: The output module uses the category token representation in the video feature representation as the global representation of the breast ultrasound video; The global representation is input into the multi-layer fully connected classification head to obtain the benign or malignant probability of the lesion output by the multi-layer fully connected classification head.

[0010] In some embodiments, determining the keyframe index based on the importance score of each frame in the breast ultrasound video includes: The static lesion saliency score for each frame is determined based on the static routing weights. The dynamic lesion salience score for each frame is determined based on the dynamic routing weights. The importance score for each frame is obtained by weighted fusion of the static lesion significance score and the dynamic lesion significance score; Keyframes are selected from all frames of the breast ultrasound video based on the importance score of each frame, and the keyframe index of the keyframes is generated.

[0011] In some embodiments, the training process of the lesion analysis model includes the following steps: Obtain a training dataset, which includes sample breast ultrasound videos and benign / malignant labels for the sample lesions corresponding to the sample breast ultrasound videos; The sample breast ultrasound video is input into the initial lesion analysis model to obtain the benign or malignant probability of the lesion output by the initial lesion analysis model. The initial lesion analysis model is the lesion analysis model to be trained. The classification cross-entropy loss of the initial lesion analysis model is determined based on the probability of benign or malignant lesions and the benign or malignant labels of the sample lesions; The first load balancing loss of the single-frame static expert is determined based on the static routing weights corresponding to the single-frame static expert during training, and the second load balancing loss of the time-series dynamic expert is determined based on the dynamic routing weights corresponding to the time-series dynamic expert. The parameters of the initial lesion analysis model are updated based on the classification cross-entropy loss, the first load balancing loss, and the second load balancing loss.

[0012] Secondly, this application provides a breast ultrasound video processing device, comprising: The acquisition module is used to acquire breast ultrasound videos; The processing module is used to input the breast ultrasound video into the lesion analysis model and obtain the lesion analysis results output by the lesion analysis model. The lesion analysis results include the probability of benign or malignant lesions and the keyframe index of the breast ultrasound video. The lesion analysis model includes a hierarchical hybrid expert layer, which comprises: A single-frame static expert is used to extract static lesion features from each frame of the breast ultrasound video. A temporal dynamics expert is used to extract the dynamic temporal features of the breast ultrasound video based on the context feature block formed by the current frame and the adjacent frames of the current frame. Shared experts are used to extract common features from all frames in the breast ultrasound video; The lesion analysis model further includes an output module, used to obtain the benign or malignant probability of the lesion based on the global representation of the breast ultrasound video, and to determine the keyframe index based on the importance score of each frame in the breast ultrasound video; the global representation is determined based on the static lesion features, the dynamic temporal features and the general features, and the importance score is determined based on the static routing weight of the single-frame static expert and the dynamic routing weight of the temporal dynamic expert.

[0013] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the above-described method when executing the program through the computer program.

[0014] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0015] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0016] The breast ultrasound video processing method, apparatus, device, medium, and product provided in this application's embodiments design single-frame static experts, temporal dynamic experts, and shared experts in a hierarchical hybrid expert layer. This achieves decoupled modeling of static lesion features and dynamic temporal features in breast ultrasound videos, avoiding the problem of inaccurate features caused by mixing the two types of features in related technologies. By synchronously outputting the benign and malignant probability of lesions and keyframe indexes through the lesion analysis model, it is not necessary to extract keyframes first and then classify them, eliminating the risk of upstream keyframe selection errors being transmitted downstream. It can accurately capture multidimensional lesion features in breast ultrasound videos, improve the accuracy of predicted benign and malignant probabilities of lesions, and output clinically interpretable keyframe indexes, which can assist doctors in making rapid and accurate diagnostic decisions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is one of the flowcharts illustrating the breast ultrasound video processing method provided in the embodiments of this application.

[0019] Figure 2This is a second schematic flowchart of the breast ultrasound video processing method provided in the embodiments of this application.

[0020] Figure 3 This is the third flowchart illustrating the breast ultrasound video processing method provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram showing the performance comparison results of the single-view classification model provided in the embodiments of this application.

[0022] Figure 5 This is a schematic diagram showing the performance comparison results of the dual-view classification model provided in the embodiments of this application.

[0023] Figure 6 ROC curve of the benign and malignant lesion classification model based on breast ultrasound video provided in the embodiments of this application.

[0024] Figure 7 PRC curve of the benign and malignant lesion classification model based on breast ultrasound video provided in the embodiments of this application.

[0025] Figure 8 The confusion matrix diagram of the single-view breast ultrasound video lesion classification model provided in the embodiments of this application.

[0026] Figure 9 The confusion matrix diagram of the dual-view breast ultrasound video lesion classification model provided in the embodiments of this application.

[0027] Figure 10 This is a schematic diagram of the structure of the breast ultrasound video processing device provided in the embodiments of this application.

[0028] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices.

[0031] The breast ultrasound video processing method provided in this application is applicable to terminals, which can be various electronic devices with a display screen and support web browsing, including servers, smartphones, tablets, laptops, and desktop computers.

[0032] Figure 1 This is one of the flowcharts illustrating the breast ultrasound video processing method provided in the embodiments of this application, such as... Figure 1 As shown, the method includes steps 110 and 120. These method steps are merely one possible implementation of this application.

[0033] Step 110: Obtain breast ultrasound video.

[0034] Specifically, the subject of the breast ultrasound video processing method provided in this application embodiment is a breast ultrasound video processing device, which can be a hardware device independently set in the terminal or a software program running in the terminal.

[0035] In practical applications, breast ultrasound videos are typically continuous video images recorded by doctors during clinical procedures when they perform dynamic scans of a patient's breast area. Breast ultrasound videos can be stored locally on the ultrasound device or remotely. The breast ultrasound video processing device reads the locally stored files to obtain the breast ultrasound video, or receives breast ultrasound videos actively transmitted by the ultrasound device.

[0036] Step 120: Input the breast ultrasound video into the lesion analysis model to obtain the lesion analysis results output by the lesion analysis model. The lesion analysis results include the probability of benign or malignant lesions and the keyframe index of the breast ultrasound video. The lesion analysis model includes a hierarchical hybrid expert layer, which comprises: Single-frame static expert, used to extract static lesion features from each frame of a breast ultrasound video; The temporal dynamics expert is used to extract dynamic temporal features of breast ultrasound videos based on the contextual feature blocks formed by the current frame and its neighboring frames. Shared experts are used to extract common features from all frames in breast ultrasound videos; The lesion analysis model also includes an output module, which is used to obtain the benign and malignant probabilities of lesions based on the global representation of breast ultrasound video, and to determine the key frame index based on the importance score of each frame in the breast ultrasound video; the global representation is determined based on static lesion features, dynamic temporal features and general features, and the importance score is determined based on the static routing weight of the single-frame static expert and the dynamic routing weight of the temporal dynamic expert.

[0037] Specifically, the lesion analysis model in this embodiment refers to a model built based on deep learning, used to analyze the nature of lesions based on breast ultrasound video and output the probability of lesion benignity or malignancy and keyframe index.

[0038] The probability of a lesion being benign or malignant is a value used to indicate whether a lesion in a breast ultrasound video is benign or malignant.

[0039] A keyframe index refers to the frame number of one or more frames in a breast ultrasound video that are identified as representative of the lesion analysis.

[0040] Static lesion features refer to the feature information extracted from a single frame of ultrasound image, reflecting the static attributes of the lesion such as morphology, boundary, and echo.

[0041] A context feature block is a feature combination composed of the current frame and its adjacent frames, used to extract dynamic temporal features.

[0042] Dynamic temporal features refer to the feature information extracted from multiple consecutive ultrasound images that reflects the changes in lesion morphology over time.

[0043] General features refer to basic image attribute features extracted from all frames that do not depend on specific lesion morphology or dynamic changes.

[0044] Global representation refers to the feature vector that is compressed and encoded from the entire breast ultrasound video content.

[0045] Importance score is a quantitative score that measures the contribution of each frame in a breast ultrasound video to the analysis of lesions.

[0046] Static route weights are used to select the active single-frame static expert from multiple single-frame static experts and assign weight values ​​to them based on their contribution.

[0047] Dynamic routing weights are used to select active time-series dynamic experts from multiple time-series dynamic experts and assign them weight values ​​based on their contribution.

[0048] The lesion analysis model in this embodiment includes an input module, a hierarchical hybrid expert Transformer encoder, and an output module.

[0049] In this embodiment, breast ultrasound video is input into the input module of the lesion analysis model. The input module performs frame segmentation and spatial feature extraction on the breast ultrasound video, and fuses location codes and category tokens to output a feature sequence. Then the feature sequence Input to hierarchical hybrid expert Transformer encoder In the middle, the hierarchical hybrid expert Transformer encoder For feature sequences Decoupled modeling of spatiotemporal features and deep feature extraction are performed to output video feature representation. The formula is shown below: ; ; in, For layered hybrid expert Transformer encoders The final layer feature sequence output after processing by stacked hierarchical hybrid expert Transformer modules; Represented by class tokens (cls); Corresponding to the first to the last frame of the breast ultrasound video. Frame feature representation; This represents the total number of frames in the breast ultrasound video.

[0050] The following section will explain in detail the internal structure and data processing flow of the hierarchical hybrid expert Transformer encoder. Figure 2 This is a second schematic flowchart of the breast ultrasound video processing method provided in the embodiments of this application, as shown below. Figure 2 As shown, the hierarchical hybrid expert Transformer encoder includes A stacked, hierarchical hybrid expert Transformer module.

[0051] In this embodiment, each hierarchical hybrid expert Transformer module includes not only standard multi-head self-attention (MHA) components, residual connection (Add) components, and layer normalization (LN) components, but also a hierarchical hybrid expert layer, abbreviated as BusMMoE layer.

[0052] Inside the hierarchical hybrid expert Transformer encoder, the first... The layered hybrid expert Transformer module will receive the first layer. The feature sequence output by the layered hybrid expert Transformer module Feature sequence After processing including multi-head self-attention computation by the multi-head self-attention component, residual connection by the residual connection component, and normalization by the layer normalization component, the feature sequence of this layer is output. For the first The processing procedure of the layered hybrid expert Transformer module is defined as follows: ; ; in, This is the intermediate feature sequence output after multi-head self-attention and residual connection; For layer normalization function; For multi-head self-attention computation function; For BusMMoE layer processing functions; A function to retrieve the current token context feature block.

[0053] In this embodiment, the BusMMoE layer is used to replace the standard feed-forward network (FFN) and is specifically designed to extract static lesion features, dynamic temporal features, and general features from breast ultrasound videos. Figure 3 This is the third flowchart illustrating the breast ultrasound video processing method provided in the embodiments of this application, as shown below. Figure 3 As shown, the BusMMoE layer includes a single-frame static expert and its corresponding static routing network, a temporal dynamic expert and its corresponding dynamic routing network, and a shared expert. The single-frame static expert, temporal dynamic expert, and shared expert all employ a standard feedforward neural network, i.e., FFN. Both the static routing network and the dynamic routing network use a Top-K strategy to activate the target single-frame static expert and the target temporal dynamic expert; in this embodiment, K is set to 1.

[0054] The feature sequence, processed by components such as multi-head self-attention, residual connection, and layer normalization, is input into the BusMMoE layer. There are multiple single-frame static experts and multiple temporal dynamic experts, each with different processing specialties. Therefore, a static routing network is used to select and activate the single-frame static expert currently identifying static lesion features, and a dynamic routing network is used to select and activate the dynamic temporal expert currently identifying dynamic temporal features. The single-frame static experts and the dynamic temporal experts extract features independently, which can achieve decoupled modeling of spatial and temporal features of breast ultrasound video. Subsequently, the index of keyframes can be output synchronously based on the dynamic routing weights.

[0055] Specifically, the static routing network calculates static routing weights for allocating features by evaluating the matching degree or responsivity of the features of the current input frame with each single-frame static expert; similarly, the dynamic routing network calculates dynamic routing weights for allocating context features by evaluating the matching degree of context feature blocks with each temporal dynamic expert. This mechanism of dynamically generating weights based on input content enables the BusMMoE layer to adaptively activate the most suitable single-frame static experts and temporal dynamic experts to process the corresponding information.

[0056] Single-frame static expert (SMoE): This focuses on capturing static lesion features in a single frame of ultrasound image, such as the lesion's contour and aspect ratio. The activated single-frame static expert utilizes its internal neural network structure to perform nonlinear transformations and feature mappings on the input feature representation of the current frame, outputting the static lesion features of that frame of ultrasound image.

[0057] Dynamic Mixture of Experts (DMoE): Focuses on capturing dynamic temporal features of consecutive frames. To improve parameter efficiency, DMoE is globally shared across the time dimension. Activated DMoE receives contextual feature blocks including information from the current frame and its preceding and following frames. By extracting differences and correlations between multiple consecutive frames, it outputs dynamic temporal features of lesions during scanning, such as deformation, relative motion, and echoes.

[0058] Shared Expert (SharedE): This technology processes each feature representation in a feature sequence directly, without relying on a routing network, extracting common features between frames, such as the overall echo level of the breast, inter-frame brightness consistency, and the regularity of the scanning operation. For the first feature sequence... Tokens Input it into the shared expert, and the shared expert's output is shown below: ; in, For the time step index of video frames; For the first feature sequence The characteristic representation of the token, the first The token corresponds to the first... Frame, i.e. Also the first Frame feature representation; Extract functions for shared experts; The function to calculate FFN.

[0059] Shared experts typically employ a standard feedforward neural network structure. Because it processes all frames indiscriminately using the same network parameters, it can effectively extract general background features that do not change drastically over time from a global perspective.

[0060] Finally, the BusMMoE layer, through the collaborative efforts of single-frame static experts, temporal dynamic experts, and shared experts, divides the modeling work among breast ultrasound video information to achieve a balanced knowledge partitioning. The final output information of the BusMMoE layer is shown below: ; in, To identify static lesion characteristics Dynamic time series characteristics and general features The video feature representation obtained after summation; The total number of static experts per frame; This represents the total number of time-series dynamic experts. This is the index number for a single-frame static expert. This is the index number for the time-series dynamics expert; To make the first The token is assigned to the first Static routing weights of a single-frame static expert; For the first The feature extraction function of a single-frame static expert is the FFN calculation function; To make the first The context feature block of the token is assigned to the first token. Dynamic routing weights for each time-series dynamic expert; For the first The feature extraction function of a time-series dynamic expert is the FFN calculation function.

[0061] It should be noted that the output of the BusMMoE layer mentioned above only represents the current [number]th [episode]. The feature state processed by the layer-by-layer hybrid expert Transformer module is then used as the input to the next layer-by-layer hybrid expert Transformer module in the layer-by-layer hybrid expert Transformer encoder. L The layered hybrid expert Transformer modules are stacked and progressively extracted and abstracted, with the final layer's output feature sequence constituting the complete video feature representation output by the hybrid expert Transformer encoder. .

[0062] The hierarchical hybrid expert Transformer encoder inputs the obtained video feature representation, static routing weights, and dynamic routing weights into the output module of the lesion analysis model. The output module performs classification processing on the global representation through a multi-layer fully connected classification head, and calculates the importance score by combining the static routing weights and dynamic routing weights. Based on the importance score of all frames in the breast ultrasound video, all frames are sorted, and several frames with higher scores are selected as the key frames with the most reference value for diagnosis. Finally, the lesion analysis results are output, which include the probability of benign or malignant lesions and the key frame index.

[0063] The breast ultrasound video processing method provided in this application designes single-frame static experts, temporal dynamic experts, and shared experts in a hierarchical hybrid expert layer. This achieves decoupled modeling of static lesion features and dynamic temporal features in breast ultrasound videos, avoiding the problem of inaccurate features caused by mixing the two types of features in related technologies. By synchronously outputting the benign and malignant probability of lesions and keyframe indexes through the lesion analysis model, it is not necessary to extract keyframes first and then classify them, eliminating the risk of upstream keyframe selection errors being transmitted downstream. It can accurately capture multidimensional lesion features in breast ultrasound videos, improve the accuracy of predicted benign and malignant probabilities of lesions, and output clinically interpretable keyframe indexes, which can assist doctors in making rapid and accurate diagnostic decisions.

[0064] It should be noted that each implementation method of this application can be freely combined, rearranged, or executed individually, and does not need to rely on or depend on a fixed execution order.

[0065] In some embodiments, the lesion analysis model further includes an input module, which is used for: The breast ultrasound video was segmented into continuous video frames; Extract spatial features from each frame of a continuous video frame; Positional coding is added to the spatial features of each frame to obtain the initial feature sequence of consecutive video frames; A category token is inserted before the sequence head of the initial feature sequence to obtain the feature sequence of consecutive video frames; The feature sequence is input into the hierarchical hybrid expert Transformer encoder.

[0066] Specifically, in this embodiment, the spatial features refer to the feature vectors extracted from a single frame of ultrasound image that reflect the spatial structure information of the image.

[0067] Location coding refers to the coding vector attached to the spatial features of each frame, which is used to identify the temporal position of that frame in the video sequence.

[0068] The initial feature sequence refers to the frame feature sequence formed after adding positional encoding to the spatial features of each frame, before the category token has been inserted.

[0069] Category tokens are trainable feature vectors inserted at the beginning of the initial feature sequence to aggregate global information of the entire video sequence during subsequent encoding.

[0070] This embodiment divides a breast ultrasound video into continuous video frames. ,in, The total number of frames in a breast ultrasound video is an empirical value based on the average duration and frame rate of a routine breast ultrasound dynamic scan video.

[0071] For the first part of breast ultrasound video frame First, spatial features of the ultrasound image frame are extracted using a backbone network. The backbone network can be the Swin-Transformer encoder, a feature extraction network pre-trained on a breast ultrasound dataset. Then, corresponding learnable positional codes are added to the spatial features. To preserve temporal information, feature representation is obtained. The formula is shown below: ; in, , for A dimensional real vector space.

[0072] Each frame corresponds to a feature representation, and the method for obtaining the feature representation of each frame is the same, so it will not be elaborated here. The initial feature sequence is the set of feature representations of all current frames. In order to perform benign and malignant classification of breast ultrasound videos, a learnable category token is inserted before the beginning of the initial feature sequence. The preprocessed feature sequence is obtained. , , .

[0073] The breast ultrasound video processing method provided in this application can effectively preserve the spatial lesion features and temporal change information of video frames by segmenting the breast ultrasound video and fusing location codes and category tokens to construct a feature sequence. This provides a standardized data foundation for efficient feature extraction and global pattern discrimination of subsequent hierarchical hybrid expert networks.

[0074] In some embodiments, the lesion analysis model further includes: A static routing network is used to activate the target single-frame static expert based on the features of the current frame and generate the static routing weights of the target single-frame static expert. A dynamic routing network is used to activate the target temporal dynamic expert based on context feature blocks and generate dynamic routing weights for the target temporal dynamic expert.

[0075] Specifically, the features of the current frame in this embodiment are obtained after the current frame has undergone backbone network and position coding processing, i.e., as described above. The feature can be obtained from the feature sequence.

[0076] The target single-frame static expert refers to the single-frame static expert selected by the static routing network from multiple single-frame static experts based on the characteristics of the current frame, and is activated to extract the static lesion morphological features of the current frame.

[0077] The target temporal dynamic expert refers to the temporal dynamic expert selected by the dynamic routing network from multiple temporal dynamic experts based on the context feature block formed by the current frame and adjacent frames, and is activated to extract the dynamic temporal features within the local time window.

[0078] In this embodiment, the static routing network is used to activate the target single-frame static expert. The static routing network will activate the target single-frame static expert corresponding to the lesion feature according to the features of the current frame, while other single-frame static experts remain silent.

[0079] Feature representation of feature sequences Static routing network Calculate a dimension as Unnormalized fractional vector To maintain the sparsity of the network, a Top-K strategy is used to select the top score vectors based on their scores in the unnormalized score vectors. The top-scoring single-frame static experts are selected as target single-frame static experts. Then, the scores of the selected target single-frame static experts are normalized using the exponential normalization (Softmax) function to generate the static routing weights of the target single-frame static experts. The formulas involved are shown below: ; ; ; ; in, and These are the learnable parameters for a static routing network. This is the index set of the selected target single-frame static experts. The index number of the index set.

[0080] Similarly, for dynamic routing networks, they are used to activate target temporal dynamic experts based on context feature blocks in order to capture motion trajectories such as lesion morphology changes or echo mutations that occur in consecutive frames.

[0081] Feature representation of feature sequences Dynamic routing network Context feature blocks constructed by perceiving adjacent frames The computational dimension is Unnormalized fractional vector Similarly, selection is based on scores. The top-scoring time-series dynamic experts are selected as target time-series dynamic experts, and their dynamic routing weights are generated through exponential normalization, as shown in the following formula: ; ; ; ; ； in, and These are learnable parameters for dynamic routing networks. The set of indexes for the selected target time-series dynamic experts; To select the top score in descending order of normalized score K A function for each index, The feature representation of the previous frame is used for the current frame. When the current frame is the first frame (t=1), the feature representation of the first frame itself is used. This is the feature representation of the frame following the current frame. When the current frame is the last frame (t=T), the feature representation of the last frame itself is taken.

[0082] The above , , and These are all fixed values ​​set based on hardware conditions such as model parameter capacity and video memory limitations.

[0083] It should be noted that the main function of temporal dynamics experts is to extract dynamic temporal features between consecutive video frames, due to the category token... Primarily used for aggregating global information, it does not correspond to a specific physical time frame, therefore there is no need to construct adjacent local time windows for it. (Category token) During the multi-head self-attention component computation of the hierarchical hybrid expert Transformer module, it interacts globally with the token of each frame to automatically absorb and obtain a category token representation containing the spatiotemporal information of the entire breast ultrasound. In this context, the Token for each frame represents the feature of that frame.

[0084] The breast ultrasound video processing method provided in this application, by designing a static routing network and a dynamic routing network and using a score ranking and normalization strategy, can accurately activate target single-frame static experts and target temporal dynamic experts with specific functions, thereby achieving decoupled extraction of spatial and temporal features and overcoming the knowledge confusion problem caused by the hybrid feature extraction of traditional hybrid expert networks.

[0085] In some embodiments, the hierarchical hybrid expert Transformer encoder obtains static lesion features, dynamic temporal features, and general features based on feature sequences, and obtains a video feature representation of the breast ultrasound video based on the static lesion features, dynamic temporal features, and general features, and inputs the video feature representation to the output module; The output module includes a multi-layer fully connected classification head, which obtains the benign and malignant probabilities of lesions based on a global representation of breast ultrasound video, including: The output module uses the category token representation in the video feature representation as the global representation of the breast ultrasound video; The global representation is input into a multi-layer fully connected classification head to obtain the benign or malignant probability of the lesion output by the multi-layer fully connected classification head.

[0086] Specifically, the input module will input the feature sequence. The input is fed into a hierarchical hybrid expert Transformer encoder. The encoder performs nonlinear mapping extraction on each feature representation in the feature sequence based on the target single-frame static expert, obtaining the static lesion features corresponding to each frame; it performs temporal correlation extraction on each feature representation in the feature sequence based on the target temporal dynamic expert, obtaining the dynamic temporal features corresponding to each frame; and it performs basic transformation extraction on each feature representation in the feature sequence based on the shared expert, obtaining the general features corresponding to each frame. For each feature representation, the static lesion features, dynamic temporal features, and general features are summed to obtain the fused features corresponding to that feature representation. All the corresponding fused features together constitute the video feature representation, which is then sent to the output module.

[0087] The output module extracts the category token representation from the video feature representation. As a global representation of the entire video frame sequence and global representation Input multi-level fully connected classification head Multi-layer fully connected classification head outputs the probability of benign or malignant lesions. The formula is shown below: ; The breast ultrasound video processing method provided in this application converges static lesion features, dynamic temporal features, and general features, and directly extracts the category token representation to input the fully connected classification head. This method can efficiently coordinate the global spatiotemporal evolution and general basic features of breast ultrasound video, thereby providing an accurate probability of benign or malignant lesions.

[0088] In some embodiments, keyframe indexes are determined based on the importance score of each frame in a breast ultrasound video, including: The static lesion saliency score for each frame is determined based on the static routing weights; The dynamic lesion saliency score for each frame is determined based on dynamic routing weights; The importance score for each frame is obtained by weighted fusion of the static lesion significance score and the dynamic lesion significance score; Keyframes are selected from all frames of the breast ultrasound video based on the importance score of each frame, and a keyframe index is generated for each keyframe.

[0089] Specifically, the static lesion saliency score refers to a quantitative score calculated based on static routing weights, which measures the saliency of static lesion features in the current frame.

[0090] The dynamic lesion saliency score is a quantitative score calculated based on dynamic routing weights, which measures the degree of significance of dynamic temporal changes in the context feature block of the current frame.

[0091] Keyframes refer to a predetermined number of representative image frames selected from all frames in a breast ultrasound video, ranked from highest to lowest importance.

[0092] This embodiment does not require an additional keyframe extraction network. Instead, it uses both static and dynamic routing networks to calculate the importance score of each frame. The static routing network is used to evaluate whether the current frame contains clear and typical lesion morphological features. The dynamic routing network is used to evaluate whether there are continuous, stable or significant lesion-related temporal features within the local time window of the current frame.

[0093] For the Frame by frame, calculate the static lesion significance score. and dynamic lesion significance score Then apply weights to both. Perform weighted fusion to obtain the importance score of the frame. The importance score for each frame is calculated in the same way, so it will not be repeated here. The importance scores of all frames are normalized, and finally, they are sorted in descending order of normalized importance scores. The frame with the highest score is selected. The frame is used as a keyframe, and the keyframe index is output. The relevant formulas are shown below: ; ; ; ; ; in, For the first The importance score of the frame after normalization; The importance score for unnormalized values, A dummy element for frame indexing; For activation functions; weights and the number of keyframes selected. These values ​​were determined based on a comparative experiment of importance assessment on a dataset labeled by medical experts.

[0094] The breast ultrasound video processing method provided in this application can simultaneously output accurate keyframe indexes without increasing the burden of additional network computing power by directly reusing static routing weights and dynamic routing weights for dual significance weighted scoring. This serves as an intuitive basis for clinical diagnosis and greatly improves the interpretability of the model.

[0095] In some embodiments, the training process of the lesion analysis model includes the following steps: Obtain the training dataset, which includes sample breast ultrasound videos and the benign or malignant labels of the sample lesions corresponding to the sample breast ultrasound videos; The sample breast ultrasound video is input into the initial lesion analysis model to obtain the benign and malignant probabilities of the lesions output by the initial lesion analysis model. The initial lesion analysis model is the lesion analysis model to be trained. The classification cross-entropy loss of the initial lesion analysis model is determined based on the probability of benign or malignant lesions and the benign or malignant labels of sample lesions. The first load balancing loss of a single-frame static expert is determined based on the static routing weights corresponding to the single-frame static experts during training, and the second load balancing loss of a time-series dynamic expert is determined based on the dynamic routing weights corresponding to the time-series dynamic experts. The parameters of the initial lesion analysis model are updated based on the classification cross-entropy loss, the first load balancing loss, and the second load balancing loss.

[0096] Specifically, classification cross-entropy loss is a loss function used to measure the difference between the probability of benign or malignant lesions output by the lesion analysis model and the true label.

[0097] The first load balancing loss refers to the regularization loss calculated based on the static routing weights of single-frame static experts, used to promote the balanced use of each single-frame static expert.

[0098] The second load balancing loss refers to the regularization loss calculated based on the dynamic routing weights of time-series dynamic experts, which is used to promote the balanced use of each time-series dynamic expert.

[0099] The initial lesion analysis model is trained end-to-end by jointly optimizing the classification cross-entropy loss and the expert load balancing loss.

[0100] First, a training dataset containing real batch data is obtained and input into the initial lesion analysis model to obtain the benign / malignant probability of the lesions output by the initial lesion analysis model. To optimize the classification process, the initial lesion analysis model is trained end-to-end by minimizing the binary cross-entropy loss. The formula is shown below: ; in, Batch size; This refers to the index number of the sample in the batch; The true label is the benign or malignant label of the lesion corresponding to the sample breast ultrasound video. To predict the probability, i.e. the probability of benign or malignant lesions output by the initial lesion analysis model, the batch size is a preset value obtained by experimental optimization based on the training hardware memory capacity and computing efficiency.

[0101] Secondly, to ensure balanced use of different experts, load balancing losses were designed for both single-frame static experts and time-series dynamic experts. Shared experts do not depend on the routing network and do not require load balancing. The total number of tokens within the batch is used as the load balancing factor. Based on this, the total number of tokens includes all video frame tokens and category tokens inserted at the beginning of the sequence. This total number is used as the denominator when calculating the frequency to achieve standardization. The first load balancing loss of a single frame static expert is calculated based on the static routing weight. The specific formula is as follows: ; ; ; in, The frequency selected by the single-frame static expert. For single-frame static experts Average routing weight received within the batch In the batch The first sample of breast ultrasound video The characteristic representation of a frame, namely a token. This is an indicator function; it takes the value 1 when the condition within the parentheses is true, and 0 otherwise.

[0102] Similarly, the second load balancing loss is calculated based on the dynamic routing weights of the time-series dynamic expert. The specific formula is as follows: ; ; ; in, The frequency at which time-series dynamic experts are selected. For time series dynamics experts Average routing weight received within the batch.

[0103] The final loss function is obtained by weighted summation of the classification cross-entropy loss, the first load balancing loss, and the second load balancing loss. and based on Update the parameters of the initial lesion analysis model using the formula shown below: ; in, The weighted hyperparameters are preset values ​​that are adjusted and determined before training to balance the gradient of the primary classification task and the gradient of the secondary task to balance the expert load.

[0104] It should be noted that although the lesion analysis results in this embodiment include the probability of lesion benignity or malignancy and the keyframe index, the keyframe index is obtained by reusing static routing weights and dynamic routing weights during the inference stage and is not involved in the calculation of training loss.

[0105] The breast ultrasound video processing method provided in this application, by jointly optimizing the classification cross-entropy loss and the load balancing loss of each expert, can ensure that the expert networks under different routing nodes are evenly activated when training and updating parameters, avoiding the model's over-reliance on a single expert and causing other experts to be idle, thereby giving full play to the feature decoupling potential of the hybrid expert architecture and improving the overall diagnostic performance of the model.

[0106] In some embodiments, to verify the effectiveness of the breast ultrasound video processing method provided in this application, the dataset used in this embodiment includes dynamic scanning breast ultrasound videos of 374 lesions. Each lesion underwent a complete surgical pathological examination, including 254 benign lesions and 120 malignant lesions. Each lesion was dynamically scanned by an ultrasound physician using transverse and longitudinal sections, resulting in transverse and longitudinal ultrasound videos. Furthermore, each ultrasound video from each perspective was annotated with 8 keyframes by two ultrasound physicians.

[0107] This embodiment uses five-fold cross-validation for evaluation. The performance of lesion benign / malignant classification is evaluated using metrics such as Area Under the ROC Curve (AUC), Precision-Recall Curve (PRC), Accuracy (ACC), F1 score, Sensitivity, Specificity, Positive Predictive Value (PPV), Negative Predictive Value (NPV), Precision, and Confusion matrix. The effectiveness of keyframe indexing is evaluated by comparing the hit rate of keyframes selected by the model with those annotated by doctors.

[0108] The total number of frames in the breast ultrasound video in this embodiment 100. The number of modules in the hierarchical hybrid expert Transformer is... The multi-head self-attention component uses 8 heads, with a vector dimension of 8. In the BusMMoE layer, both the single-frame static expert and the temporal dynamic expert are implemented using FFN, and the number of single-frame static experts is [not specified]. The Top-1 strategy is adopted, that is... Number of time series dynamic experts , Number of shared experts In the hyperparameter settings, the keyframe scoring weights Weighted hyperparameters of the loss function Batch size =16, the optimization algorithm uses Adam, and the initial learning rate is... The overall implementation uses the Python language and the PyTorch deep learning framework.

[0109] This embodiment validates the proposed lesion analysis model (BusMMoE model) on a real dataset. First, the model's performance is compared on a task involving the classification of benign and malignant lesions based on single-view breast ultrasound video. Figure 4 This is a schematic diagram illustrating the performance comparison results of the single-view classification model provided in the embodiments of this application, such as... Figure 4 As shown, the AUC of the BUSMMoE model is 84.0%±6.6% and the ACC is 77.0%±6.0% from the cross-sectional view. From the longitudinal view, the AUC of the BUSMMoE model is 82.8%±4.8% and the ACC is 79.7%±4.3%. Whether from the cross-sectional or longitudinal view, the BusMMoE model outperforms the baseline Transformer model.

[0110] This embodiment then compares the model performance on a task of determining benign and malignant lesions based on dual-view breast ultrasound video. Figure 5 This is a schematic diagram illustrating the performance comparison results of the dual-view classification model provided in the embodiments of this application, as shown below. Figure 5 As shown, the BusMMoE model has an AUC of 86.7% ± 4.7% and an F1 score of 78.5% ± 6.3% under dual-view conditions, while the MBT-Trust model has an AUC of 84.9% ± 6.7% and an F1 score of 78.1% ± 6.7%. Overall, the BusMMoE model achieves better classification performance than the baseline model.

[0111] Figure 6 The ROC curve of the benign / malignant lesion classification model based on breast ultrasound video provided in the embodiments of this application is shown. Figure 7 The PRC curve of the lesion benign and malignant classification model based on breast ultrasound video provided in the embodiments of this application is shown below. Figure 6 and Figure 7 As shown, the classification ability of the dual-view classification model represented by the solid line is better than that of the single-view classification model represented by the dashed line; the BusMMoE model shows the best ability to distinguish between benign and malignant lesions in both single-view and dual-view classification tasks. Figure 8 This is a confusion matrix diagram of the single-view breast ultrasound video lesion classification model provided in the embodiments of this application. Figure 9 This is a confusion matrix diagram of the dual-view breast ultrasound video lesion classification model provided in the embodiments of this application. (See diagram below.) Figure 8 As shown, the BusMMoE model detected 86 / 120 malignant cases in the longitudinal section view model, with a malignant recall of 71.7%, but misclassified 42 / 254 benign cases as malignant. Figure 9As shown, the Att-MBT-Trust model exhibited the highest benign specificity (94.1%) among the dual-view classification models, with 15 / 254 benign false positives but a malignant recall of 59.2%. The BusMMoE model detected 85 / 120 malignant cases and missed 35 cases in the multi-view model, while misclassifying 34 / 254 benign cases as malignant, demonstrating a relatively balanced control of malignant detection and benign false positives. These results indicate that the hierarchical hybrid expert structure of the BusMMoE model can effectively model breast ultrasound video features and achieve better classification results for benign and malignant lesions.

[0112] This embodiment verifies the effectiveness of keyframe indexing on a real dataset. The method in this application obtains the importance score of each frame based on a dynamic routing network and a static routing network. It selects the 8 frames with the highest importance scores from both horizontal and vertical viewpoint videos as keyframes, and then compares them with the 8 keyframes annotated by the doctor. Experimental results show that the model hits at least one frame in 97.59% of the samples, and the keyframes output by the model are generally near the keyframes annotated by the doctor.

[0113] This embodiment also includes a case analysis to verify the effectiveness of the proposed technical solution. This embodiment analyzes a case of benign lesions (29-05-20210205-001-01) in a transverse ultrasound video. The BusMMoE model not only provides the correct benign classification, but frames 58, 70, and 76 are completely identical to the frames annotated by the doctor, with an average nearest distance of 1.1 frames between the predicted and annotated keyframe indices. This embodiment then analyzes a case of malignant lesions (29-07-20211020-001) in a longitudinal ultrasound video. The results show that the BusMMoE model provides the correct malignant classification, with a complete hit in frame 101, and an average nearest distance of 3.1 frames between the predicted and annotated keyframe indices. These results demonstrate that the keyframe scores provided by this technical solution have a certain focusing ability on diagnostically relevant areas. The model can automatically locate diagnostically relevant segments near the doctor-annotated keyframes, and the output keyframe indices can serve as evidence supporting the benign / malignant classification of lesions.

[0114] Based on the results of the above embodiments, the technical solution of this application has the following advantages: (1) This application adopts a single-stage breast ultrasound video processing framework, which completes the classification of benign and malignant lesions and the output of keyframe indexes in the same model, without the need to first determine keyframes through an independent keyframe selection process and then input them into the downstream diagnostic model. As a result, the transmission of keyframe selection errors to the lesion classification stage can be reduced, and keyframe localization and lesion benign and malignant classification can be completed collaboratively under a unified goal. This processing method is also closer to the clinical process of ultrasound physicians completing the diagnosis of benign and malignant lesions and selecting keyframes as diagnostic basis during dynamic scanning.

[0115] (2) This application designs single-frame static experts, temporal dynamic experts, and shared experts in a hierarchical hybrid expert layer. Among them, single-frame static experts are used to extract static lesion features in a single-frame ultrasound image, temporal dynamic experts are used to extract dynamic temporal features within a local time window, and shared experts are used to extract common features between different video frames. By modeling different types of features in a differentiated and collaborative manner, the mutual interference between static lesion features and dynamic temporal features can be reduced, and the possibility of confusion or conflict in the knowledge learned by different experts can be reduced, thereby improving the feature representation ability of the model and the ability to distinguish between benign and malignant lesions.

[0116] (3) This application comprehensively utilizes the routing responses of single-frame static expert branches and temporal dynamic expert branches to calculate the keyframe importance score of each video frame, and determines the keyframe index based on the keyframe importance score. Therefore, this application does not require setting up an additional keyframe extraction network, and can output the keyframe index while outputting the benign and malignant classification probability of lesions. The output keyframes can provide corresponding image evidence for the benign and malignant classification results of lesions, which is beneficial to improving the interpretability of the model output results.

[0117] (4) Experimental results on the real breast ultrasound video dataset used in this application show that, compared with the selected baseline model, the method in this application achieves better performance in the task of classifying benign and malignant lesions, and the output keyframe index is similar to the doctor's annotation results. The above results indicate that the technical solution of this application can simultaneously complete the classification of benign and malignant breast lesions and keyframe localization, which can provide assistance to ultrasound doctors in lesion diagnosis and keyframe selection, and has the potential to improve clinical efficiency.

[0118] The breast ultrasound video processing apparatus provided in the embodiments of this application is described below. The breast ultrasound video processing apparatus described below can be referred to in correspondence with the breast ultrasound video processing method described above.

[0119] Figure 10 This is a schematic diagram of the structure of the breast ultrasound video processing device provided in the embodiments of this application, as shown below. Figure 10 As shown, the device includes an acquisition module 1010 and a processing module 1020.

[0120] Module 1010 is used to acquire breast ultrasound videos; The processing module 1020 is used to input breast ultrasound video into the lesion analysis model and obtain the lesion analysis results output by the lesion analysis model. The lesion analysis results include the probability of benign or malignant lesions and the keyframe index of the breast ultrasound video. The lesion analysis model includes a hierarchical hybrid expert layer, which comprises: Single-frame static expert, used to extract static lesion features from each frame of a breast ultrasound video; The temporal dynamics expert is used to extract dynamic temporal features of breast ultrasound videos based on the contextual feature blocks formed by the current frame and its neighboring frames. Shared experts are used to extract common features from all frames in breast ultrasound videos; The lesion analysis model also includes an output module, which is used to obtain the benign and malignant probabilities of lesions based on the global representation of breast ultrasound video, and to determine the key frame index based on the importance score of each frame in the breast ultrasound video; the global representation is determined based on static lesion features, dynamic temporal features and general features, and the importance score is determined based on the static routing weight of the single-frame static expert and the dynamic routing weight of the temporal dynamic expert.

[0121] Specifically, according to the embodiments of this application, any multiple modules in the acquisition module 1010 and the processing module 1020 can be combined into one module, or any one of the modules can be split into multiple modules.

[0122] Alternatively, at least some of the functionality of one or more of the modules in the acquisition module 1010 and the processing module 1020 may be combined with at least some of the functionality of other modules and implemented in one module.

[0123] According to embodiments of this application, at least one of the acquisition module 1010 and the processing module 1020 can be at least partially implemented as hardware circuitry, such as a Field Programmable Gate Array (FPGA), Programmable Logic Array (PLA), System-on-a-Chip, System-on-a-Substrate, System-on-Package, Application Specific Integrated Circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three methods of software, hardware, and firmware, or in a suitable combination of any of these.

[0124] Alternatively, at least one of the acquisition module 1010 and the processing module 1020 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0125] In some embodiments, the lesion analysis model includes a hierarchical hybrid expert Transformer encoder, which includes a hierarchical hybrid expert layer. The lesion analysis model also includes an input module, which is used for: The breast ultrasound video was segmented into continuous video frames; Extract spatial features from each frame of a continuous video frame; Positional coding is added to the spatial features of each frame to obtain the initial feature sequence of consecutive video frames; A category token is inserted before the sequence head of the initial feature sequence to obtain the feature sequence of consecutive video frames; The feature sequence is input into the hierarchical hybrid expert Transformer encoder.

[0126] In some embodiments, the lesion analysis model further includes: A static routing network is used to activate the target single-frame static expert based on the features of the current frame and generate the static routing weights of the target single-frame static expert. A dynamic routing network is used to activate the target temporal dynamic expert based on context feature blocks and generate dynamic routing weights for the target temporal dynamic expert.

[0127] In some embodiments, the hierarchical hybrid expert Transformer encoder obtains static lesion features, dynamic temporal features, and general features based on feature sequences, and obtains a video feature representation of the breast ultrasound video based on the static lesion features, dynamic temporal features, and general features, and inputs the video feature representation to the output module; The output module includes a multi-layer fully connected classification head, which obtains the benign and malignant probabilities of lesions based on a global representation of breast ultrasound video, including: The output module uses the category token representation in the video feature representation as the global representation of the breast ultrasound video; The global representation is input into a multi-layer fully connected classification head to obtain the benign or malignant probability of the lesion output by the multi-layer fully connected classification head.

[0128] In some embodiments, keyframe indexes are determined based on the importance score of each frame in a breast ultrasound video, including: The static lesion saliency score for each frame is determined based on the static routing weights; The dynamic lesion saliency score for each frame is determined based on dynamic routing weights; The importance score for each frame is obtained by weighted fusion of the static lesion significance score and the dynamic lesion significance score; Keyframes are selected from all frames of the breast ultrasound video based on the importance score of each frame, and a keyframe index is generated for each keyframe.

[0129] In some embodiments, the training process of the lesion analysis model includes the following steps: Obtain the training dataset, which includes sample breast ultrasound videos and the benign or malignant labels of the sample lesions corresponding to the sample breast ultrasound videos; The sample breast ultrasound video is input into the initial lesion analysis model to obtain the benign and malignant probabilities of the lesions output by the initial lesion analysis model. The initial lesion analysis model is the lesion analysis model to be trained. The classification cross-entropy loss of the initial lesion analysis model is determined based on the probability of benign or malignant lesions and the benign or malignant labels of sample lesions. The first load balancing loss of a single-frame static expert is determined based on the static routing weights corresponding to the single-frame static experts during training, and the second load balancing loss of a time-series dynamic expert is determined based on the dynamic routing weights corresponding to the time-series dynamic experts. The parameters of the initial lesion analysis model are updated based on the classification cross-entropy loss, the first load balancing loss, and the second load balancing loss.

[0130] It should be noted that the breast ultrasound video processing device provided in this application embodiment can implement all the method steps implemented in the above-mentioned breast ultrasound video processing method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0131] Figure 11 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 11 As shown, the electronic device may include a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 11110, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 11110. The processor 1110 can call the computer program in the memory 1130 to execute the above-described method.

[0132] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional modules and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the methods provided in the above embodiments.

[0134] On the other hand, embodiments of this application also provide a non-transitory computer-readable storage medium storing a computer program for causing a processor to execute the methods provided in the above embodiments.

[0135] The non-transitory computer-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0136] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for processing breast ultrasound video, characterized in that, include: Obtain breast ultrasound video; The breast ultrasound video is input into the lesion analysis model to obtain the lesion analysis results output by the lesion analysis model. The lesion analysis results include the probability of benign or malignant lesions and the keyframe index of the breast ultrasound video. The lesion analysis model includes a hierarchical hybrid expert layer, which comprises: A single-frame static expert is used to extract static lesion features from each frame of the breast ultrasound video. A temporal dynamics expert is used to extract the dynamic temporal features of the breast ultrasound video based on the context feature block formed by the current frame and the adjacent frames of the current frame. Shared experts are used to extract common features from all frames in the breast ultrasound video; The lesion analysis model further includes an output module, used to obtain the benign or malignant probability of the lesion based on the global representation of the breast ultrasound video, and to determine the keyframe index based on the importance score of each frame in the breast ultrasound video; the global representation is determined based on the static lesion features, the dynamic temporal features and the general features, and the importance score is determined based on the static routing weight of the single-frame static expert and the dynamic routing weight of the temporal dynamic expert.

2. The breast ultrasound video processing method according to claim 1, characterized in that, The lesion analysis model includes a hierarchical hybrid expert Transformer encoder, which includes the hierarchical hybrid expert layer. The lesion analysis model also includes an input module, which is used for: The breast ultrasound video is segmented into continuous video frames; Extract the spatial features of each frame in the continuous video frames; Add positional coding to the spatial features of each frame to obtain the initial feature sequence of the consecutive video frames; A category token is inserted before the sequence header of the initial feature sequence to obtain the feature sequence of the consecutive video frames; The feature sequence is input into the hierarchical hybrid expert Transformer encoder.

3. The breast ultrasound video processing method according to claim 1, characterized in that, The lesion analysis model also includes: A static routing network is used to activate a target single-frame static expert based on the features of the current frame and generate the static routing weights of the target single-frame static expert. A dynamic routing network is used to activate a target temporal dynamic expert based on the context feature block and generate the dynamic routing weights of the target temporal dynamic expert.

4. The breast ultrasound video processing method according to claim 2, characterized in that, The hierarchical hybrid expert Transformer encoder obtains the static lesion features, the dynamic temporal features, and the general features based on the feature sequence, and obtains the video feature representation of the breast ultrasound video based on the static lesion features, the dynamic temporal features, and the general features, and inputs the video feature representation to the output module; The output module includes a multi-layer fully connected classification head, and the process of obtaining the benign or malignant probability of the lesion based on the global representation of the breast ultrasound video includes: The output module uses the category token representation in the video feature representation as the global representation of the breast ultrasound video; The global representation is input into the multi-layer fully connected classification head to obtain the benign or malignant probability of the lesion output by the multi-layer fully connected classification head.

5. The breast ultrasound video processing method according to claim 1, characterized in that, Determining the keyframe index based on the importance score of each frame in the breast ultrasound video includes: The static lesion saliency score for each frame is determined based on the static routing weights. The dynamic lesion salience score for each frame is determined based on the dynamic routing weights. The importance score for each frame is obtained by weighted fusion of the static lesion significance score and the dynamic lesion significance score; Keyframes are selected from all frames of the breast ultrasound video based on the importance score of each frame, and the keyframe index of the keyframes is generated.

6. The breast ultrasound video processing method according to claim 1, characterized in that, The training process of the lesion analysis model includes the following steps: Obtain a training dataset, which includes sample breast ultrasound videos and benign / malignant labels for the sample lesions corresponding to the sample breast ultrasound videos; The sample breast ultrasound video is input into the initial lesion analysis model to obtain the benign or malignant probability of the lesion output by the initial lesion analysis model. The initial lesion analysis model is the lesion analysis model to be trained. The classification cross-entropy loss of the initial lesion analysis model is determined based on the probability of benign or malignant lesions and the benign or malignant labels of the sample lesions; The first load balancing loss of the single-frame static expert is determined based on the static routing weights corresponding to the single-frame static expert during training, and the second load balancing loss of the time-series dynamic expert is determined based on the dynamic routing weights corresponding to the time-series dynamic expert. The parameters of the initial lesion analysis model are updated based on the classification cross-entropy loss, the first load balancing loss, and the second load balancing loss.

7. A breast ultrasound video processing device, characterized in that, include: The acquisition module is used to acquire breast ultrasound videos; The processing module is used to input the breast ultrasound video into the lesion analysis model and obtain the lesion analysis results output by the lesion analysis model. The lesion analysis results include the probability of benign or malignant lesions and the keyframe index of the breast ultrasound video. The lesion analysis model includes a hierarchical hybrid expert layer, which comprises: A single-frame static expert is used to extract static lesion features from each frame of the breast ultrasound video. A temporal dynamics expert is used to extract the dynamic temporal features of the breast ultrasound video based on the context feature block formed by the current frame and the adjacent frames of the current frame. Shared experts are used to extract common features from all frames in the breast ultrasound video; The lesion analysis model further includes an output module, used to obtain the benign or malignant probability of the lesion based on the global representation of the breast ultrasound video, and to determine the keyframe index based on the importance score of each frame in the breast ultrasound video; the global representation is determined based on the static lesion features, the dynamic temporal features and the general features, and the importance score is determined based on the static routing weight of the single-frame static expert and the dynamic routing weight of the temporal dynamic expert.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the breast ultrasound video processing method according to any one of claims 1 to 6 through the computer program.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the breast ultrasound video processing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the breast ultrasound video processing method as described in any one of claims 1 to 6.

Images