Reliable breast cancer diagnosis method and system based on multi-view ultrasonic video fusion

Abstract

The invention discloses a reliable breast cancer diagnosis method and system based on multi-view ultrasonic video fusion, and belongs to the technical field of medical diagnosis. The method comprises the following steps: acquiring a breast ultrasonic video; the breast ultrasonic videos comprise ultrasonic videos of a transverse view angle and a longitudinal view angle; inputting the breast ultrasound video into the segmentation model frame by frame for segmentation to obtain a focus segmentation result; determining a key frame according to a focus segmentation result; the key frame comprises a key frame sequence of a transverse view angle and a longitudinal view angle; and inputting the key frame sequences of the transverse view angle and the longitudinal view angle into a focus classification model, and outputting a diagnosis result including a focus benign and malignant classification result and prediction uncertainty through multi-view fusion and uncertainty estimation. The reliable breast cancer diagnosis method based on multi-view ultrasonic video fusion has the advantages of high accuracy and high reliability, key frames in the ultrasonic video can be automatically positioned, focus features in the multi-view video are fused, and an accurate and reliable diagnosis result is given.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical diagnosis, in particular to a reliable breast cancer diagnosis method and system based on multi-view ultrasound video fusion. BACKGROUND

[0002] Breast cancer is a major health problem worldwide, and its incidence has been rising in many countries over the past few decades. By 2020, breast cancer has surpassed lung cancer to become the most common cancer, and is the leading cause of cancer-related deaths among women worldwide. Early breast cancer can be treated in time, and the prognosis is better. Therefore, timely and accurate identification of the benign and malignant of breast tumors has very important clinical significance. Breast ultrasound (BUS) is widely used for early screening and diagnosis due to its non-invasive, real-time, economical and high sensitivity to dense breast. However, its diagnostic effect is heavily dependent on the experience of the operator, especially for those cases with speckle noise, fuzzy boundary and limited spatial resolution, which face the bottleneck of strong subjectivity of artificial interpretation, low efficiency and shortage of qualified doctors.

[0003] To overcome the above limitations, the research of computer-aided diagnosis (CAD) system is emerging. Early methods rely on artificial design features to build machine learning models, which are time-consuming and labor-intensive, and the effect is general. In recent years, deep learning technology, especially convolutional neural networks (CNNs), vision transformers (ViTs) and their hybrid architectures, have shown significant advantages in the automatic detection and classification of breast lesions. For example, Qi et al. proposed a CNN method combined with region enhancement mechanism and cross-training algorithm for breast ultrasound image diagnosis. In a dataset containing 8145 ultrasound images, the method achieved an accuracy of about 94.48% in differentiating malignant tumors. The self-attention mechanism in ViTs can effectively capture long-distance dependencies. Iqbal et al. proposed a U-Net hybrid architecture integrating Swin-Transformer and CNN, which effectively captured global spatial dependencies and solved the problem of feature representation for irregular-shaped tumors. Qian et al. used SE-ResNet architecture to build an AI system for clinical application, which processed multi-modal breast ultrasound images and achieved a breast cancer diagnosis performance of 0.955. However, the above methods all rely on manual extraction of key frames from ultrasound videos, and the classification model is limited by the quality of the selected key frames. Patent CN113855079A proposes a real-time detection and breast disease auxiliary analysis method based on breast ultrasound images, which uses a fixed 1-second frame extraction strategy, but still requires manual screening of four-angle images, and lacks lesion segmentation assistance, which can easily miss key information. Patent CN112002407A proposes a breast cancer diagnosis device and method based on ultrasound video, which uses a feature extraction network and a lesion detection network to process single-frame image sequences, and then uses centralized learning to make a differential diagnosis, ignoring the temporal correlation in the sequence and lacking prediction uncertainty measurement in the classification process, making the diagnosis results lack reliability. These advances have brought new prospects for the automation and precision of breast cancer diagnosis.

[0004] Despite the remarkable achievements, current mainstream methods and systems still have some limitations: first, most of the research focuses on analyzing the static images manually selected by sonographers. The manual selection of key frames relies on experience and lacks automation, and the quality of the selection also limits the performance of the model, and it also loses the temporal correlation information contained in the dynamic scan video which is crucial for diagnosis; second, existing systems mainly rely on single-view image or single-view video input, ignoring the complementary information of lesions presented in the cross-sectional and longitudinal scanning processes. Some breast lesions, especially atypical lesions, often exhibit different ultrasound characteristics when the probe direction changes. Finally, existing systems can only give benign and malignant classification results, and lack of quantitative evaluation of the uncertainty of the prediction results. In clinical practice, this black-box perception of model reliability has seriously hindered the promotion and application of CAD systems in high-cost medical scenarios. SUMMARY

[0005] The purpose of the present application is to provide a reliable breast cancer diagnosis method and system based on multi-view ultrasound video fusion, to solve the problems of existing technologies and systems such as manual selection of key frames, single dimension of single-view input information, and lack of uncertainty measurement of prediction results, to realize automatic extraction of ultrasound video key frames, evidence-based fusion of complementary information in multi-view dynamic ultrasound video, accurate prediction of benign and malignant categories and estimation of individual prediction uncertainty, and to provide more accurate and reliable breast cancer diagnosis results.

[0006] Specifically, the present application first designs a key frame positioning module based on lesion size. This module not only automatically identifies the lesion area in the ultrasound video through a segmentation model, but also selects key frames based on lesion size. Second, in order to better fuse multi-view ultrasound video, a multi-view fusion strategy combining feature level and decision level is designed: at the feature fusion level, a multi-view ultrasound video fusion module guided by the segmentation model is designed, which realizes multi-view video feature fusion through an attention bottleneck mechanism; at the decision fusion level, the decision opinions of single-view videos are first obtained using subjective logic theory, and then the decision opinions of multiple views are aggregated based on evidence fusion method. The method proposed in the present application not only gives the probability of benign and malignant categories of the lesion, but also quantifies the uncertainty of individual prediction, greatly improving the reliability of breast cancer diagnosis.

[0007] To solve the above technical problems, the present application provides a reliable breast cancer diagnosis method and system based on multi-view ultrasound video fusion, comprising: acquiring a breast ultrasound video; the breast ultrasound video includes ultrasound videos of cross-sectional and longitudinal views; inputting the breast ultrasound video frame by frame into a segmentation model for segmentation to obtain a lesion segmentation result; Determine a key frame according to the lesion segmentation result; the key frame includes a key frame sequence of a transverse view and a longitudinal view; Input the key frame sequence of the transverse view and the longitudinal view into a lesion classification model, perform multi-view fusion and uncertainty estimation, and output a diagnosis result.

[0008] Preferably, the key frame is determined according to the lesion segmentation result, and specifically includes the following steps: Calculate the predicted lesion area size of the lesion segmentation result; Use a Near-K strategy or a Top-K strategy to select K candidate key frames from the ultrasound video of the transverse view and the longitudinal view based on the lesion area, as the key frame sequence; The Near-K strategy is to select the video frame with the largest lesion area, and select key frames near the frame as the key frame sequence; The Top-K strategy is to select key frames with the largest lesion area as the key frame sequence.

[0009] Preferably, the lesion classification model includes a transverse view evidence network based on a Swin Transformer, a longitudinal view evidence network, and a fusion view evidence network based on a bottleneck attention mechanism; The transverse view evidence network, the longitudinal view evidence network, and the fusion view evidence network are all composed of a Swin Transformer encoder, an intra-view Transformer module, and an MLP module with a ReLU activation layer.

[0010] Preferably, the key frame sequence of the transverse view and the longitudinal view is input into the lesion classification model, and multi-view fusion and uncertainty estimation are performed to output a diagnosis result, specifically including the following steps: Process the key frame sequence of the transverse view and the longitudinal view through the transverse view evidence network and the longitudinal view evidence network respectively to obtain a transverse view evidence vector and a longitudinal view evidence vector; Process the key frame sequence of the transverse view evidence vector and the longitudinal view evidence through the fusion view evidence network to obtain a fusion view evidence vector; Associate the transverse view evidence vector, the longitudinal view evidence vector, and the fusion view evidence vector with Dirichlet parameters to map to obtain an opinion based on the transverse view, an opinion based on the longitudinal view, and an opinion based on the fusion view. Combine the opinion based on the transverse view, the opinion based on the longitudinal view, and the opinion based on the fusion view , get the joint opinion , output the lesion benign and malignant classification probability and prediction uncertainty as the diagnosis result.

[0011] Preferably, the key frame sequence of the transverse view and the longitudinal view are processed by the transverse view evidence network and the longitudinal view evidence network respectively to obtain the transverse view evidence vector and the longitudinal view evidence vector, specifically including the following steps: The key frame sequence is encoded by the Swin Transformer encoder, and position embedding representation is added to retain time information to obtain the embedded representation of the key frame sequence; The embedded representation of the key frame sequence is input into the intra-view Transformer module based on the time self-attention mechanism to learn the time dependency within each view sequence, and a feature representation fused with the inter-frame time dependency within the view is obtained; The feature representation fused with the inter-frame time dependency within the view is input into the MLP module with ReLU activation layer to obtain a non-negative output as the evidence vector; the evidence vector includes the transverse view evidence vector and the longitudinal view evidence vector .

[0012] Preferably, the key frame sequence of the transverse view evidence vector and the longitudinal view evidence is processed by the fusion view evidence network to obtain the fusion view evidence vector, specifically including the following steps: The key frame sequence of the transverse view and the longitudinal view is encoded by the Swin Transformer encoder, and position embedding representation is added to retain time information to obtain the embedded representation of the key frame sequence; The embedded representation of the key frame sequence is input into the intra-view Transformer module based on the time self-attention mechanism to learn the time dependency within each view sequence, and a feature representation fused with the inter-frame time dependency within the view is obtained; The feature representation fused with the inter-frame time dependency within the view is input into the pre-feedback layer with ReLU activation layer to obtain the fusion view evidence vector .

[0013] Preferably, the transverse view evidence vector, the longitudinal view evidence vector and the fusion view evidence vector are associated with the Dirichlet parameter to map to obtain the transverse view-based opinion , the longitudinal view-based opinion and the fusion view-based opinion , specifically including the following steps: Given the video frame sequence of the th view , learn evidence vectors of one class ; determining parameters of Dirichlet distribution ; associating evidence vectors with Dirichlet parameters as , the Dirichlet distribution is mapped as opinion , obtaining opinion including opinion based on lateral perspective , opinion based on longitudinal perspective and opinion based on fusion perspective ; and satisfying condition , wherein is trust quality, is Dirichlet intensity; is predicted uncertainty, given video frame sequence of perspective , probability of sample being predicted as class c .

[0014] Preferably, combining opinion based on lateral perspective , opinion based on longitudinal perspective and opinion based on fusion perspective , obtaining joint opinion , outputting lesion benignity classification probability and prediction uncertainty as diagnostic result, specifically comprising the following steps: combining lateral perspective opinion and longitudinal perspective opinion to obtain , the specific calculation formula is as follows: ; ; wherein is a measure for normalizing the degree of conflict between the two opinions; combining and using Dempster combination rule to obtain joint opinion , obtaining probability of benignity classification of lesion based on joint perspective , wherein , also obtaining predicted uncertainty .

[0015] Preferably, for the single-perspective video frame sequence of the i-th sample, evidence is obtained from the evidence network, parameters of Dirichlet distribution are obtained by using subjective logic, and multinomial opinion is formed, wherein is class assignment probability on simplex determined by Dirichlet parameters . The cross-entropy loss adjusted based on the evidence-based method is: ; in It is a bivariate function. It is the unique hot label of sample i; A KL divergence term is introduced to regularize the evidence for mislabeled items, shrinking it to 0: ; in , It is a balancing factor; A multi-task strategy was adopted to further enhance the learning of joint opinions, and the overall loss function is: ; in This represents an overall loss.

[0016] This invention also provides a reliable breast cancer diagnostic system based on multi-view ultrasound video fusion, comprising: The data processing module is used to acquire breast ultrasound videos; The lesion segmentation module is used to input the breast ultrasound video frame by frame into the segmentation model for segmentation to obtain the lesion segmentation results; The keyframe extraction module is used to determine keyframes based on the lesion segmentation results; The lesion classification module is used to input keyframe sequences from horizontal and vertical perspectives into the lesion classification model, and output diagnostic results after multi-view fusion and uncertainty estimation.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention presents a reliable breast cancer diagnosis method based on multi-view ultrasound video fusion, which boasts high accuracy and reliability. It can automatically locate keyframes in ultrasound videos, fuse lesion features from multiple viewpoints, and provide accurate and reliable diagnostic results. This helps physicians make rapid and accurate diagnoses in clinical practice, thereby improving clinical efficiency. Experimental verification shows that this method effectively improves diagnostic accuracy while estimating the uncertainty of individual predictions, demonstrating high precision and high reliability in breast ultrasound video diagnosis tasks. Attached Figure Description

[0018] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0019] Figure 1 This is a schematic diagram of a reliable breast cancer diagnosis method based on multi-view ultrasound video fusion. Figure 2This is a flowchart of keyframe screening based on lesion size prediction using a segmentation model; Top-K strategy, K=8; Figure 3 This is a schematic diagram of uncertainty distribution analysis; Figure 4 This is a schematic diagram of a reliable breast cancer diagnostic system based on multi-view ultrasound video fusion. Detailed Implementation

[0020] Numerous specific details are set forth in the following description to provide a full understanding of the invention. However, the invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0021] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0022] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0023] The present invention will now be described in further detail with reference to the accompanying drawings: To better illustrate the technical effects of the present invention, the present invention provides the following specific embodiments to illustrate the above technical process: Example 1: A reliable method for breast cancer diagnosis based on multi-view ultrasound video fusion, such as... Figure 1 As shown, it includes the following steps: Step 1: Lesion identification and keyframe extraction in breast ultrasound video. The potential keyframes in the dynamic ultrasound video are automatically located by a segmentation-guided detection model, and the visible lesion areas in these keyframes are predicted. The aim is to filter out irrelevant scans and focus on potential keyframes containing lesion areas.

[0024] Step two: breast lesion reliable classification based on multi-view ultrasound video, the transverse and longitudinal key frame sequence containing the segmented lesion region is input into the transformer-based evidence multi-view classification framework for integration to achieve reliable breast lesion classification. The specific process is as follows: 1. Automatic lesion recognition and key frame extraction Since the breast ultrasound video usually contains redundant information, the lesion may only appear in a small range of video frames. Therefore, the present application first develops a lesion segmentation model based on Swin-Transformer-Unet to automatically detect breast lesions in ultrasound video frames. Specifically, the proposed segmentation model is composed of an encoder and a decoder, and is connected to each other through a skip connection. The encoder is constructed based on the widely used visual backbone network Swin Transformer, and the decoder generates a prediction map through upsampling operation. The training of the segmentation model includes two stages, first, pre-training using the annotated ultrasound images in the public BUSI dataset, and then further fine-tuning on the annotated ultrasound key frames in the BUSV dataset of the present application.

[0025] During the clinical dynamic scanning examination process, the sonographer identifies representative frames by moving the probe, then repeatedly scans adjacent frames to observe imaging features, and makes a diagnosis based on these key frames. Therefore, the present application automatically locates the key frames in any K video segments based on the size of the lesion region predicted by the segmentation model.

[0026] The present application proposes two key frame selection strategies for selecting candidate key frames from dynamic ultrasound video. Specifically, first, each breast ultrasound video is input frame by frame into the corresponding trained segmentation model to obtain the segmentation results of all frames. Then, the predicted lesion size of each frame in the video is calculated, i.e. , where is the number of frames in the video. Finally, considering that lesion size is an important indicator in diagnosis, the frame with the largest lesion area is considered as the key frame, i.e. .

[0027] To utilize the temporal dependency between frames, two key frame sequence extraction strategies are proposed for selecting candidate key frames from dynamic ultrasound video. The first strategy is the strategy, which selects another key frames around the "maximum frame" key frame, i.e. , where . The second is the strategy, which selects the key frames with the largest lesion area.

[0028] Through the above-described lesion segmentation and keyframe extraction process, this invention extracts two keyframe sequences with lesion segmentation results from ultrasound videos from transverse and longitudinal perspectives.

[0029] 2. Reliable classification of breast cancer lesions based on multi-view ultrasound video This invention proposes a TrustMBT (Trust Multi-Perspective) breast lesion classification model based on evidence theory. The model mainly comprises an evidence network, uncertainty estimation, and a combination rule. The evidence network aims to encode each perspective as an evidence vector. Specifically, for longitudinal and transverse perspectives, this invention designs a Transformer-based evidence network, utilizing a temporal self-attention mechanism to encode temporal dependencies into evidence vectors. This invention constructs a feature-level fusion perspective to fuse multi-perspective information at the feature level, integrating complementary information from two specific perspectives. Then, a fusion perspective evidence network based on a bi-branch visual Transformer and attention bottleneck is designed to encode the fusion perspective into evidence vectors. Regarding uncertainty estimation, subjective logic is applied to transform the aforementioned evidence vectors into parameters of a Dirichlet distribution, thus successfully learning the opinions specific to the two perspectives as well as the opinion of the fusion perspective. To enhance multi-perspective classification, this invention further integrates multi-perspective information at the evidence level. Using Dempster's combination rule, joint opinions are aggregated based on the opinions of all perspectives, thereby obtaining the final probability and overall uncertainty for each category.

[0030] (1) Obtaining single-perspective evidence through evidence networks Specifically, this invention designs a transverse perspective evidence network based on the Swing Transformer. Longitudinal perspective evidence network And a fusion perspective evidence network based on bottleneck attention mechanism This process is used to obtain the evidence vector for the corresponding viewpoint. Since the evidence networks for horizontal and vertical views are similar, the horizontal viewpoint is used as an example. The evidence network consists of three modules: a Swin Transformer encoder, an in-view Transformer block, and a multilayer perceptron with ReLU activation layers. Specifically, the encoder of the pre-trained segmentation model is first used to process the keyframe sequence of the horizontal viewpoint. Separate encodings are performed, and positional embeddings are added to preserve temporal information, resulting in embedded representations of the keyframe sequence. Then, it passes through a Transformer module based on a temporal self-attention mechanism to learn the temporal dependencies within each view sequence. Next, it is input into an MLP module using ReLU as the activation function to obtain a non-negative output, which serves as the evidence vector for the lateral view. = = ,in =2 indicates the number of categories. Similarly, the evidence vector from a longitudinal perspective... It can also be obtained in the same way.

[0031] Evidence Networks from a Fusion Perspective This invention designs an inter-view transformer based on a bottleneck attention mechanism to enhance multi-view learning. Compared to traditional pairwise self-attention mechanisms, the inter-view transformer aims to fuse complementary information from two viewpoints in a more efficient manner. Specifically, for the viewpoint features learned by the transformer within two specific viewpoints, all cross-modal attention flows are restricted to a small number of fusion bottlenecks. This requires the inter-view transformer to organize and compress relevant information from the two viewpoints and share necessary information. Similar to the evidence network for specific viewpoints, the classification labels of the two viewpoints are concatenated and passed to a pre-feedback layer with ReLU activation layers to obtain a fused viewpoint evidence vector. .

[0032] (2) Obtaining decision-making opinions through subjective logic This invention applies subjective logic to estimate the probability of prediction for each class and the uncertainty of the prediction. Specifically, given the first... A sequence of video frames from a single perspective First, learn through evidence networks. Evidence vectors of each category According to subjective logic, the parameters of the Dirichlet distribution... It can be determined. A simple way to associate the evidence vector with the Dirichlet parameters is as follows: Then the Dirichlet distribution can be mapped to opinions. And meet the conditions ,in It's about trust quality. It is the Dirichlet intensity. It is the uncertainty of prediction, given a perspective. Given a video frame sequence, the probability that a sample is predicted to be of class c. .

[0033] In the task of classifying benign and malignant breast lesions, opinions based on a lateral perspective Opinions based on a longitudinal perspective and opinions based on a fusion perspective All of these can be obtained through the methods described above.

[0034] (3) Dempster's combination rule for decision-making opinion fusion Dempster's combination rules are used to combine opinions from a lateral perspective. Opinions from a longitudinal perspective and integrated perspectives In order to obtain a joint opinion Specifically, firstly, opinions from a horizontal perspective... and opinions from a longitudinal perspective Combining to obtain The specific calculation formula is as follows: ; ; in It is a measure of the degree of conflict between two opinions used for standardization.

[0035] In the above text, the horizontal and vertical perspectives can be fused at the feature level to form a new fusion perspective, and then the opinions corresponding to the fusion perspective can be obtained through uncertainty estimation. Therefore, this invention further uses Dempster's combination rule to... and The group reached a joint opinion .

[0036] By using the Dempster combination rule twice, the lateral, longitudinal, and novel fusion perspectives were integrated at the evidentiary level, resulting in probabilities for classifying benign and malignant lesions based on this combined perspective. ,in It can also provide information about the uncertainty of predictions. .

[0037] (4) Learning paradigm The classification model can be trained end-to-end. Taking a single viewpoint as an example, for the video frame sequence of the i-th sample from a single viewpoint, evidence is obtained by an evidence network, and the parameters of the Dirichlet distribution are obtained using subjective logic to form a multinomial opinion. ,in It is determined by the Dirichlet parameters The probability of class assignment on a defined simplex. Compared to traditional softmax-based classifiers, the evidence-based method's adjusted cross-entropy loss is: ; in It is a bivariate function. This is the one-hot label of sample i. Furthermore, this invention introduces a KL divergence term to regularize the evidence of mislabeled samples, shrinking it to 0: ; in , It is a balancing factor.

[0038] This invention employs a multi-task strategy to further enhance the learning of joint opinions; therefore, the overall loss function is: .

[0039] The experimental tests and performance analysis of the above-mentioned reliable breast cancer diagnostic method based on multi-view ultrasound video fusion are as follows: 1. Experimental Dataset The dataset is the BUSV multi-view dynamic ultrasound video dataset, which includes 304 breast patients with pathologically confirmed benign or malignant lesions. All patients underwent dynamic breast ultrasound examinations, acquiring transverse and longitudinal ultrasound videos of 374 breast lesions (254 benign lesions and 120 malignant lesions). Each video contains one lesion, and all lesions have complete surgical pathology results. Dynamic ultrasound videos of all target lesions were acquired in both transverse and longitudinal sections, forming the BUSV multi-view dynamic ultrasound video dataset used for model development. To determine the regions of interest (ROIs) of the lesions, all ultrasound videos were manually annotated and cross-checked by two experienced sonographers. In addition, to train the lesion segmentation model, 5984 keyframes with annotated lesion boundaries were collected, and 780 breast ultrasound images with masked annotations were collected from the publicly available BUSI dataset for training the segmentation model.

[0040] 2. Performance evaluation indicators This invention uses the Dice similarity coefficient (DSC) to evaluate the performance of lesion segmentation models. In classification tasks, model performance is measured by AUC, ACC, precision, recall, and F1-score. The Mann-Whitney U test is used to compare the significance of uncertainty estimates between different groups.

[0041] 3. Evaluation of Experimental Results (1) Evaluation of the segmentation model Table 1 Performance of the breast lesion segmentation model

[0042] As shown in Table 1, the average Dice coefficient of the segmentation model pre-trained on the BUSI dataset is 78.18% ± 2.41%, and the median is 91.05% ± 0.53%. Compared with existing studies on breast lesion segmentation, the performance of the model in this invention is comparable to or even better. After fine-tuning using ultrasound keyframes labeled in the BUSI dataset, the average Dice coefficient of the model increased to 82.95% ± 1.82%, and the median was 89.20% ± 0.84%, indicating that the segmented regions of the model highly overlap with the regions labeled by experts.

[0043] (2) Keyframe extraction effect The segmentation model aims to detect breast lesions from ultrasound video frames and further assist in the selection of keyframes. Given that lesions only appear in a small number of frames of the ultrasound video, this invention employs a keyframe selection strategy based on predicted lesion size. Figure 2 The Top-K (K=8) keyframe selection strategy is demonstrated. The 8 frames with the largest predicted lesion area are selected from an ultrasound video as the keyframe set (marked in green). It can be found that one of the selected keyframes is exactly the keyframe marked by the ultrasound physician (marked in purple).

[0044] (3) The effect of breast cancer lesion classification The present invention compares the proposed multi-view classification model with traditional multi-view classification baseline methods. As shown in Table 2, the TrustMBT classification model proposed in this invention has an AUC of 84.99% and an ACC of 82.10%, which is better than all baseline models, proving the effectiveness of the decision-level and feature-level fusion strategy.

[0045] Table 2 Performance Comparison of Breast Cancer Classification Models

[0046] (4) Uncertainty distribution analysis To assess whether uncertainty estimation can convey reliable predictions for each lesion, this invention analyzes the uncertainty distribution of correctly and incorrectly classified samples, such as... Figure 3 As shown, the uncertainty of correctly classified samples is generally lower than that of misclassified samples.

[0047] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules, units, or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units, modules, or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0048] The units may or may not be physically separate. The components shown as units can be one or more physical units, meaning they can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0049] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0050] In particular, according to embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this invention. It should be noted that the computer-readable medium described above in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0051] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0052] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A reliable breast cancer diagnosis method based on multi-view ultrasound video fusion, characterized in that, The method comprises the following steps: acquiring a breast ultrasound video; the breast ultrasound video comprises ultrasound videos of a transverse view and a longitudinal view; inputting the breast ultrasound video frame by frame into a segmentation model for segmentation to obtain a lesion segmentation result; determining a key frame according to the lesion segmentation result; the key frame comprises a key frame sequence of the transverse view and the longitudinal view; inputting the key frame sequence of the transverse view and the longitudinal view into a lesion classification model, performing multi-view fusion and uncertainty estimation, and outputting a diagnosis result, which comprises a benign and malignant classification result of the lesion and a prediction uncertainty. 2.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 1, characterized in that, According to the lesion segmentation result, the key frame is determined, specifically comprising the following steps: calculating the predicted lesion area size of the lesion segmentation result; adopting a Near-K strategy or a Top-K strategy to screen K candidate key frames from the ultrasound videos of the transverse view and the longitudinal view based on the lesion area, as the key frame sequence; The Near-K strategy is as follows: select the video frame with the largest lesion area, and select [missing information] in the vicinity of that frame. These keyframes together form a keyframe sequence. The Top-K strategy is: selecting a key frame with the largest lesion area as a key frame sequence. the Top-K strategy is: selecting a key frame with the largest lesion area as a key frame sequence. 3.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 2, characterized in that, The lesion classification model comprises a transverse view evidence network, a longitudinal view evidence network based on a Swin Transformer, and a fusion view evidence network based on a bottleneck attention mechanism; The transverse view evidence network, the longitudinal view evidence network and the fusion view evidence network are all composed of a Swin Transformer encoder, an intra-view Transformer module and an MLP module with a ReLU activation layer. 4.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 3, characterized in that, Inputting the key frame sequence of the transverse view and the longitudinal view into the lesion classification model, performing multi-view fusion and uncertainty estimation, and outputting a diagnosis result, specifically comprising the following steps: processing the key frame sequence of the transverse view and the longitudinal view through the transverse view evidence network and the longitudinal view evidence network respectively to obtain a transverse view evidence vector and a longitudinal view evidence vector; processing the key frame sequence of the transverse view evidence vector and the longitudinal view evidence through the fusion view evidence network to obtain a fusion view evidence vector; mapping the lateral perspective evidence vector, the longitudinal perspective evidence vector, and the fused perspective evidence vector to Dirichlet parameters to obtain a lateral perspective-based opinion , a longitudinal perspective-based opinion , and a fused perspective-based opinion ; combining lateral-view-based opinions , longitudinal-view-based opinions and fusion-view-based opinions to obtain a joint opinion and output the lesion benignity / malignancy classification probability and prediction uncertainty as the diagnosis result. 5.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 4, characterized in that, Processing the key frame sequence of the transverse view and the longitudinal view through the transverse view evidence network and the longitudinal view evidence network respectively to obtain a transverse view evidence vector and a longitudinal view evidence vector, specifically comprising the following steps: encoding the key frame sequence using a Swin Transformer encoder while adding position embedding representation to preserve time information to obtain an embedded representation of the key frame sequence; the embedded representation of the key frame sequence is processed through an intra-view Transformer module based on a time self-attention mechanism to learn the time dependence within each view sequence, to obtain a feature representation fused with the intra-view inter-frame time dependence; inputting the feature representation fused with intra-view inter-temporal dependency into a MLP module with a ReLU activation layer to obtain a non-negative output as an evidence vector; the evidence vector includes a horizontal-view evidence vector and a longitudinal-view evidence vector . 6.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 5, wherein, processing the key frame sequence of the transverse view evidence vector and the longitudinal view evidence through the fusion view evidence network to obtain a fusion view evidence vector, specifically comprising the following steps: encoding the key frame sequence of the transverse view and the longitudinal view using a Swin Transformer encoder while adding position embedding representation to preserve time information to obtain an embedded representation of the key frame sequence; The embedded representation of the key frame sequence is passed through an intra-view Transformer module based on a time self-attention mechanism to learn the time dependencies within each view sequence, obtaining a feature representation that fuses the intra-view inter-frame time dependencies; The feature representation fused with the intra-view inter-time dependency is input to a pre-feedback layer with a ReLU activation layer to obtain a fused view evidence vector . 7.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 6, characterized in that, mapping the transverse-view evidence vector, the longitudinal-view evidence vector and the fusion-view evidence vector to Dirichlet parameters to obtain a transverse-view-based opinion , a longitudinal-view-based opinion and a fusion-view-based opinion , specifically comprising the following steps: Given a sequence of video frames of a first viewpoint , a vector of evidence is learned for each of a plurality of classes by an evidence network Determining parameters of a dirichlet distribution ; The evidence vector is linked to the Dirichlet parameters as The Dirichlet distribution is mapped to the opinion The opinion includes a lateral view-based opinion a longitudinal view-based opinion and a fusion view-based opinion ; and satisfies the condition where is the trust quality, is the Dirichlet strength; is the predicted uncertainty, given the video frame sequence of views, the probability that the sample is predicted to be of class c . 8.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 7, characterized in that, combining the lateral-view-based opinion the longitudinal-view-based opinion and the fusion-view-based opinion to obtain a joint opinion outputting the lesion benignity / malignancy classification probability and prediction uncertainty as a diagnosis result, specifically comprising the following steps: combining the lateral perspective opinion and the longitudinal perspective opinion yields , the specific calculation formula is as follows: ； ； wherein is a measure of the degree of conflict between the two opinions for normalization; using the Dempster combination rule and to obtain a joint opinion , obtaining a probability of a classification of the lesion as malignant or benign based on the joint perspective wherein also obtaining an uncertainty of the prediction . 9.The reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to claim 8, characterized in that, For the single-view video frame sequence of the i-th sample, the evidence is obtained by the evidence network, and the parameters of the Dirichlet distribution are obtained by using the subjective logic to form a multinomial opinion wherein is the category assignment probability on the simplex determined by the Dirichlet parameters ​ The adjusted cross-entropy loss of the evidence-based method is: ； wherein is a binary function, is a one-hot label for sample i; A KL divergence term is introduced to regularize the evidence of the wrong label, so that it shrinks to 0: ； wherein , is the balance factor; A multi-task strategy is adopted to further improve the learning of joint opinions, and the overall loss function is: ； wherein is the overall loss.

10. A reliable breast cancer diagnosis system based on multi-view ultrasound video fusion, for implementing the reliable breast cancer diagnosis method based on multi-view ultrasound video fusion according to any one of claims 1-9, characterized in that, It includes: The data processing module is used to acquire the breast ultrasound video; The lesion segmentation module is used to input the breast ultrasound video frame by frame into the segmentation model for segmentation to obtain a lesion segmentation result; The key frame extraction module is used to determine the key frame according to the lesion segmentation result; The lesion classification module is used to input the key frame sequence of the transverse view and the longitudinal view into the lesion classification model, perform multi-view fusion and uncertainty estimation, and output a diagnosis result.

Images