A breast lump full-automatic screening method and system based on multi-parameter ultrasonic tomography and multi-example learning

Abstract

This invention belongs to the field of medical imaging artificial intelligence and discloses a fully automated method and system for screening breast masses based on multi-parameter ultrasound tomography and multi-example learning. The method includes the following steps: S1: Prepare multi-parameter data to obtain input patch sequences; S2: Construct and train a first deep learning network; S3: Construct and train a second deep learning network; S4: Input the input patch sequence of the breast to be screened into the trained first deep learning network to obtain k key slices; then, input these k key slices into the trained second deep learning network to obtain the classification result of whether the breast to be screened has a mass or not, thus achieving automatic screening. This invention achieves efficient and accurate breast screening by integrating multi-parameter ultrasound features with a two-stage deep learning architecture, effectively solving the problems of low sensitivity in breast screening and reliance on operator experience in conventional ultrasound.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging artificial intelligence, and more specifically, relates to a fully automated screening method and system for breast masses based on multi-parameter ultrasound tomography and multi-example learning. It utilizes three-dimensional ultrasound tomography (USCT) multi-parameter imaging technology, combined with deep learning algorithms, to perform fully automated and standardized screening and assisted diagnosis of breast masses, which is particularly suitable for large-scale screening of people with dense breasts. Background Technology

[0002] Early screening for breast cancer is crucial for reducing mortality. Current standard screening methods, such as mammography, have poor penetration in dense breasts (common in Asian women), with a sensitivity of only about 58%. While handheld ultrasound can improve sensitivity as a supplement, it suffers from significant operator dependence, long examination times, and poor repeatability, making it unsuitable for large-scale standardized screening. Automated breast volume ultrasound imaging reduces operator dependence, but currently lacks effective automated screening algorithms targeting the entire breast.

[0003] Ultrasound computed tomography (USCT) provides standardized, radiation-free 3D whole-breast imaging, simultaneously acquiring multi-parameter data on reflection (anatomical structure) and transmission (sound velocity, attenuation, and other functional information). However, the sheer volume and complexity of USCT image data, coupled with clinicians' lack of experience in interpreting these images, limits its widespread adoption. Existing artificial intelligence research largely focuses on segmenting or classifying known lesions, lacking an effective approach to automatically discovering and screening masses "from scratch" using whole-breast scan data. Therefore, there is an urgent clinical need to develop a fully automated screening system that can fully utilize the multi-parameter information from USCT and possesses high generalization capabilities. Summary of the Invention

[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, the present invention aims to provide a fully automated breast mass screening method and system based on multi-parameter ultrasound tomography and multi-example learning. Simultaneously, by integrating multi-parameter ultrasound features with a two-stage deep learning architecture, and based on reflective modal images reflecting tissue anatomical information and sound velocity modal images reflecting tissue physical properties, this invention achieves efficient and accurate breast mass screening. This invention effectively solves the problems of low sensitivity in breast screening and reliance on operator experience in conventional ultrasound, achieving standardized, high-precision automated screening, and realizing automated breast mass screening based on AI-assisted USCT.

[0005] To achieve the above objectives, according to a first aspect of the present invention, a fully automated screening method for breast masses based on multi-parameter ultrasound tomography and multi-example learning is provided, characterized by comprising the following steps: S1: Prepare multi-parameter data, including the following sub-steps: S1-1: Collect three-dimensional ultrasound computed tomography (USCT) data of several breasts tested with known mass / no mass screening results and breasts to be screened; among which, For any breast tissue being tested: the 3D USCT data simultaneously includes the initial 3D USCT reflectance image volume and the initial 3D USCT sound velocity image volume of the entire breast; the initial 3D USCT reflectance image volume is composed of an initial USCT reflectance image slice sequence, and the initial 3D USCT sound velocity image volume is composed of an initial USCT sound velocity image slice sequence; the total number of slices in the initial 3D USCT reflectance image volume is the same as the total number of slices in the initial 3D USCT sound velocity image volume; Furthermore, for the three-dimensional USCT data of several breasts with known masses, the location information of the corresponding masses is also known; S1-2: For any breast tissue being tested: Spatially align all three-dimensional USCT data so that the length and width of any USCT reflection image slice are the same as the length and width of any USCT sound velocity image slice, and perform pixel value normalization processing to obtain the normalized USCT reflection image slice sequence and the normalized USCT sound velocity image slice sequence. S1-3: For any breast tissue being tested: extract the glandular tissue region of each slice as the region of interest from the normalized USCT reflectance image slice sequence, and obtain a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. S1-4: For any breast tissue being tested: The normalized USCT reflectance image slice sequence, the normalized USCT sound velocity image slice sequence, and the binary glandular tissue mask image sequence are fused in the channel dimension to obtain the input patch sequence. S2: Construct a lesion detection network as the first deep learning network. The first deep learning network includes a first backbone feature extraction module, a feature pyramid for constructing multi-scale feature representation, and a detection output module connected in sequence. This network can take the input patch sequence of any breast tissue being tested as input, perform a full-volume scan on the input patch sequence, extract multi-scale features using the network, and generate slice-level candidate lesion confidence scores. Based on the confidence scores, the k input patches with the highest confidence scores are selected as k key slices; where k is a preset positive number greater than 1. The first deep learning network is trained using the input patch sequence corresponding to each of the tested breasts with known screening results of having / not having lumps. The training label is the location of the corresponding lump, thus obtaining the trained first deep learning network. S3: Construct a Multiple Instant Learning (MIL) classification network as the second deep learning network. This second deep learning network employs the MIL framework and includes a second backbone feature extraction module for extracting key slice features, an attention weight calculation module for calculating the weights of each key slice, a feature aggregation module for weighted summation of key slice features, and a binary classification output module for outputting breast cancer screening results. This network can simultaneously use the k key slices as input. The attention weight calculation module automatically learns the weight of each key slice for the final diagnosis using an attention mechanism. The second backbone feature extraction module extracts corresponding features from the key slices. The network uses the feature aggregation module to weighted summation of the features obtained from the second backbone feature extraction module and the weights obtained from the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module classifies the global feature vectors and outputs the classification result of whether the tested breast has a lump or not. The second deep learning network is trained using k key slices corresponding to each breast in several breasts with known screening results of having / not having lumps. The training label is the screening results of having / not having lumps in the breasts, resulting in the trained second deep learning network. S4: Input the input patch sequence of the breast to be screened into the trained first deep learning network to obtain k key slices corresponding to the breast to be screened; then, input these k key slices into the trained second deep learning network to obtain the classification result of whether the breast to be screened has a mass or not, thus realizing the automatic screening of the breast to be screened.

[0006] As a further preferred embodiment of the present invention, in steps S1-4, fusion is performed at the channel dimension, specifically as follows: The images, including the i-th slice in the normalized USCT reflectance image slice sequence, the i-th slice in the normalized USCT sound velocity image slice sequence, and the i-th glandular tissue mask in the binary glandular tissue mask sequence, are stitched together to form a multi-channel input patch. The input patch uses the i-th slice in the normalized USCT reflectance image slice sequence as the current slice. Where i is 1, ..., N, and N is the total number of slices in the volume of the three-dimensional USCT reflection image; Thus, an input patch sequence consisting of N input patches is obtained; Preferably, in steps S1-4, the context information of the current slice is also incorporated when performing fusion at the channel level, specifically: The images, including the (ij), ..., i, ..., (i+j)th slices from the normalized USCT reflectance image slice sequence, the (ij), ..., i, ..., (i+j)th slices from the normalized USCT sound velocity image slice sequence, and the i-th glandular tissue mask image from the binary glandular tissue mask image sequence, are stitched together to form a multi-channel input patch. This input patch sequence uses the i-th slice from the normalized USCT reflectance image slice sequence as the current slice. Where j is a preset positive integer, i is (1+j), ..., (Nj), and N is the total number of slices of the three-dimensional USCT reflection image volume; Thus, an input patch sequence consisting of (N-2j) input patches is obtained; More preferably, j is 1.

[0007] According to a second aspect of the present invention, a fully automated screening method for breast masses based on multi-parameter ultrasound tomography and multi-example learning is provided, characterized by comprising the following steps: (1): Prepare multi-parameter data, including the following sub-steps: (1-1): Three-dimensional ultrasound computed tomography (USCT) data were collected from several breasts with known mass / no mass screening results and breasts to be screened; among them, For any breast tissue being tested: the 3D USCT data simultaneously includes the initial 3D USCT reflectance image volume and the initial 3D USCT sound velocity image volume of the entire breast; the initial 3D USCT reflectance image volume is composed of an initial USCT reflectance image slice sequence, and the initial 3D USCT sound velocity image volume is composed of an initial USCT sound velocity image slice sequence; the total number of slices in the initial 3D USCT reflectance image volume is the same as the total number of slices in the initial 3D USCT sound velocity image volume; Furthermore, for the three-dimensional USCT data of several breasts with known masses, the location information of the corresponding masses is also known; (1-2): For any breast tissue being tested: All three-dimensional USCT data are spatially aligned so that the length and width of any USCT reflection image slice are the same as the length and width of any USCT sound velocity image slice, and pixel values ​​are normalized to obtain the normalized USCT reflection image slice sequence and the normalized USCT sound velocity image slice sequence. (1-3): For any breast tissue being tested: extract the glandular tissue region of each slice from the normalized USCT reflectance image slice sequence as the region of interest, and obtain a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. (1-4): For any breast tissue being tested: the normalized USCT reflectance image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel reflectance input data sequence; the normalized USCT sound velocity image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel sound velocity input data sequence. (2): Construct a lesion detection network as the first deep learning network, including a third feature extraction network, a fourth feature extraction network, a first cross-modal computation module, a feature pyramid module for constructing multi-scale feature representations, and a detection output module. The third feature extraction network is used to form a multi-scale feature representation of a reflection mode for each multi-channel reflection input data in the multi-channel reflection input data sequence, thereby obtaining a sequence of multi-scale feature representations of the reflection mode. The fourth feature extraction network is used to form a multi-scale feature representation of a sound speed mode for each multi-channel sound speed input data in the multi-channel sound speed input data sequence, thereby obtaining a sequence of multi-scale feature representations of the sound speed mode. The multi-scale feature representations of the reflection mode in the sequence of multi-scale feature representations of the reflection mode correspond one-to-one with the multi-scale feature representations of the sound speed mode in the sequence of multi-scale feature representations of the sound speed mode. For each corresponding reflection mode... The first cross-modal computation module uses the multi-scale feature representation of the reflection modality as the query vector Q, and the multi-scale feature representation of the sound velocity modality as the key vector K and value vector V. Through cross-attention computation, a fused cross-modal feature representation is obtained, thus yielding a sequence of cross-modal feature representations. The output of the first cross-modal computation module is connected to the detection output module via a feature pyramid module for constructing the multi-scale feature representation. The first deep learning network can perform a full-volume scan of the cross-modal feature representation sequence based on any tested breast tissue. It extracts multi-scale features using the network and generates slice-level candidate lesion confidence scores. Based on the confidence scores, it selects the k cross-modal feature representations with the highest confidence scores as k key cross-modal feature representations, where k is a preset positive number greater than 1. The first deep learning network is trained using the sequence of multi-channel reflection input data and the sequence of multi-channel sound velocity input data corresponding to each of the several breasts tested with known screening results of having / not having lumps. The training label is the location of the corresponding lump, thereby obtaining the trained first deep learning network. (3): Construct a multi-instance learning (MIL) classification network as the second deep learning network. This network adopts the MIL framework and includes a fifth feature extraction network, a sixth feature extraction network, a second cross-modal computation module, an attention weight calculation module for calculating the weights of each key cross-modal feature representation, a feature aggregation module for weighted summation of the key cross-modal feature representations, and a binary classification output module for outputting breast cancer screening results. This second deep learning network can simultaneously take the k multi-channel reflection input data and k multi-channel sound velocity input data corresponding to the k key cross-modal feature representations as input. The fifth feature extraction network is used to calculate the weights of each key cross-modal feature representation. Multi-channel reflection input data are used to form multi-scale feature representations of reflection modes, thereby obtaining k multi-scale features of reflection modes. The sixth feature extraction network is used to form multi-scale feature representations of sound speed modes from these k multi-channel sound speed input data, thereby obtaining k multi-scale features of sound speed modes. The multi-scale feature representations of the k reflection modes correspond one-to-one with the multi-scale feature representations of the k sound speed modes. For a corresponding multi-scale feature representation of one reflection mode and one multi-scale feature representation of one sound speed mode, the second cross-modal calculation module is used to use the multi-scale feature representation of the reflection mode as the query vector Q and the multi-scale feature representation of the sound speed mode as the key vector K. The sum vector V is used to obtain a fused cross-modal feature representation through cross-attention calculation, thereby obtaining k cross-modal feature representations. The attention weight calculation module automatically learns the weight of each of the k cross-modal feature representations for the final diagnosis using the attention mechanism. The network uses the feature aggregation module to perform a weighted summation of the cross-modal feature representations with the weights obtained by the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module is used to classify the global feature vectors and output the classification result of whether the tested breast has a lump or not. The second deep learning network is trained using k key cross-modal features corresponding to each breast in several breasts with known screening results of having / not having lumps, along with k multi-channel reflection input data and k multi-channel sound velocity input data. The training label is the screening results of having / not having lumps in the breasts being tested, resulting in the trained second deep learning network. (4): Input the sequence of multi-channel reflection input data and the sequence of multi-channel sound velocity input data of the breast to be screened into the trained first deep learning network to obtain k key cross-modal feature representations corresponding to the breast to be screened; then, input the k multi-channel reflection input data and k multi-channel sound velocity input data corresponding to the k key cross-modal feature representations into the trained second deep learning network to obtain the classification result of whether the breast to be screened has a mass or not, thereby realizing the automatic screening of the breast to be screened.

[0008] As a further preferred embodiment of the present invention, steps (1-4) specifically include: The image, including the i-th slice in the normalized USCT reflectance image slice sequence and the i-th glandular tissue mask in the binary glandular tissue mask sequence, is stitched together to form a multi-channel reflectance input data; where i is 1, ..., N, and N is the total number of slices in the three-dimensional USCT reflectance image volume; thus, a multi-channel reflectance input data sequence including N multi-channel reflectance input data is obtained. The image, including the i-th slice from the normalized USCT sound velocity image slice sequence, is stitched together with the i-th glandular tissue mask from the binary glandular tissue mask image sequence to form a multi-channel sound velocity input data; where i is 1, ..., N, and N is the total number of slices in the three-dimensional USCT sound velocity image volume; thus, a multi-channel sound velocity input data sequence including N multi-channel reflection input data is obtained; Preferably, in steps (1-4), when performing fusion at the channel level, the context information of the current slice is also taken into account, specifically: The images, including the (ij), ..., i, ..., (i+j)th slices from the normalized USCT reflectance image slice sequence and the i-th glandular tissue mask from the binary glandular tissue mask sequence, are stitched together to form a single multi-channel reflectance input data; where j is a preset positive integer, i is (1+j), ..., (Nj), and N is the total number of slices in the three-dimensional USCT reflectance image volume; thus, a multi-channel reflectance input data sequence containing (N-2j) multi-channel reflectance input data is obtained. The images, including the (ij), ..., i, ..., (i+j)th slices from the normalized USCT sound velocity image slice sequence and the i-th glandular tissue mask from the binary glandular tissue mask image sequence, are stitched together to form a multi-channel sound velocity input data; where j is a preset positive integer, i is (1+j), ..., (Nj), and N is the total number of slices in the three-dimensional USCT sound velocity image volume; thus, a multi-channel sound velocity input data sequence containing (N-2j) multi-channel sound velocity input data is obtained. More preferably, j is 1.

[0009] As a further preferred embodiment of the present invention, in step S2 or step (2), k is preferably 5; The first deep learning network uses a random initialization method; Wherein, when the first deep learning network includes a backbone feature extraction module, the backbone feature extraction module adopts a network based on the Transformer architecture or a network based on a convolutional neural network, preferably a network based on the Swing Transformer; when the first deep learning network includes a third feature extraction network and a fourth feature extraction network, the third feature extraction network and the fourth feature extraction network are independently selected from: a network based on the Transformer architecture and a network based on a convolutional neural network. The detection output module uses Faster R-CNN.

[0010] As a further preferred embodiment of the present invention, the second backbone feature extraction module in the second deep learning network in step S3 has the same structure as the first backbone feature extraction module in the first deep learning network in step S2; preferably, the second backbone feature extraction module reuses the network parameters of the first backbone feature extraction module in the first deep learning network trained in step S2 as the initial weights. In step (3), the fifth feature extraction network in the second deep learning network has the same structure as the third feature extraction network in the first deep learning network in step (2); in step (3), the sixth feature extraction network in the second deep learning network has the same structure as the fourth feature extraction network in the first deep learning network in step (2); preferably, the fifth feature extraction network reuses the network parameters of the third feature extraction network in the first deep learning network trained in step (2) as initial weights; the sixth feature extraction network reuses the network parameters of the fourth feature extraction network in the first deep learning network trained in step (2) as initial weights.

[0011] As a further preferred embodiment of the present invention, in step S1-1 or step (1-1), for several breasts with known screening results of having / not having a mass, the volume of their three-dimensional USCT reflective image is also marked with the lesion boundary box. In step S2 or step (2), the first deep learning network can generate the bounding box coordinates of the corresponding candidate lesion while generating the confidence score of the candidate lesion at the slice level. Furthermore, if step S4 or step (4) yields a classification result indicating that the breast to be screened has a mass, then the fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning further includes step S5 or step (5): Clustering is performed on all candidate lesion bounding box coordinates obtained in step S2 to obtain the bounding box coordinates of the three-dimensional lesion. Then, based on the bounding box coordinates of the three-dimensional lesion, the volume of the initial three-dimensional USCT reflection image is cropped in three dimensions to obtain a three-dimensional region of interest (ROI) containing the target lesion. Then, the three-dimensional ROI is resampled or scaled to a preset length, width and height size, and then input into a 3D ResNet18 convolutional neural network for benign / malignant binary classification, thereby obtaining the classification result of benign / malignant mass.

[0012] As a further preferred embodiment of the present invention, the fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning further includes a step of displaying overlaid heat maps: Based on the trained second deep learning network, a heatmap is generated using class activation mapping technology and superimposed on the initial 3D USCT reflection image volume to prompt the user for review.

[0013] According to a third aspect of the present invention, a fully automated breast mass screening system based on multi-parameter ultrasound tomography and multi-example learning is provided. The system is characterized by its ability to obtain classification results of whether the breast to be screened has a mass or not, thereby achieving automated screening of the breast to be screened. The system includes the following functional modules: The multi-parameter data preprocessing module is used to spatially align all three-dimensional ultrasound computed tomography (USCT) data of the breast to be screened, ensuring that the length and width dimensions of any USCT reflectance image slice are the same as those of any USCT sound velocity image slice. Pixel value normalization is then performed to obtain normalized USCT reflectance image slice sequences and normalized USCT sound velocity image slice sequences. The three-dimensional USCT data simultaneously includes the initial three-dimensional USCT reflectance image volume and the initial three-dimensional USCT sound velocity image volume of the entire breast. The initial three-dimensional USCT reflectance image volume is composed of the initial USCT reflectance image slice sequence, and the initial three-dimensional USCT sound velocity image volume is composed of the initial USCT sound velocity image slice sequence. The total number of slices in the initial three-dimensional USCT reflectance image volume is the same as the total number of slices in the initial three-dimensional USCT sound velocity image volume. The glandular tissue region of each slice is extracted as the region of interest from the normalized USCT reflectance image slice sequence, resulting in a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. The normalized USCT reflection image slice sequence, the normalized USCT sound velocity image slice sequence, and the binary gland tissue mask image sequence are fused in the channel dimension to obtain the input patch sequence. The first deep learning network functional module includes a first backbone feature extraction module, a feature pyramid for constructing multi-scale feature representations, and a detection output module connected in sequence. This network can take the input patch sequence of the tested breast as input, perform a full-volume scan on the input patch sequence, extract multi-scale features using the network, generate slice-level candidate lesion confidence scores, and select the k input patches with the highest confidence scores as k key slices based on the confidence scores; where k is a preset positive number greater than 1. The first deep learning network functional module was trained, and the training label was the location of the corresponding mass in each of the tested breasts with known mass / no mass screening results. The second deep learning network functional module: The second deep learning network adopts the Multiple Instant Learning (MIL) framework, including a second backbone feature extraction module for extracting key slice features, an attention weight calculation module for calculating the weights of each key slice, a feature aggregation module for weighted summation of key slice features, and a binary classification output module for outputting breast cancer screening results. This network can simultaneously use the k key slices as input, and through the attention weight calculation module, automatically learn the weight of each key slice for the final diagnosis using an attention mechanism. The second backbone feature extraction module extracts corresponding features from the key slices. The network uses the feature aggregation module to weighted summation of the features obtained from the second backbone feature extraction module and the weights obtained from the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module classifies the global feature vectors and outputs the classification result of whether the tested breast has a lump or not. The second deep learning network functional module was trained, and the training labels were the screening results of lumps / no lumps in several tested breasts with known lumps / no lumps.

[0014] According to a fourth aspect of the present invention, a fully automated breast mass screening system based on multi-parameter ultrasound tomography and multi-example learning is provided. The system is characterized by its ability to obtain classification results of whether the breast to be screened has a mass or not, thereby achieving automated screening of the breast to be screened. The system includes the following functional modules: The multi-parameter data preprocessing module is used to spatially align all three-dimensional ultrasound computed tomography (USCT) data of the breast to be screened, ensuring that the length and width dimensions of any USCT reflectance image slice are the same as those of any USCT sound velocity image slice. Pixel value normalization is then performed to obtain normalized USCT reflectance image slice sequences and normalized USCT sound velocity image slice sequences. The three-dimensional USCT data simultaneously includes the initial three-dimensional USCT reflectance image volume and the initial three-dimensional USCT sound velocity image volume of the entire breast. The initial three-dimensional USCT reflectance image volume is composed of the initial USCT reflectance image slice sequence, and the initial three-dimensional USCT sound velocity image volume is composed of the initial USCT sound velocity image slice sequence. The total number of slices in the initial three-dimensional USCT reflectance image volume is the same as the total number of slices in the initial three-dimensional USCT sound velocity image volume. The glandular tissue region of each slice is extracted as the region of interest from the normalized USCT reflectance image slice sequence, resulting in a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. The normalized USCT reflectance image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel reflectance input data sequence; the normalized USCT sound velocity image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel sound velocity input data sequence. The first deep learning network functional module includes a third feature extraction network, a fourth feature extraction network, a first cross-modal computation module, a feature pyramid module for constructing multi-scale feature representations, and a detection output module. The third feature extraction network is used to form a multi-scale feature representation of one reflection mode for each multi-channel reflection input data in the multi-channel reflection input data sequence, thereby obtaining a sequence of multi-scale feature representations of the reflection modes. The fourth feature extraction network is used to form a multi-scale feature representation of one sound speed mode for each multi-channel sound speed input data in the multi-channel sound speed input data sequence, thereby obtaining a sequence of multi-scale feature representations of the sound speed modes. The multi-scale feature representations of the reflection modes in the sequence of multi-scale feature representations of the reflection modes correspond one-to-one with the multi-scale feature representations of the sound speed modes in the sequence of multi-scale feature representations of the sound speed modes. For each corresponding multi-scale feature representation of one reflection mode and one multi-scale feature representation of one sound speed mode, the first cross-modal computation module uses the multi-scale feature representation of the reflection mode as the query vector Q and the multi-scale feature representation of the sound speed mode as the key vector K. The sum vector V is used to obtain a fused cross-modal feature representation through cross-attention computation, thereby obtaining a sequence of cross-modal feature representations. The output of the first cross-modal computation module is connected to the detection output module through a feature pyramid module for constructing multi-scale feature representations. The first deep learning network can perform a full-volume scan on the sequence of cross-modal feature representations of the tested breast, extract multi-scale features using the network, and generate slice-level candidate lesion confidence scores. Based on the confidence scores, the k cross-modal feature representations with the highest confidence scores are selected as k key cross-modal feature representations, where k is a preset positive number greater than 1. The first deep learning network functional module was trained, and the training label was the location of the corresponding mass in each of the tested breasts with known mass / no mass screening results. The second deep learning network functional module: The second deep learning network adopts the Multiple Instant Learning (MIL) framework, including a fifth feature extraction network, a sixth feature extraction network, a second cross-modal computation module, an attention weight calculation module for calculating the weights of each key cross-modal feature representation, a feature aggregation module for weighted summation of the key cross-modal feature representations, and a binary classification output module for outputting breast cancer screening results; this second deep learning network can simultaneously take k multi-channel reflection input data and k multi-channel sound velocity input data corresponding to the k key cross-modal feature representations as input, wherein the fifth feature extraction network is used to process these k multi-channel... The reflected input data are used to form multi-scale feature representations of the reflection modes, thereby obtaining k multi-scale features of the reflection modes. The sixth feature extraction network is used to form multi-scale feature representations of the sound speed modes from these k multi-channel sound speed input data, thereby obtaining k multi-scale features of the sound speed modes. The multi-scale feature representations of the k reflection modes correspond one-to-one with the multi-scale feature representations of the k sound speed modes. For a corresponding multi-scale feature representation of one reflection mode and one multi-scale feature representation of one sound speed mode, the second cross-modal calculation module is used to use the multi-scale feature representation of the reflection mode as the query vector Q and the multi-scale feature representation of the sound speed mode as the key vector K. The sum vector V is used to obtain a fused cross-modal feature representation through cross-attention calculation, thereby obtaining k cross-modal feature representations. The attention weight calculation module automatically learns the weight of each of the k cross-modal feature representations for the final diagnosis using the attention mechanism. The network uses the feature aggregation module to perform a weighted summation of the cross-modal feature representations with the weights obtained by the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module is used to classify the global feature vectors and output the classification result of whether the tested breast has a lump or not. The second deep learning network functional module was trained, and the training labels were the screening results of lumps / no lumps in several tested breasts with known lumps / no lumps.

[0015] Compared with the prior art, the present invention can achieve the following beneficial effects through the above-described technical solutions: (1) Improve breast screening performance: It combines the multi-parameter information of reflection (structure) and sound velocity (tissue hardness) of USCT, and introduces a fibroglandular tissue mask to suppress fat background noise (because the mass will only appear in the gland and at the gland boundary), which significantly improves the detection sensitivity (≥81%, as shown in Table 1 below) and specificity (≥84%, as shown in Table 1 below) in the breast.

[0016] (2) Full-process automation and standardization: The entire process from whole breast scan to diagnostic conclusion can be completed without human intervention, eliminating the operator dependence of conventional ultrasound and facilitating large-scale promotion.

[0017] (3) This invention can utilize three-dimensional ultrasound computed tomography (USCT) data of the breast to be screened (including both the three-dimensional USCT reflection image volume and the three-dimensional USCT sound velocity image volume of the whole breast; that is, it uses both the reflection modal image reflecting tissue anatomical information and the sound velocity image reflecting tissue physical properties) to achieve fully automated screening for the presence or absence of masses in the breast to be screened. Furthermore, for positive cases (i.e., breasts with masses), this invention can further crop out the three-dimensional region of interest of the lesion and input it into an existing three-dimensional convolutional neural network with benign and malignant classification function to classify benign and malignant diseases, thereby achieving benign and malignant diagnosis.

[0018] (4) High interpretability: Through MIL’s attention weights and heatmaps, the slices and spatial locations of lesions can be accurately located, providing users (especially doctors) with intuitive verification basis. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall architecture of the breast lump screening model based on multi-instance learning in Example 1.

[0020] Figure 2 This is a visual explanation of the model decision-making process in Embodiment 2 of the present invention; wherein, Figure 2 In the diagram, A, B, C, and D correspond to the current slices of k key slices (k=5) from four representative cases (including benign and malignant masses). For each case: the top numerical bars Slice and Attn represent the slice number (slice numbering in 3D USCT data skips the first and last two slices, and for the remaining slices, it starts from 0 and increases sequentially from the first to the last) and the attention weight assigned to that slice in the multi-instance learning framework, where the attention weight Attn indicates the slice's contribution to the global diagnosis; the middle grayscale image is the original USCT reflectance slice, with the red box indicating the true location of the lesion; the bottom color image is the corresponding class activation heatmap, with warm-colored areas representing highly correlated areas identified by the model (as a supplementary explanation: Figure 2 The display of the decimal point for Attn in C is limited, so the sum of Attn for these 5 slices is actually 1; Figure 2 The same applies to D in the middle.

[0021] Figure 3 The area under the receiver operating characteristic curve (AUC) is the area under the receiver operating characteristic curve when the method of Embodiment 1 of the present invention is used for screening on a three-center dataset; wherein, Figure 3In this context, 'a' corresponds to performing internal five-fold cross-validation on the dataset from the first data center. Figure 3 In this context, 'b' corresponds to using the dataset from the second center as an independent external test set 1. Figure 3 In this context, 'c' corresponds to using the dataset from the third center as an independent external test set 2. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] In summary, the fully automated breast mass screening method based on multi-parameter ultrasound computed tomography (USCT) and multi-instance learning in this invention includes: acquiring the volume of the three-dimensional USCT reflection image and the volume of the sound velocity image of the whole breast after reconstruction of the subject; preprocessing the image, extracting continuous slices and combining them with automatically segmented fibroglandular tissue masks to construct multi-channel fused input data containing structural information, quantitative acoustic information, and anatomical constraint information; constructing a two-stage deep learning model. In the first stage, a detection network (which can be based on Swin Transformer) is used to scan the whole breast volume to generate slice-level lesion confidence scores. In the second stage, based on the multi-instance learning framework, key slice instances are selected according to the scores, and features are aggregated through a slice attention module to output breast-level screening conclusions.

[0024] The method may mainly include the following steps: (I) Multi-parameter image acquisition. Acquire ultrasound tomography (USCT) data of the subjects to be screened, including reflectance modal images reflecting tissue anatomy and transmission modal images reflecting tissue physical properties (i.e., sound velocity images).

[0025] (II) Multi-channel data construction. Spatial alignment and normalization processing are performed on images of different modalities; combined with automatically extracted regions of interest (such as breast tissue regions), multi-channel fused data that integrates structural information, functional information and anatomical constraint information is constructed.

[0026] (III) First Stage – Initial Screening of Candidate Lesions and Localization of Key Slices. A first deep learning network (detection network) is constructed. The multi-channel fused data is input into the network for full-volume scanning. The network is used to extract multi-scale features and generate lesion confidence scores at the slice level (for the current slice, first generate a lesion-level confidence score, i.e., a confidence score for each lesion; then, take the maximum value of the confidence scores of all lesions on each current slice as the confidence score of this current slice). Based on the scores, a set of key slices containing potential lesions is selected.

[0027] (IV) Second Stage – Global Discrimination Based on Multiple Instance Learning. A second deep learning network (classification network) is constructed, using the Multiple Instance Learning (MIL) framework; the selected set of key slices is used as the input bag, and the slices are used as instances. Instance features are aggregated through feature extraction and attention mechanisms to generate a global feature representation.

[0028] (V) Screening result output. Map the global feature representation to the screening result (e.g., presence of a mass / absence of a mass) to complete the automated screening.

[0029] Specifically, this invention provides a general architecture for fully automated breast mass screening based on multi-parameter ultrasound computed tomography (USCT) and multiple instance learning (MIL), the core logic of which includes the following steps: 1. Construction of multi-parameter data fusion: The system first acquires three-dimensional USCT data of the subject's entire breast, including anatomical structure modal images (i.e., reflection images) and tissue physical property modal images (i.e., sound velocity images), and performs the following processing: Preprocessing: Unify image data from different modalities to the same spatial resolution (i.e., spatial alignment) and perform pixel value normalization.

[0030] Anatomical constraints: For the normalized anatomical structure modal image, in order to reduce the interference of a large amount of fat and background water in the whole breast scan, the system introduces anatomical prior knowledge (such as an automatically segmented breast glandular tissue mask, which can use any existing model that can automatically segment breast images and obtain glandular tissue masks) to obtain a binary glandular tissue mask map, thereby concentrating computational resources on the effective tissue region.

[0031] Multi-channel fusion: The above structural information (i.e., normalized anatomical structure modal image), physical property information (i.e., normalized tissue physical property modal image), and anatomical constraint information (i.e., binary gland tissue mask image) are fused in the channel dimension or feature dimension to construct a rich multi-channel input data and obtain an input patch (especially by further combining contextual information, i.e., several neighboring slices before and after the current slice to construct multi-channel input data).

[0032] 2. Phase One: Full Volume Scan and Key Instance Screening Construct a lesion detection network that can extract image features and generate region proposals.

[0033] A sequence of whole-volume breast slices from a specific breast tissue is sequentially input into the lesion detection network for scanning. The network first extracts features from each input patch to obtain feature representations characterizing breast tissue structure and abnormal areas. Then, based on these feature representations, it detects potential lesion regions and outputs the location representation and confidence score of the corresponding candidate regions. Therefore, the network does not directly provide a final diagnostic conclusion, but rather quantitatively assesses whether there are suspected lesions in the current slice of each input patch, thereby obtaining a corresponding confidence score for suspected lesions.

[0034] All slices are sorted according to their scores, and the top-k key slices with the highest confidence levels are automatically selected (k is a preset positive integer, preferably k=5 in the following examples). This step simulates the process by which radiologists quickly browse to identify suspicious slices.

[0035] 3. Second Stage: Global Diagnosis Based on Multi-Instance Learning Construct a multi-instance learning classification network, using the key slice set selected above as input.

[0036] Feature aggregation: An attention mechanism is used to automatically learn the weight of each key slice in relation to the final diagnosis. Slices containing significant lesion features are assigned high weights, while slices that are misselected as noise are assigned low weights. The sum of the weights of all k key slices equals 1.

[0037] Global output: Generate a global feature vector at the breast level based on the weighted features, and output the final screening conclusion (i.e., whether the tested breast has a lump or not) through a classifier.

[0038] Example 1: Based on the aforementioned general architecture, this embodiment introduces an optimal parameter configuration scheme that has been experimentally verified. For example... Figure 1As shown, the core data processing pipeline is as follows: First, the reflection image, sound velocity image, and automatically generated anatomical mask of the 3D USCT are combined to construct 7-channel input data; then, the data enters the detection network, which analyzes the full-volume data to calculate the lesion confidence score and automatically selects the top-k most suspicious slices (denoted as key slices); finally, these key slices containing rich information are aggregated through a slice attention mechanism to generate the final breast cancer screening prediction result.

[0039] Specifically, this embodiment includes the following steps: Step 1: Specific data preprocessing and construction Resolution uniformity: This embodiment uses both reflection and sound velocity images. The original reflection image typically has a high resolution (the reflection image used in this embodiment has a resolution of 2048×2048), while the sound velocity image has a lower resolution (the sound velocity image used in this embodiment has a resolution of 512×512). In this embodiment, the reflection image is downsampled to 512×512 for alignment, and then the pixel values ​​of the reflection and sound velocity images are normalized to [0, 1].

[0040] Mask generation: Referring to existing techniques, a pre-trained U-Net segmentation network is used to extract a binary mask of breast tissue from the normalized reflectance image (for details on the network construction, see: Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical imagesegmentation. In: 2015 Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 234–241). This U-Net segmentation network can be constructed and trained using the nnU-Net training framework. During network training, a deep supervision mechanism is enabled to improve the segmentation ability of the output layers at different scales for breast tissue regions. The loss function is a combination of Dice loss and cross-entropy loss. The optimizer uses stochastic gradient descent (SGD) with an initial learning rate of 0.01, a weight decay coefficient of 0.00003, and a momentum of 0.99, updated using Nesterov momentum. The learning rate scheduling strategy employs a multinomial decay strategy. The training consists of 50 epochs, with 500 training iterations and 100 validation iterations per epoch. During training, foreground oversampling is used on the training samples to increase the probability of the target region appearing in the training samples, with a foreground oversampling ratio of 0.1. Furthermore, data augmentation is performed on the input images during training, including at least one of the following: spatial transformation, rotation, scaling, mirroring, adding Gaussian noise, Gaussian blur, brightness transformation, contrast transformation, low-resolution simulation, and Gamma transformation. After training, the network parameters are fixed to automatically generate a binary mask of breast tissue from the reflection image of the breast to be tested, suppressing interference from background and fat regions in subsequent screening.

[0041] 7-channel tensor construction: To utilize the 3D spatial context, the reflection images of the current slice i and its adjacent slices i-1 and i+1 (a total of 3 layers) are selected, along with the corresponding 3-layer sound velocity images (i, i-1, i+1), and a mask for the current layer i (slices with slice numbers at the beginning and end are not used as the current slice in this embodiment due to the lack of contextual information). These 7 images are stitched together to construct an input patch of size 512×512×7. Thus, for a 3D ultrasound computed tomography (USCT) image of a specific breast tissue, (N-2) 512×512×7 input patches can be obtained, where N is the total number of slices in the 3D USCT reflection image volume. Of course, the contextual information corresponding to neighboring slices (i.e., slices i-1 and i+1) is optional; using contextual information can enhance features.

[0042] Step 2: Specific implementation of the first-stage model This embodiment employs a lesion detection network based on a target detection framework to locate and detect suspected masses in whole breast slices. This lesion detection network includes at least a backbone feature extraction module and a detection output module (i.e., a detection head). This embodiment also includes a feature pyramid module for constructing multi-scale feature representations. The backbone feature extraction module can employ either a Transformer-based network or a convolutional neural network. For example, a Transformer-based network could be the Swin Transformer (for specific construction examples, see existing works such as Liu Z, Lin Y, Cao Y et al (2021) Swin transformer: hierarchical vision transformer using shifted windows. In: 2021 IEEE / CVF International Conference on Computer Vision (ICCV). IEEE, 9992-10002). A convolutional neural network could be ResNet (for specific construction examples, see existing works such as He K, Zhang X, Ren S, Sun J (2016) Deepresidual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 770–778).This embodiment uses Faster R-CNN (for specific construction examples, see existing technology Ren S, He K, Girshick R, Sun J (2017) Faster R-CNN: towards real-time object detection with region proposal networks. IEEE TransPatternAnal Mach Intell 39(6):1137-1149) as the detection framework (i.e., the detection output module), and uses Swin Transformer as the backbone network, combined with feature pyramid network (i.e., feature pyramid module; for specific construction examples, see existing technology Lin TY, Dollár P, Girshick R, He K, Hariharan B, Belongie S (2017) Feature pyramid networks for object detection. In: 2017IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 936-944) to extract multi-scale features, so as to output the bounding box coordinates of candidate lesions and the corresponding confidence scores. Since the input data in this embodiment is a 7-channel tensor, the number of input channels in the input layer of the backbone network is changed from 3 to 7. During training, input patches with lesion bounding boxes are used as training samples (that is, the training labels are the locations of the corresponding masses) to supervise the training of the lesion detection network, enabling it to learn the location representation of suspected masses and the corresponding confidence scores; the parameters of the lesion detection network in the first stage are randomly initialized.

[0043] In this embodiment, the lesion detection network in the first stage is trained using the AdamW optimizer with an initial learning rate of 0.0001, a weight decay coefficient of 0.05, and 27 training epochs. In the first 1000 iterations of training, a linear warm-up strategy is used to gradually increase the learning rate, which is then decayed back to 0.1 at the 21st and 24th epochs. During training, a multi-scale enhancement strategy is employed on the input image, i.e., while maintaining the aspect ratio, the image is randomly scaled to a preset size between 480 and 800 pixels. Simultaneously, at least one data augmentation method, such as random horizontal flipping or random cropping, can be combined to improve the robustness of the detection network to lesions of different scales and spatial distributions.

[0044] The detection network performs inference on each of the (N-2) 512×512×7 input patches of a whole breast slice, outputting the bounding box coordinates and corresponding lesion confidence scores for multiple candidate lesions within each input patch. The bounding box coordinates can be represented by the coordinates of the top-left and bottom-right corners of a rectangle, or by the coordinates of the center point, width, and height. For each input patch, the candidate lesion score with the highest confidence is taken as the slice-level suspicion score, and all input patches are sorted according to this score. The top-5 input patches with the highest scores are selected as key slices. Simultaneously, the bounding box coordinates of all candidate lesions within all input patches can also be used for subsequent 3D region of interest cropping.

[0045] Step 3: Specific implementation of the second-stage model Feature Extraction and Aggregation: The five selected key slices are input as a multi-instance learning package into the second-stage classification network. The second-stage classification network includes: a backbone feature extraction module for extracting key slice features, an attention weight calculation module for calculating the weights of each key slice, a feature aggregation module for weighted summation of key slice features, and a binary classification output module for outputting breast cancer screening results. The second-stage classification network employs an end-to-end training strategy and reuses the Swin Transformer backbone network parameters trained in the first-stage detection network as initial weights to retain the local discriminative features learned in the lesion detection stage. After the input image is uniformly adjusted to 512×512 pixels, the corresponding high-dimensional feature representation is extracted through the backbone network; then, the features of the five key slices are input into the slice attention module, and the attention weights of each key slice are calculated using learnable fully connected layers, thereby achieving feature aggregation.

[0046] In this embodiment, the second-stage classification network is trained using the AdamW optimizer with a weight decay coefficient of 0.00001. To balance the inheritance of the detection features from the first stage with adaptation to the global screening task, a differential learning rate strategy is adopted for the backbone network and the attention classification head, with the learning rate of the backbone network and the attention classification head both set to 0.00001. During training, the ReduceLROnPlateau learning rate scheduling strategy is used, with the validation set AUC as the monitoring metric. When the validation set AUC no longer increases within a preset number of epochs, the learning rate is decayed by 0.5; patience can be set to 10. The training consists of 30 epochs, with a batch size of 1, and the loss function is the binary cross-entropy loss function.

[0047] Classification output: The five feature vectors are weighted and summed (i.e., matrix multiplication), and the disease probability (i.e., presence of a lump / absence of a lump) between 0 and 1 is output through the Sigmoid activation function. During training, the breast-level screening result label (i.e., the screening result of presence / absence of a lump in the tested breast) is used as the supervision signal to perform binary classification learning on the entire multi-instance learning package.

[0048] Example 2: This embodiment discusses the expansion of assistive functions. To further enhance clinical applicability, based on this invention, the following functional modules are also included in addition to Embodiment 1: 1. Assisting in the diagnosis of benign and malignant diseases (3D classification) For screening results that are positive (i.e., a mass is present), the bounding box coordinates of all candidate lesions on all input patches are clustered using the bounding box coordinates of the candidate lesions output by the first-stage detection network to obtain the bounding box coordinates of the three-dimensional lesions. Then, based on these three-dimensional bounding box coordinates, a three-dimensional crop is performed around the location of the candidate lesion within the three-dimensional volume of the original high-resolution reflectance image to obtain a three-dimensional region of interest (ROI) containing the target lesion. Combining the current slice containing the candidate lesion and several adjacent slices, three-dimensional volume data containing lesion context information is extracted along the depth direction, and the three-dimensional ROI is resampled or scaled to a fixed size of 96×96×32. The 3D ROI is input into a 3D ResNet18 convolutional neural network for benign / malignant binary classification (for example, the specific construction of this network can be found in existing technology He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 770–778). The 3D ResNet18 network includes multiple sequentially connected 3D residual convolutional modules and a terminal classification module, used to extract spatial morphological features from the 3D lesion data and output benign / malignant classification results; the network can be implemented using existing publicly available 3D residual network architectures. The network utilizes 3D convolutional kernels to simultaneously extract the morphological and volumetric spatial features of the lesions. During training, 3D ResNet18 can be trained under supervision using 3D ROI data of lesions with benign or malignant labels. The labels are derived from pathological results or clinical diagnosis results. The network uses the AdamW optimizer with an initial learning rate of 0.0001, a weight decay coefficient of 0.05, 50 training epochs, and a batch size of 8.

[0049] 2. Results visualization and interpretability Key evidence display: The system interface automatically displays the original high-resolution reflection image corresponding to the current slice i in the key slice with the highest weight.

[0050] Heatmap localization: Based on the pre-trained second deep learning network (MIL), a heatmap is generated using class activation mapping techniques and overlaid on the original reflectance image to precisely indicate the suspected lesion areas of interest to the model (e.g., Figure 2 (As shown), to assist the doctor in reviewing. Figure 2 The model in this invention has been verified to be able to automatically focus on slices containing lesions with high weights and to accurately locate lesions in space using heatmaps, demonstrating good interpretability.

[0051] Example 3: This embodiment presents an implementation scheme different from the channel-dimensional fusion method in Embodiment 1. In Embodiment 1, the reflection image, sound velocity image, and mask image are directly stitched together at the input end to construct a single multi-channel input patch; while in this embodiment, the data of different modalities are first encoded through their respective feature extraction networks, and then cross-modal interaction and fusion are performed at the feature level to achieve feature-dimensional fusion.

[0052] Specifically, in this embodiment, for the current slice i, the reflection images of the current slice and its adjacent slices i-1 and i+1 are selected, and combined with the mask image of the current layer i to construct four-channel reflection input data; simultaneously, the sound velocity images of the current slice and its adjacent slices i-1 and i+1 are selected, and combined with the mask image of the current layer i to construct four-channel sound velocity input data. For slices with slice numbers located at the beginning and end, due to the lack of complete context slice information, they are not used as the current slice in this embodiment. Thus, for a certain three-dimensional USCT data of a breast being tested, (N-2) sets of four-channel reflection branch input data and (N-2) sets of four-channel sound velocity branch input data can be obtained, wherein the total number of slices in the three-dimensional USCT reflection image volume and the total number of slices in the three-dimensional USCT sound velocity image volume are both N.

[0053] In this embodiment, the four-channel reflection input data is input into the third backbone feature extraction network, and the four-channel sound velocity input data is input into the fourth backbone feature extraction network. The third and fourth backbone feature extraction networks can adopt the same structure or different structures, such as a Transformer-based network or a convolutional neural network-based network, respectively. The two backbone feature extraction networks output feature maps at multiple scales, thereby forming multi-scale feature representations of the reflection mode and the sound velocity mode.

[0054] To achieve effective interaction between the two modal features, this embodiment introduces cross-attention modules at multiple scale levels to fuse the reflection modal features and the sound velocity modal features (i.e., fusing the multi-scale feature representations of the reflection modality and the sound velocity modality using cross-attention modules). Specifically, at each scale level, using the reflection modal features as the query vector Q and the sound velocity modal features as the key vector K and value vector V, a fused cross-modal feature representation is obtained through cross-attention calculation (for all slices of the three-dimensional USCT reflection image volume, with a total of N slices, a cross-modal feature representation sequence including N cross-modal feature representations can be obtained). Through this method, anatomical structural information in the reflection image can actively guide the selection and enhancement of tissue physical property information in the sound velocity image, thereby improving the characterization ability of lesion-related regions.

[0055] After completing cross-attention interactions at each scale level, the fused multi-scale features can be further input into the detection head for candidate lesion detection, outputting the bounding box coordinates and corresponding confidence scores of the candidate lesions. Subsequently, following the method described in Example 1, all cross-modal feature representations are sorted, and the top-k cross-modal feature representations with the highest scores are selected as key cross-modal feature representations. The fused features corresponding to the key slices are then input into the second-stage multi-instance learning classification network (this second deep learning network also adopts the multi-instance learning MIL framework, including a fifth feature extraction network, a sixth feature extraction network, a second cross-modal calculation module, an attention weight calculation module for calculating the weights of each key cross-modal feature representation, a feature aggregation module for weighted summation of the key cross-modal feature representations, and a binary classification output module for outputting breast cancer screening results; this second deep learning network can simultaneously use k multi-channel reflection input data and k multi-channel sound velocity input data corresponding to k key cross-modal feature representations as... The input consists of a fifth feature extraction network used to generate multi-scale feature representations of the reflection modes for each of the k multi-channel reflection input data, thus obtaining multi-scale features of the k reflection modes; a sixth feature extraction network used to generate multi-scale feature representations of the sound speed modes for each of the k multi-channel sound speed input data, thus obtaining multi-scale features of the k sound speed modes; the multi-scale feature representations of the k reflection modes correspond one-to-one with the multi-scale feature representations of the k sound speed modes; for each corresponding multi-scale feature representation of a reflection mode and a sound speed mode, the second cross-modal computation module is used to perform calculations using the multi-scale feature representation of the reflection mode as... To query vector Q, the multi-scale feature representation of the sound speed modality is used as the key vector K and value vector V. A fused cross-modal feature representation is obtained through cross-attention computation, resulting in k cross-modal feature representations. The attention weight calculation module automatically learns the weight of each of the k cross-modal feature representations for the final diagnosis using the attention mechanism. The network then uses a feature aggregation module to perform a weighted summation of the cross-modal feature representations with the weights obtained from the attention weight calculation module, resulting in a set of global feature vectors. The binary classification output module classifies the global feature vectors and outputs a classification of whether the tested breast has a lump or not. The results show that the fifth feature extraction network in the second deep learning network has the same structure as the third feature extraction network in the first deep learning network; the sixth feature extraction network in the second deep learning network has the same structure as the fourth feature extraction network in the first deep learning network; in addition, the fifth feature extraction network reuses the network parameters of the third feature extraction network in the first deep learning network as initial weights; the sixth feature extraction network reuses the network parameters of the fourth feature extraction network in the first deep learning network as initial weights), and weighted aggregation is performed through a slice attention mechanism to obtain breast cancer screening results.

[0056] In other words, the main difference between this embodiment and Embodiment 1 lies in the first-stage input construction method and the cross-modal feature fusion method (the cross-modal feature sequence obtained in this embodiment will not be input into the complete first deep learning network in Embodiment 1. Since the cross-modal feature sequence has already been obtained through two feature extraction networks, it only needs to be input into the first deep learning network composed of feature pyramid and detection head; the first deep learning network in Embodiment 1 is composed of feature extraction, feature pyramid, and detection head output, and the detection head is the detection output module). However, the key slice screening (corresponding to key cross-modal feature representation in this embodiment) and breast-level classification output can be the same as in Embodiment 1. Furthermore, the three-dimensional benign and malignant auxiliary classification in Embodiment 2 can also be directly applied to this embodiment.

[0057] Furthermore, the cross-attention fusion in this embodiment can be performed at a single-scale feature layer output by the backbone network, or it can be performed at multiple scale feature layers respectively; for example, cross-attention interaction can be performed at multiple scale levels to simultaneously preserve shallow detail information and deep semantic information. Through this feature dimension fusion method, images of different modalities do not need to be directly stitched at the input end, but features are extracted separately first, and then targeted cross-modal alignment and fusion are performed, which helps to enhance the complementarity between the reflection modality and the sound velocity modality and improve the performance of lesion detection and screening.

[0058] Performance verification: The method described in Example 1 above (i.e., using Swing Transformer, k=5, 7-channel input, etc.) was used to perform performance validation on a multi-center dataset containing 506 subjects (941 breasts). The specific screening performance statistics of the method on different test sets are shown in Table 1 (in Table 1, the dataset used for "internal five-fold cross-validation" comes from the same hospital and is denoted as the first center; the dataset used for "independent external test set 1" comes from the same hospital and is denoted as the second center; the dataset used for "independent external test set 2" comes from the same hospital and is denoted as the third center; the first, second, and third centers are different from each other; and it should be noted that informed consent was obtained from the subjects when all data were acquired). The area under the receiver operating characteristic curve (AUC) is shown below. Figure 3 As shown.

[0059] Table 1. Screening performance statistics of the method of the present invention on a three-center dataset.

[0060] The above embodiments are merely examples. For instance, in addition to selecting the current slice i and its adjacent slices i-1 and i+1 (a total of 3 layers) to construct a 7-channel tensor, one can also use only the current slice i (a total of 1 layer) to construct a 3-channel tensor, or select context information of other spans (for example, select the current slice i and its adjacent slices i-2, i-1, i+1, i+2 (a total of 5 layers) to construct an 11-channel tensor).

[0061] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A fully automated screening method for breast masses based on multi-parameter ultrasound tomography and multi-example learning, characterized in that, Includes the following steps: S1: Prepare multi-parameter data, including the following sub-steps: S1-1: Collect three-dimensional ultrasound computed tomography (USCT) data of several breasts tested with known mass / no mass screening results and breasts to be screened; among which, For any breast tissue being tested: the 3D USCT data simultaneously includes the initial 3D USCT reflectance image volume and the initial 3D USCT sound velocity image volume of the entire breast; the initial 3D USCT reflectance image volume is composed of an initial USCT reflectance image slice sequence, and the initial 3D USCT sound velocity image volume is composed of an initial USCT sound velocity image slice sequence; the total number of slices in the initial 3D USCT reflectance image volume is the same as the total number of slices in the initial 3D USCT sound velocity image volume; Furthermore, for the three-dimensional USCT data of several breasts with known lumps, the location information of the corresponding lumps is also known; S1-2: For any breast tissue being tested: Spatially align all three-dimensional USCT data so that the length and width of any USCT reflection image slice are the same as the length and width of any USCT sound velocity image slice, and perform pixel value normalization processing to obtain the normalized USCT reflection image slice sequence and the normalized USCT sound velocity image slice sequence. S1-3: For any breast tissue being tested: extract the glandular tissue region of each slice as the region of interest from the normalized USCT reflectance image slice sequence, and obtain a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. S1-4: For any breast tissue being tested: The normalized USCT reflectance image slice sequence, the normalized USCT sound velocity image slice sequence, and the binary glandular tissue mask image sequence are fused in the channel dimension to obtain the input patch sequence. S2: Construct a lesion detection network as the first deep learning network. The first deep learning network includes a first backbone feature extraction module, a feature pyramid for constructing multi-scale feature representation, and a detection output module connected in sequence. This network can take the input patch sequence of any breast tissue being tested as input, perform a full-volume scan on the input patch sequence, extract multi-scale features using the network, and generate slice-level candidate lesion confidence scores. Based on the confidence scores, the k input patches with the highest confidence scores are selected as k key slices; where k is a preset positive number greater than 1. The first deep learning network is trained using the input patch sequence corresponding to each of the tested breasts with known screening results of having / not having lumps. The training label is the location of the corresponding lump, thus obtaining the trained first deep learning network. S3: Construct a Multiple Instant Learning (MIL) classification network as the second deep learning network. This second deep learning network employs the MIL framework and includes a second backbone feature extraction module for extracting key slice features, an attention weight calculation module for calculating the weights of each key slice, a feature aggregation module for weighted summation of key slice features, and a binary classification output module for outputting breast cancer screening results. This network can simultaneously use the k key slices as input. The attention weight calculation module automatically learns the weight of each key slice for the final diagnosis using an attention mechanism. The second backbone feature extraction module extracts corresponding features from the key slices. The network uses the feature aggregation module to weighted summation of the features obtained from the second backbone feature extraction module and the weights obtained from the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module classifies the global feature vectors and outputs the classification result of whether the tested breast has a lump or not. The second deep learning network is trained using k key slices corresponding to each breast in several breasts with known screening results of having / not having lumps. The training label is the screening results of having / not having lumps in the breasts, resulting in the trained second deep learning network. S4: Input the input patch sequence of the breast to be screened into the trained first deep learning network to obtain k key slices corresponding to the breast to be screened; then, input these k key slices into the trained second deep learning network to obtain the classification result of whether the breast to be screened has a mass or not, thus realizing the automatic screening of the breast to be screened.

2. The fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning as described in claim 1, characterized in that, In steps S1-4, fusion is performed at the channel level, specifically: The images, including the i-th slice in the normalized USCT reflectance image slice sequence, the i-th slice in the normalized USCT sound velocity image slice sequence, and the i-th glandular tissue mask in the binary glandular tissue mask sequence, are stitched together to form a multi-channel input patch. The input patch uses the i-th slice in the normalized USCT reflectance image slice sequence as the current slice. Where i is 1, ..., N, and N is the total number of slices in the volume of the three-dimensional USCT reflection image; Thus, an input patch sequence consisting of N input patches is obtained; Preferably, in steps S1-4, the context information of the current slice is also incorporated when performing fusion at the channel level, specifically: The images, including the (ij), ..., i, ..., (i+j)th slices from the normalized USCT reflectance image slice sequence, the (ij), ..., i, ..., (i+j)th slices from the normalized USCT sound velocity image slice sequence, and the i-th glandular tissue mask image from the binary glandular tissue mask image sequence, are stitched together to form a multi-channel input patch. This input patch sequence uses the i-th slice from the normalized USCT reflectance image slice sequence as the current slice. Where j is a preset positive integer, i is (1+j), ..., (Nj), and N is the total number of slices of the three-dimensional USCT reflection image volume; Thus, an input patch sequence consisting of (N-2j) input patches is obtained; More preferably, j is 1.

3. A fully automated screening method for breast masses based on multi-parameter ultrasound tomography and multi-example learning, characterized in that, Includes the following steps: (1): Prepare multi-parameter data, including the following sub-steps: (1-1): Three-dimensional ultrasound computed tomography (USCT) data were collected from several breasts with known mass / no mass screening results and breasts to be screened; among them, For any breast tissue being tested: the 3D USCT data simultaneously includes the initial 3D USCT reflectance image volume and the initial 3D USCT sound velocity image volume of the entire breast; the initial 3D USCT reflectance image volume is composed of an initial USCT reflectance image slice sequence, and the initial 3D USCT sound velocity image volume is composed of an initial USCT sound velocity image slice sequence; the total number of slices in the initial 3D USCT reflectance image volume is the same as the total number of slices in the initial 3D USCT sound velocity image volume; Furthermore, for the three-dimensional USCT data of several breasts with known lumps, the location information of the corresponding lumps is also known; (1-2): For any breast tissue being tested: All three-dimensional USCT data are spatially aligned so that the length and width of any USCT reflection image slice are the same as the length and width of any USCT sound velocity image slice, and pixel values ​​are normalized to obtain the normalized USCT reflection image slice sequence and the normalized USCT sound velocity image slice sequence. (1-3): For any breast tissue being tested: extract the glandular tissue region of each slice from the normalized USCT reflectance image slice sequence as the region of interest, and obtain a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. (1-4): For any breast tissue being tested: the normalized USCT reflectance image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel reflectance input data sequence; the normalized USCT sound velocity image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel sound velocity input data sequence. (2): Construct a lesion detection network as the first deep learning network, including a third feature extraction network, a fourth feature extraction network, a first cross-modal computation module, a feature pyramid module for constructing multi-scale feature representations, and a detection output module. The third feature extraction network is used to form a multi-scale feature representation of a reflection mode for each multi-channel reflection input data in the multi-channel reflection input data sequence, thereby obtaining a sequence of multi-scale feature representations of the reflection mode. The fourth feature extraction network is used to form a multi-scale feature representation of a sound speed mode for each multi-channel sound speed input data in the multi-channel sound speed input data sequence, thereby obtaining a sequence of multi-scale feature representations of the sound speed mode. The multi-scale feature representations of the reflection mode in the sequence of multi-scale feature representations of the reflection mode correspond one-to-one with the multi-scale feature representations of the sound speed mode in the sequence of multi-scale feature representations of the sound speed mode. For the corresponding multi-scale feature representation of a reflection mode and a multi-scale feature representation of a sound speed mode, the first cross-modal computation module is used to use the multi-scale feature representation of the reflection mode as the query vector Q and the multi-scale feature representation of the sound speed mode as the key vector K. The sum vector V is used to obtain a fused cross-modal feature representation through cross-attention computation, thereby obtaining a sequence of cross-modal feature representations. The output of the first cross-modal computation module is connected to the detection output module through a feature pyramid module for constructing multi-scale feature representations. The first deep learning network can perform a full-volume scan on the sequence of cross-modal feature representations of any tested breast tissue, extract multi-scale features using the network, and generate slice-level candidate lesion confidence scores. Based on the confidence scores, the k cross-modal feature representations with the highest confidence scores are selected as k key cross-modal feature representations, where k is a preset positive number greater than 1. The first deep learning network is trained using the sequence of multi-channel reflection input data and the sequence of multi-channel sound velocity input data corresponding to each of the several breasts tested with known screening results of having / not having lumps. The training label is the location of the corresponding lump, thereby obtaining the trained first deep learning network. (3): Construct a multi-instance learning (MIL) classification network as the second deep learning network. This network adopts the MIL framework and includes a fifth feature extraction network, a sixth feature extraction network, a second cross-modal computation module, an attention weight calculation module for calculating the weights of each key cross-modal feature representation, a feature aggregation module for weighted summation of the key cross-modal feature representations, and a binary classification output module for outputting breast cancer screening results. This second deep learning network can simultaneously take the k multi-channel reflection input data and k multi-channel sound velocity input data corresponding to the k key cross-modal feature representations as input. The fifth feature extraction network is used to calculate the weights of each key cross-modal feature representation. Multi-channel reflection input data are used to form multi-scale feature representations of reflection modes, thereby obtaining k multi-scale features of reflection modes. The sixth feature extraction network is used to form multi-scale feature representations of sound speed modes from these k multi-channel sound speed input data, thereby obtaining k multi-scale features of sound speed modes. The multi-scale feature representations of the k reflection modes correspond one-to-one with the multi-scale feature representations of the k sound speed modes. For a corresponding multi-scale feature representation of one reflection mode and one multi-scale feature representation of one sound speed mode, the second cross-modal calculation module is used to use the multi-scale feature representation of the reflection mode as the query vector Q and the multi-scale feature representation of the sound speed mode as the key vector K. The sum vector V is used to obtain a fused cross-modal feature representation through cross-attention calculation, thereby obtaining k cross-modal feature representations. The attention weight calculation module automatically learns the weight of each of the k cross-modal feature representations for the final diagnosis using the attention mechanism. The network uses the feature aggregation module to perform a weighted summation of the cross-modal feature representations with the weights obtained by the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module is used to classify the global feature vectors and output the classification result of whether the tested breast has a lump or not. The second deep learning network is trained using k key cross-modal features corresponding to each breast in several breasts with known screening results of having / not having lumps, along with k multi-channel reflection input data and k multi-channel sound velocity input data. The training label is the screening results of having / not having lumps in the breasts being tested, resulting in the trained second deep learning network. (4): Input the sequence of multi-channel reflection input data and the sequence of multi-channel sound velocity input data of the breast to be screened into the trained first deep learning network to obtain k key cross-modal feature representations corresponding to the breast to be screened; then, input the k multi-channel reflection input data and k multi-channel sound velocity input data corresponding to the k key cross-modal feature representations into the trained second deep learning network to obtain the classification result of whether the breast to be screened has a mass or not, thereby realizing the automatic screening of the breast to be screened.

4. The fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning as described in claim 3, characterized in that, Steps (1-4) are as follows: The image, including the i-th slice in the normalized USCT reflectance image slice sequence and the i-th glandular tissue mask in the binary glandular tissue mask sequence, is stitched together to form a multi-channel reflectance input data; where i is 1, ..., N, and N is the total number of slices in the three-dimensional USCT reflectance image volume; thus, a multi-channel reflectance input data sequence including N multi-channel reflectance input data is obtained. The image, including the i-th slice from the normalized USCT sound velocity image slice sequence, is stitched together with the i-th glandular tissue mask from the binary glandular tissue mask image sequence to form a multi-channel sound velocity input data; where i is 1, ..., N, and N is the total number of slices in the three-dimensional USCT sound velocity image volume; thus, a multi-channel sound velocity input data sequence including N multi-channel reflection input data is obtained; Preferably, in steps (1-4), when performing fusion at the channel level, the context information of the current slice is also taken into account, specifically: The images, including the (ij), ..., i, ..., (i+j)th slices from the normalized USCT reflectance image slice sequence and the i-th glandular tissue mask from the binary glandular tissue mask sequence, are stitched together to form a single multi-channel reflectance input data; where j is a preset positive integer, i is (1+j), ..., (Nj), and N is the total number of slices in the three-dimensional USCT reflectance image volume; thus, a multi-channel reflectance input data sequence containing (N-2j) multi-channel reflectance input data is obtained. The images, including the (ij), ..., i, ..., (i+j)th slices from the normalized USCT sound velocity image slice sequence and the i-th glandular tissue mask from the binary glandular tissue mask image sequence, are stitched together to form a multi-channel sound velocity input data; where j is a preset positive integer, i is (1+j), ..., (Nj), and N is the total number of slices in the three-dimensional USCT sound velocity image volume; thus, a multi-channel sound velocity input data sequence containing (N-2j) multi-channel sound velocity input data is obtained. More preferably, j is 1.

5. The fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning as described in any one of claims 1 to 4, characterized in that, In step S2 or step (2), k is preferably 5; The first deep learning network uses a random initialization method; Wherein, when the first deep learning network includes a backbone feature extraction module, the backbone feature extraction module adopts a network based on the Transformer architecture or a network based on a convolutional neural network, preferably a network based on SwinTransformer; when the first deep learning network includes a third feature extraction network and a fourth feature extraction network, the third feature extraction network and the fourth feature extraction network are independently selected from: a network based on the Transformer architecture and a network based on a convolutional neural network. The detection output module uses Faster R-CNN.

6. The fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning as described in claim 5, characterized in that, In step S3, the second backbone feature extraction module in the second deep learning network has the same structure as the first backbone feature extraction module in the first deep learning network in step S2; preferably, the second backbone feature extraction module reuses the network parameters of the first backbone feature extraction module in the first deep learning network trained in step S2 as the initial weights. In step (3), the fifth feature extraction network in the second deep learning network has the same structure as the third feature extraction network in the first deep learning network in step (2); in step (3), the sixth feature extraction network in the second deep learning network has the same structure as the fourth feature extraction network in the first deep learning network in step (2); preferably, the fifth feature extraction network reuses the network parameters of the third feature extraction network in the first deep learning network trained in step (2) as initial weights; the sixth feature extraction network reuses the network parameters of the fourth feature extraction network in the first deep learning network trained in step (2) as initial weights.

7. The fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning as described in any one of claims 1 to 4, characterized in that, In step S1-1 or step (1-1), for several breasts with known screening results of having / not having a mass, the volume of their three-dimensional USCT reflective image also includes a lesion boundary box annotation. In step S2 or step (2), the first deep learning network can generate the bounding box coordinates of the corresponding candidate lesion while generating the confidence score of the candidate lesion at the slice level. Furthermore, if step S4 or step (4) yields a classification result indicating that the breast to be screened has a mass, then the fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning further includes step S5 or step (5): Cluster the bounding box coordinates of all candidate lesions obtained in step S2 to obtain the bounding box coordinates of the three-dimensional lesions. Then, based on the bounding box coordinates of the three-dimensional lesions, perform three-dimensional cropping on the volume of the initial three-dimensional USCT reflection image to obtain the three-dimensional region of interest (ROI) containing the target lesion. Then, the three-dimensional region of interest (ROI) is resampled or scaled to a preset length, width, and height, and then input into a 3D ResNet18 convolutional neural network for binary classification of benign and malignant masses, thereby obtaining the classification result of benign / malignant masses.

8. The fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning as described in any one of claims 1 to 4, characterized in that, The fully automated breast mass screening method based on multi-parameter ultrasound tomography and multi-example learning also includes a step of displaying overlaid heat maps: Based on the trained second deep learning network, a heatmap is generated using class activation mapping technology and superimposed on the initial 3D USCT reflection image volume to prompt the user for review.

9. A fully automated breast mass screening system based on multi-parameter ultrasound tomography and multi-example learning, characterized in that, This system can classify the presence or absence of lumps in the breast tissue to be screened, enabling automatic screening of the breast tissue. The system includes the following functional modules: The multi-parameter data preprocessing module is used to spatially align all three-dimensional ultrasound computed tomography (USCT) data of the breast to be screened, ensuring that the length and width dimensions of any USCT reflectance image slice are the same as those of any USCT sound velocity image slice. Pixel value normalization is then performed to obtain normalized USCT reflectance image slice sequences and normalized USCT sound velocity image slice sequences. The three-dimensional USCT data simultaneously includes the initial three-dimensional USCT reflectance image volume and the initial three-dimensional USCT sound velocity image volume of the entire breast. The initial three-dimensional USCT reflectance image volume is composed of the initial USCT reflectance image slice sequence, and the initial three-dimensional USCT sound velocity image volume is composed of the initial USCT sound velocity image slice sequence. The total number of slices in the initial three-dimensional USCT reflectance image volume is the same as the total number of slices in the initial three-dimensional USCT sound velocity image volume. The glandular tissue region of each slice is extracted as the region of interest from the normalized USCT reflectance image slice sequence, resulting in a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. The normalized USCT reflection image slice sequence, the normalized USCT sound velocity image slice sequence, and the binary gland tissue mask image sequence are fused in the channel dimension to obtain the input patch sequence. The first deep learning network functional module includes a first backbone feature extraction module, a feature pyramid for constructing multi-scale feature representations, and a detection output module connected in sequence. This network can take the input patch sequence of the tested breast as input, perform a full-volume scan on the input patch sequence, extract multi-scale features using the network, generate slice-level candidate lesion confidence scores, and select the k input patches with the highest confidence scores as k key slices based on the confidence scores; where k is a preset positive number greater than 1. The first deep learning network functional module was trained, and the training label was the location of the corresponding mass in each of the tested breasts with known mass / no mass screening results. The second deep learning network functional module: The second deep learning network adopts the Multiple Instant Learning (MIL) framework, including a second backbone feature extraction module for extracting key slice features, an attention weight calculation module for calculating the weights of each key slice, a feature aggregation module for weighted summation of key slice features, and a binary classification output module for outputting breast cancer screening results. This network can simultaneously use the k key slices as input, and through the attention weight calculation module, automatically learn the weight of each key slice for the final diagnosis using an attention mechanism. The second backbone feature extraction module extracts corresponding features from the key slices. The network uses the feature aggregation module to weighted summation of the features obtained from the second backbone feature extraction module and the weights obtained from the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module classifies the global feature vectors and outputs the classification result of whether the tested breast has a lump or not. The second deep learning network functional module was trained, and the training labels were the screening results of lumps / no lumps in several tested breasts with known lumps / no lumps.

10. A fully automated breast mass screening system based on multi-parameter ultrasound tomography and multi-example learning, characterized in that, This system can classify the presence or absence of lumps in the breast tissue to be screened, enabling automatic screening of the breast tissue. The system includes the following functional modules: The multi-parameter data preprocessing module is used to spatially align all three-dimensional ultrasound computed tomography (USCT) data of the breast to be screened, ensuring that the length and width dimensions of any USCT reflectance image slice are the same as those of any USCT sound velocity image slice. Pixel value normalization is then performed to obtain normalized USCT reflectance image slice sequences and normalized USCT sound velocity image slice sequences. The three-dimensional USCT data simultaneously includes the initial three-dimensional USCT reflectance image volume and the initial three-dimensional USCT sound velocity image volume of the entire breast. The initial three-dimensional USCT reflectance image volume is composed of the initial USCT reflectance image slice sequence, and the initial three-dimensional USCT sound velocity image volume is composed of the initial USCT sound velocity image slice sequence. The total number of slices in the initial three-dimensional USCT reflectance image volume is the same as the total number of slices in the initial three-dimensional USCT sound velocity image volume. The glandular tissue region of each slice is extracted as the region of interest from the normalized USCT reflectance image slice sequence, resulting in a binary glandular tissue mask sequence that corresponds one-to-one with each normalized USCT reflectance image slice. The normalized USCT reflectance image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel reflectance input data sequence; the normalized USCT sound velocity image slice sequence and the binary glandular tissue mask image sequence are fused in the channel dimension to form a multi-channel sound velocity input data sequence. The first deep learning network functional module includes a third feature extraction network, a fourth feature extraction network, a first cross-modal computation module, a feature pyramid module for constructing multi-scale feature representations, and a detection output module. The third feature extraction network is used to form a multi-scale feature representation of one reflection mode for each multi-channel reflection input data in the multi-channel reflection input data sequence, thereby obtaining a sequence of multi-scale feature representations of the reflection modes. The fourth feature extraction network is used to form a multi-scale feature representation of one sound speed mode for each multi-channel sound speed input data in the multi-channel sound speed input data sequence, thereby obtaining a sequence of multi-scale feature representations of the sound speed modes. The multi-scale feature representations of the reflection modes in the sequence of multi-scale feature representations of the reflection modes correspond one-to-one with the multi-scale feature representations of the sound speed modes in the sequence of multi-scale feature representations of the sound speed modes. For each corresponding multi-scale feature representation of one reflection mode and one multi-scale feature representation of one sound speed mode, the first cross-modal computation module uses the multi-scale feature representation of the reflection mode as the query vector Q and the multi-scale feature representation of the sound speed mode as the key vector K. The sum vector V is used to obtain a fused cross-modal feature representation through cross-attention computation, thereby obtaining a sequence of cross-modal feature representations. The output of the first cross-modal computation module is connected to the detection output module through a feature pyramid module for constructing multi-scale feature representations. The first deep learning network can perform a full-volume scan on the sequence of cross-modal feature representations of the tested breast, extract multi-scale features using the network, and generate slice-level candidate lesion confidence scores. Based on the confidence scores, the k cross-modal feature representations with the highest confidence scores are selected as k key cross-modal feature representations, where k is a preset positive number greater than 1. The first deep learning network functional module was trained, and the training label was the location of the corresponding mass in each of the tested breasts with known mass / no mass screening results. The second deep learning network functional module: The second deep learning network adopts the Multiple Instant Learning (MIL) framework, including a fifth feature extraction network, a sixth feature extraction network, a second cross-modal computation module, an attention weight calculation module for calculating the weights of each key cross-modal feature representation, a feature aggregation module for weighted summation of the key cross-modal feature representations, and a binary classification output module for outputting breast cancer screening results; this second deep learning network can simultaneously take k multi-channel reflection input data and k multi-channel sound velocity input data corresponding to the k key cross-modal feature representations as input, wherein the fifth feature extraction network is used to process these k multi-channel... The reflected input data are used to form multi-scale feature representations of the reflection modes, thereby obtaining k multi-scale features of the reflection modes. The sixth feature extraction network is used to form multi-scale feature representations of the sound speed modes from these k multi-channel sound speed input data, thereby obtaining k multi-scale features of the sound speed modes. The multi-scale feature representations of the k reflection modes correspond one-to-one with the multi-scale feature representations of the k sound speed modes. For a corresponding multi-scale feature representation of one reflection mode and one multi-scale feature representation of one sound speed mode, the second cross-modal calculation module is used to use the multi-scale feature representation of the reflection mode as the query vector Q and the multi-scale feature representation of the sound speed mode as the key vector K. The sum vector V is used to obtain a fused cross-modal feature representation through cross-attention calculation, thereby obtaining k cross-modal feature representations. The attention weight calculation module automatically learns the weight of each of the k cross-modal feature representations for the final diagnosis using the attention mechanism. The network uses the feature aggregation module to perform a weighted summation of the cross-modal feature representations with the weights obtained by the attention weight calculation module, thereby obtaining a set of global feature vectors. The binary classification output module is used to classify the global feature vectors and output the classification result of whether the tested breast has a lump or not. The second deep learning network functional module was trained, and the training labels were the screening results of lumps / no lumps in several tested breasts with known lumps / no lumps.

Images