Breast cancer axillary lymph node metastasis risk prediction method and system, device and medium

Abstract

The application discloses a breast cancer axillary lymph node metastasis risk prediction method and system, equipment and medium, and relates to the technical field of deep learning and medical image processing. The method comprises the following steps: acquiring structured clinical pathological data and multi-modal ultrasound image data of a patient; encoding the missing state of the structured clinical pathological data to generate a structured representation; inputting the structured representation into a super network to generate a dynamic parameter set; generating a basic image representation of the multi-modal ultrasound image data through a dynamic self-adaptive aggregation mechanism; extracting shallow features from real images for the existing modalities, and using learnable baseline features to occupy the missing modalities; modulating the deep features of the basic image representation according to the dynamic parameter set to output a metastasis risk prediction probability. The application can stably operate under the condition that the ultrasound modalities are incomplete, realize deep modulation of the structured clinical prior to image feature extraction, and improve the prediction accuracy and clinical applicability.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and medical image processing technology, and in particular to methods, systems, devices, and media for predicting the risk of breast cancer axillary lymph node metastasis. Background Technology

[0002] Breast cancer is one of the most common malignant tumors in women, and the status of axillary lymph node metastasis is a key factor affecting the clinical stage, surgical plan, and prognostic assessment of breast cancer patients. Accurate preoperative assessment of axillary lymph node metastasis risk helps avoid unnecessary axillary lymph node dissection, reduces the incidence of postoperative complications, and improves the accuracy of individualized treatment decisions.

[0003] Currently, the main sources of information for preoperative prediction of axillary lymph node metastasis in breast cancer fall into two categories: first, structured clinicopathological data, such as patient age, tumor size, pathological grade, hormone receptor expression status, proliferation-related indicators (Ki-67), and other routine test results; and second, ultrasound imaging data. In ultrasound imaging, B-mode ultrasound can provide structural information such as tumor morphology, boundaries, internal echoes, and relationships with surrounding tissues; strain elastography can reflect the relative stiffness of the lesion tissue; and color Doppler flow imaging can reflect blood perfusion within and around the lesion. These three ultrasound modalities are significantly complementary, and theoretically, combined analysis can improve the predictive ability for axillary lymph node metastasis.

[0004] However, in real clinical settings, multimodal ultrasound data is often difficult to acquire stably and completely. B-mode ultrasound is usually used as a basic examination and its acquisition is relatively standardized; while functional ultrasound modalities such as strain elastography and color Doppler flow imaging are affected by various factors such as equipment conditions, operator experience, patient cooperation, and examination time, resulting in significant uncertainty regarding missing data. This data acquisition method determines that clinical data naturally exhibits the characteristic of having both partially missing structured clinical pathology data and irregular missing data from ultrasound imaging modalities.

[0005] To address the aforementioned issues, existing technologies include several research protocols that combine structured clinicopathological data with medical imaging data for predicting axillary lymph node metastasis in breast cancer. These protocols can be summarized into the following main categories: Post-fusion schemes for structured data and image data. These methods typically encode the structured data and image data independently first, then concatenate the feature vectors of the two classes directly before the classification layer, and finally output the transfer risk result through a classifier. This type of method is relatively straightforward, but the structured data only participates in the decision-making stage and cannot deeply influence the image feature extraction process.

[0006] A multimodal image fusion scheme with fixed input channels. This type of method typically uses three modalities as a fixed number of input channels for joint modeling. It relies on the premise that all modalities are available and spatially consistent registration is possible. If any functional modality is missing, the input structure cannot meet the model requirements.

[0007] A hypernetwork scheme driven by structured data to generate master network parameters. This type of method uses structured data to generate weight parameters for certain layers of the imaging backbone network, enabling the master network to form individualized feature extraction paths for different patients. This scheme has significant advantages in deep fusion of heterogeneous data, but existing implementations are mostly designed for single image modalities and have not yet formed a complete and feasible technical path for the dynamic changes of multimodal input combinations in breast ultrasound.

[0008] In summary, existing technologies have the following shortcomings when dealing with the coexistence of partially missing structured data and irregular missing ultrasound image modalities in real clinical settings: (1) The multimodal image input method lacks adaptability to real clinical scenarios. It assumes that all image modalities are complete during the training and inference phases, and cannot handle common situations where functional modalities are missing. (2) The processing methods for missing image modalities lack clinical robustness, and generative completion methods may introduce uncontrollable biases caused by forged images; (3) The fusion level between structured clinical pathology information and imaging information is too shallow, and the clinical prior to the extraction of imaging features has not been deeply modulated. (4) The missing data handling mechanism lacks a hierarchical design and fails to distinguish between missing structured data and missing image modalities.

[0009] Therefore, how to construct a stable and reliable prediction model for axillary lymph node metastasis in breast cancer under the condition that both structured clinical pathology data and multimodal ultrasound images have irregular gaps has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0010] To achieve the above-mentioned objectives and other advantages of the present invention, a first objective of the present invention is to provide a method for predicting the risk of axillary lymph node metastasis in breast cancer, comprising the following steps: Acquire structured clinicopathological data and multimodal ultrasound imaging data of patients; the multimodal ultrasound imaging data includes at least one of B-mode ultrasound images, strain elastography images, and color Doppler flow imaging images; Missing states are encoded in the structured clinicopathological data to generate a structured representation; The structured representation is input into the hypernetwork to generate a dynamic parameter set; The multimodal ultrasound image data is used to generate basic image representations through a dynamic adaptive aggregation mechanism; wherein, for existing modalities, shallow features are extracted using real images; for missing modalities, learnable baseline features are used for placeholders. Based on the dynamic parameter set, deep feature modulation is performed on the basic image representation to output the predicted probability of transfer risk.

[0011] Furthermore, the missing state encoding of the structured clinicopathological data specifically includes: The structured features, after numerical imputation, are concatenated with the corresponding missing markers to obtain the structured representation.

[0012] Further, the step of inputting the structured representation into the hypernetwork to generate a dynamic parameter set includes: The structured representation is encoded by a structured encoder in a hypernetwork to obtain a patient-level clinical prior representation; The dynamic parameter set is generated by a parameter generator in the hypernetwork based on the clinical prior characterization.

[0013] Furthermore, the step of generating a basic image representation from the multimodal ultrasound image data through a dynamic adaptive aggregation mechanism includes: Shallow features were extracted from the modalities present in B-mode ultrasound images, strain elastography images, and color Doppler blood flow imaging images, respectively, and the corresponding learnable baseline features were used to fill in the missing modalities. The shallow features of each modality are concatenated to obtain the input of the fusion module; The fusion module input and the modal existence state vector input are input to the fusion weight calculation unit to calculate the fusion response score of each modality; The fusion response scores of each modality are converted into fusion weights based on the normalized exponential mapping; The shallow features of each modality are weighted and summed according to the fusion weights to obtain the basic image representation.

[0014] Furthermore, the modal existence state vector is used to indicate the presence of each of the B-mode ultrasound image, strain elastography image, and color Doppler blood flow imaging image.

[0015] Furthermore, the step of performing deep feature modulation on the basic image representation based on the dynamic parameter set includes: The basic image representation is input into a modulated main network, which is composed of L modulated feature extraction modules connected in series, where L≥1; The dynamic parameter set is loaded into a preset modulatory layer in the modulatory main network; Feature extraction is performed sequentially through L modulated feature extraction modules to obtain deep image representation; After global feature aggregation of the deep image representation, the predicted probability of migration risk is output through a classification layer.

[0016] Furthermore, it also includes training steps: During the training phase, at least one of the strain elastic imaging modal and color Doppler blood flow imaging modal is randomly masked with a preset probability. The state of the randomly masked modal is used to participate in shallow feature extraction and fusion weight calculation to simulate the functional modality loss scenario in real clinical practice.

[0017] Furthermore, when outputting the predicted probability of transfer risk, a weighted binary classification loss function is used for optimization, with different loss weights assigned to positive and negative samples respectively.

[0018] A second objective of this invention is to provide a breast cancer axillary lymph node metastasis risk prediction system, comprising: The data acquisition module is used to acquire the patient's structured clinicopathological data and multimodal ultrasound imaging data; the multimodal ultrasound imaging data includes at least one of B-mode ultrasound images, strain elastography images, and color Doppler flow imaging images. The structured data encoding module is used to encode the missing states of the structured clinicopathological data and generate a structured representation. The hypernetwork module is used to generate a dynamic parameter set based on the structured representation; The multimodal image fusion module is used to generate basic image representations from the multimodal ultrasound image data through a dynamic adaptive aggregation mechanism; wherein, for existing modalities, shallow features are extracted using real images; for missing modalities, learnable baseline features are used for placeholders; The feature modulation and prediction module is used to perform deep feature modulation on the basic image representation based on the dynamic parameter set and output the predicted probability of transfer risk.

[0019] Furthermore, when the structured data encoding module encodes the missing state of the structured clinical pathology data, it concatenates the numerically imputed structured features with the corresponding missing markers to obtain the structured representation.

[0020] Furthermore, the hypernetwork module includes: A structured encoder is used to encode the structured representation to obtain a patient-level clinical prior representation; A parameter generator is used to generate the dynamic parameter set based on the clinical prior characterization.

[0021] Furthermore, the multimodal image fusion module includes: The shallow feature extraction branch is set up separately for B-mode ultrasound images, strain elastic imaging images and color Doppler blood flow imaging images, and is used to extract shallow features from the existing modes. Baseline feature library, which stores learnable baseline features corresponding to each missing modality; The fusion weight calculation unit is used to calculate the fusion response score of each modality based on the shallow features of each modality and the modality existence state vector, and to calculate the fusion weight of each modality based on the normalized exponential mapping. The weighted aggregation unit is used to perform weighted summation of the shallow features of each modality according to the fusion weights to obtain the basic image representation.

[0022] Furthermore, the feature modulation and prediction module includes: The modulated main network consists of L modulated feature extraction modules connected in series, where L≥1; The parameter loading unit is used to load the dynamic parameter set generated by the supernetwork module into the preset modulatorable layer in the modulatorable main network; The classification unit is used to perform global feature aggregation on the deep image representation obtained after deep feature modulation, and then output the predicted probability of the transfer risk through the classification layer.

[0023] Furthermore, the system also includes a training module, which employs an image modality random discarding strategy for training. The image modality random discarding strategy is as follows: during the training phase, at least one of the strain elastic imaging modality and the color Doppler blood flow imaging modality is randomly masked with a preset probability to simulate a functional modality loss scenario in real clinical settings.

[0024] A third objective of the present invention is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0025] A fourth objective of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0026] Compared with the prior art, the beneficial effects of the present invention are: Existing technologies often rely on fixed and complete multimodal inputs, while this invention allows the number of image modalities to change dynamically, thus enabling direct processing of incomplete ultrasound data commonly found in real clinical settings. This design significantly improves the deployability and practical application value of the model.

[0027] This invention does not employ a generative completion method for missing images. Instead, it achieves structural compatibility and information occupancy in the feature space. This strategy avoids inputting falsified image content that is not supported by real observations into the model, which helps improve the clinical credibility and security of the system.

[0028] This invention does not simply splice clinical and imaging features at the end of the model, but rather uses a hypernetwork to allow clinical and pathological information to enter the feature extraction stage of the main network. Therefore, structured priors can directly influence the formation of mid-to-deep representations in the main imaging network, improving the depth and effectiveness of multi-source heterogeneous information fusion.

[0029] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 Procedure for predicting the risk of axillary lymph node metastasis in breast cancer Figure 1 ; Figure 2 Procedure for predicting the risk of axillary lymph node metastasis in breast cancer Figure 2 ; Figure 3 A flowchart for generating a dynamic parameter set by inputting structured representations into the hypernetwork; Figure 4 A flowchart for generating basic image representations from multimodal ultrasound image data through a dynamic adaptive aggregation mechanism; Figure 5 Flowchart for deep feature modulation of basic image representation based on dynamic parameter set; Figure 6 A flowchart of a breast cancer axillary lymph node metastasis risk prediction system; Figure 7 This is a schematic diagram of a computer device. Figure 8 This is a schematic diagram of a computer-readable storage medium. Detailed Implementation

[0031] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0032] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of this invention.

[0033] The drawing numbers in this application are only used to distinguish the steps in the scheme and are not used to limit the execution order of the steps. The specific execution order is as described in the specification.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0035] This invention provides a deep learning-based system and method for predicting the risk of axillary lymph node metastasis in breast cancer. The technical solution of this invention simultaneously addresses the following problems existing in the prior art: First, it enables structured clinicopathological data not only to participate in the final classification but also to deeply participate in the parameter formation and feature extraction process of the main imaging network; second, it enables the main imaging network to operate stably even with a variable number of input modalities, adapting to various real-world clinical scenarios such as only B-mode ultrasound, simultaneous B-mode ultrasound and color Doppler flow imaging, or all three modalities; third, it performs structural-level compatibility processing for missing imaging modalities without using image generation of missing modalities; fourth, it designs targeted coding and compensation mechanisms for missing structured data and missing imaging modalities respectively, improving the robustness and interpretability of the overall model. The specific solution is as follows: Example 1

[0036] A method for predicting the risk of breast cancer axillary lymph node metastasis, such as Figure 1 , Figure 2 As shown, it includes the following steps: S100. Acquire the patient's structured clinicopathological data and multimodal ultrasound imaging data; the multimodal ultrasound imaging data includes at least one of B-mode ultrasound images, strain elastography images, and color Doppler flow imaging images. This embodiment organizes and preprocesses the patient's structured clinicopathological data. The structured clinicopathological data includes, but is not limited to, patient age, tumor size, pathological grade, hormone receptor expression status (including estrogen receptor ER and progesterone receptor PR), proliferation-related indicators (Ki-67), and other routine test results. In actual clinical settings, some fields in the above structured data may be missing.

[0037] In other embodiments, the structured data also includes: radiomics quantitative features (morphology, texture, wavelet features, etc. extracted from ultrasound images), laboratory test results (such as complete blood count, tumor marker CA15-3, etc.), molecular subtyping information (Luminal A, Luminal B, HER2 overexpression, triple negative, etc.), gene expression information (such as Oncotype DX recurrence score), and other patient characteristics that can be organized in tabular form.

[0038] The aforementioned extended structured information can also characterize the patient's overall clinical status and participate in the generation of main network parameters, all of which are extended applications of the technical concept of this invention.

[0039] S200. Encode the missing state of the structured clinicopathological data to generate a structured representation; In some embodiments, encoding the missing state of the structured clinicopathological data specifically includes: The structured features, after numerical imputation, are concatenated with the corresponding missing markers to obtain the structured representation.

[0040] Let the structured feature vector after numerical imputation be... The corresponding missing label vector is m. Here, the dimension of the missing label vector m is the same as that of the structured feature vector. Similarly, each dimension takes a value of 0 or 1. When a field is missing, the corresponding position is marked as 1; otherwise, it is marked as 0. The structured feature vector and the missing value vector are concatenated to obtain the final structured representation of the input model. Here, Concat represents the feature concatenation operation.

[0041] Secondly, the ultrasound image data is preprocessed. Let the possible set of ultrasound modalities a patient might possess be: in, This represents a B-mode ultrasound image. Represents strain elastic imaging images, This represents a color Doppler blood flow imaging image. Further, the modal existence state vector is defined as: in, This indicates that the corresponding mode exists. This indicates that the corresponding modality is missing. For all actual modal images, lesion regions are cropped and scaled to a uniform size of 224×224 pixels to ensure consistency in subsequent inputs.

[0042] S300. Input the structured representation into the hypernetwork to generate a dynamic parameter set; In some embodiments, such as Figure 3 As shown, the step of inputting the structured representation into the hypernetwork to generate a dynamic parameter set includes: S310. The structured representation is encoded by a structured encoder in a hypernetwork to obtain a patient-level clinical prior representation; S320. Generate the dynamic parameter set based on the clinical prior characterization using a parameter generator in the hypernetwork.

[0043] In this embodiment, the hypernetwork is a neural network structure capable of dynamically generating parameters for another network based on input data. In this embodiment, the hypernetwork comprises two core components: a structured encoder and a structured encoder. and parameter generator .

[0044] Structured encoder It consists of a three-layer fully connected neural network, with each layer followed by a batch normalization layer and a ReLU activation function. The input dimension is the sum of the structured feature dimension and the missing label dimension, and the output dimension is 256-dimensional. Through a structured encoder, patient-level clinical prior representations are obtained: Parameter generator It consists of a multi-layer fully connected network, whose output dimension matches the total dimension of the dynamic parameters of the main network to be generated. The parameter generator is based on clinical prior characterization. Generate a set of dynamic parameters for modulating specific layers of the main network: in, This represents the deep prior representation derived from structured clinicopathological information. This represents a set of dynamic parameters generated by the supernetwork to modulate specific layers of the main network, including but not limited to convolutional layer weights, convolutional layer biases, and scaling and offset factors of batch normalized layers.

[0045] Through the above mapping, structured clinicopathological information no longer only participates in decision-making at the classification stage, but also directly influences the parameter formation process of the main network through the hypernetwork. This gives subsequent image feature extraction a clear, individualized prior constraint. Different patients will obtain different main network parameters based on their respective structured clinicopathological information, achieving personalized feature extraction.

[0046] S400. Generate basic image representations from the multimodal ultrasound image data through a dynamic adaptive aggregation mechanism; wherein, for existing modalities, shallow features are extracted using real images; for missing modalities, learnable baseline features are used for occupancy. In some embodiments, such as Figure 4 As shown, the step of generating a basic image representation from the multimodal ultrasound image data through a dynamic adaptive aggregation mechanism includes: S410. Extract shallow features from the modes present in B-mode ultrasound images, strain elastography images, and color Doppler blood flow imaging images respectively, and call the corresponding learnable baseline features to fill the missing modes. At the front end of the main network, shallow feature extraction branches corresponding to the three ultrasound modalities are set up respectively. Each shallow feature extraction branch adopts a lightweight convolutional neural network structure, specifically including 3 convolutional layers (3×3 kernel size, stride of 1, padding of 1), each convolutional layer is followed by a batch normalization layer and a ReLU activation function, and finally downsampling is performed through a 2×2 max pooling layer (stride of 2). The output feature map size of each branch is 112×112, and the number of channels is 64.

[0047] Let the shallow feature extraction function for the k-th mode be... The corresponding learnable baseline features are Then its output feature is defined as: When mode k actually exists ( ), using real images Extracting shallow features; when mode k is missing ( Instead of performing image-level completion, it uses the corresponding learnable baseline features. Placeholders are used to maintain consistency in the subsequent network input structure. Baseline features can be learned. It is a trainable parameter tensor with the same dimension as the shallow features of the corresponding modality. The parameters are initialized using the Xavier initialization method and updated through backpropagation during model training. This design avoids the forgery bias that may be introduced by image-level generation completion, while achieving stable compatibility with missing modalities at the network structure level.

[0048] S420. The shallow features of each modality are concatenated to obtain the input of the fusion module; After obtaining the shallow features of the three modalities, they are concatenated to obtain the input of the fusion module: S430. Input the fusion module and the modal existence state vector into the fusion weight calculation unit to calculate the fusion response score of each modality; wherein, the modal existence state vector is used to indicate whether the B-mode ultrasound image, strain elastic imaging image and color Doppler blood flow imaging image exist respectively.

[0049] Subsequently, the fusion module combines the feature concatenation results with the modal state vectors to calculate the fused response score for each modality. The fusion module consists of two fully connected network layers: the first layer maps the input features to 128 dimensions, and the second layer outputs a 3-dimensional vector, corresponding to the response scores for each of the three modalities. in, , and These represent the fusion response scores of the three modalities in the current sample. Let s be the encoded representation of the modal existence state vector s, and let s be the fusion response score calculation network.

[0050] S440. Based on the normalized exponential mapping, the fusion response scores of each modality are converted into fusion weights; Based on the response scores, the fusion weights for each modality are calculated using a normalized exponential mapping (Softmax function). For any modality k, the fusion weight is defined as: in, Let represent the fusion weights of mode k, and satisfy . .

[0051] S450. The shallow features of each modality are weighted and summed according to the fusion weights to obtain the basic image representation.

[0052] Ultimately, the basic image representation is defined as a weighted sum of shallow features across all modalities: Through the aforementioned dynamic adaptive aggregation mechanism, the main network front-end can adaptively form a unified basic image representation based on the modal combination and characteristic response intensity of the current sample. Regardless of whether the patient has one, two, or three ultrasound modalities, this mechanism can stably output a basic image representation with consistent dimensions for subsequent deep network processing.

[0053] In other embodiments, the main network does not employ a convolutional residual structure, but instead uses a visual representation network (ViT) based on an attention mechanism. Specifically, it represents the basic image representation... The image is divided into 16×16 blocks. Each block is linearly mapped to obtain an image block embedding vector, and positional encoding is added. The sequence of image block embedding vectors is then fed into a Transformer encoder, which consists of multiple multi-head self-attention layers and feedforward network layers stacked alternately.

[0054] For shallow feature extraction of different ultrasound modalities, this embodiment constructs independent sequence representation branches and then fuses them through a cross-modal attention mechanism. Missing modalities are still filled using learnable baseline features, maintaining the overall approach. The dynamic parameters generated by the hypernetwork are used to modulate the query, key, and value projection matrices in the Transformer encoder and the weight parameters of the feedforward network.

[0055] In other embodiments, modality fusion is placed in the middle to later stages of the main network. Specifically, the shallow features of the three ultrasound modalities are independently input into the main network, and each undergoes preliminary processing by a three-layer modalizable feature extraction module to obtain the mid-level feature representations of the three modalities. Then, these three mid-level feature representations are input into a dynamic adaptive aggregation module to calculate the fusion weights of each modality and perform a weighted summation to obtain the fused feature representation. Finally, the fused feature representation is further input into a subsequent modalizable feature extraction module for deep processing.

[0056] This alternative solution also satisfies the basic idea of ​​dynamically adjusting the fusion process based on the presence of modalities, and is an alternative implementation of the present invention.

[0057] In other embodiments, the parameters generated by the supernetwork are also used to generate channel recalibration coefficients and attention weights.

[0058] Specifically, a channel attention mechanism (Squeeze and Excitation, SE) is introduced in each moduloizable feature extraction module. The hypernetwork generates scaling factors for each channel based on structured clinical prior representations, which are used for adaptive recalibration of each channel of the feature map. Let the input feature map be... Its number of channels is C, and the channel scaling factor vector generated by the hypernetwork is... The output after channel attention modulation is: Here, ⊙ represents element-wise multiplication. This design allows for more precise control over the channel importance of image features in structured clinical information, further enhancing the fusion depth.

[0059] In other embodiments, an independent learnable embedding vector is set for each modality. When a modality is missing, this embedding vector is used as the representation of that modality, and its dimension is the same as the shallow feature dimension. And / or, a gating vector is set. When the mode exists When the mode is missing After shallow feature extraction, suppression is performed using a gating vector: When a modality is missing, its feature contribution is set to zero, and subsequent fusion is based solely on the features of the existing modalities. And / or, when calculating the attention mask, the modality presence state vector is used as a conditional input, causing the attention weights to approach zero on the missing modalities, thereby achieving adaptive modality selection.

[0060] The purpose of the above alternatives is to avoid image-level forgery and to achieve stable compatibility with missing modalities at the network structure level.

[0061] S500. Perform deep feature modulation on the basic image representation based on the dynamic parameter set, and output the predicted probability of transfer risk.

[0062] In some embodiments, such as Figure 5 As shown, the step of performing deep feature modulation on the basic image representation based on the dynamic parameter set includes: S510. Input the basic image representation into the modulating main network, wherein the modulating main network is composed of L modulating feature extraction modules connected in series, where L≥1; S520. Load the dynamic parameter set into a preset modulating layer in the modulating main network; The unified basic image representation obtained in this embodiment Input the main network, and simultaneously load the set of dynamic parameters θ generated by the supernetwork into the preset modulated layer in the main network.

[0063] S530: Features are extracted sequentially through L modulated feature extraction modules to obtain deep image representation; In this embodiment, the main network adopts an improved residual network structure, consisting of L modulated feature extraction modules connected in series. In this embodiment, L=4. Each modulated feature extraction module contains two convolutional layers and one residual connection. The weights and biases of the first convolutional layer are dynamically generated by the supernetwork, and the second convolutional layer uses shared parameters. In addition, the scaling factor and offset factor of the batch normalization layer in each module are also dynamically generated by the supernetwork.

[0064] Suppose the main network consists of L modulated feature extraction modules connected in series. For the th... There are 3 modules, and their input features are denoted as follows. The output features are The corresponding dynamic parameters are The calculation process of this module can then be represented as follows: in, Indicates the first The computation function of a moduliable feature extraction module; The dynamic parameters generated and assigned to this module for the supernetwork specifically include the kernel weights and biases of the first convolutional layer, and the scaling factor γ and offset factor β of the batch normalization layer.

[0065] If expressed as a whole, the deep mapping of the main network can be written as: in, This represents the global mapping function of the deep layers of the main network. This represents the deep imaging characterization obtained under clinical prior constraints.

[0066] Through the above design, different patients will form different deep feature extraction paths based on their individual clinicopathological information. Specifically, the hypernetwork generates personalized convolutional kernel weights and batch normalization parameters based on the patient's structured clinicopathological data, enabling the main network to adaptively adjust according to the patient's overall clinical status when processing image features, thereby achieving deep coupling between structured clinicopathological information and ultrasound image information.

[0067] S540. After performing global feature aggregation on the deep image representation, the predicted probability of the transfer risk is output through the classification layer.

[0068] Characterize the obtained deep images Input classification module. First, spatial dimension compression of deep image representation is performed through global average pooling: in, This represents a global average pooling operation, which compresses the spatial dimension to 1×1 and outputs a feature vector with a dimension of 512.

[0069] Then, the pooled feature vector is input into the classification layer. The classification layer is a fully connected layer with an input dimension of 512 and an output dimension of 1. The output is mapped to the (0,1) interval using the Sigmoid activation function to obtain the predicted probability of axillary lymph node metastasis in the patient. in, Let w and b represent the predicted probability of risk transfer, and w and b be the trainable weights and biases of the classification layer. The Sigmoid activation function is defined as follows: .

[0070] During the training phase, to improve the model's adaptability to situations where real-world clinical modalities are missing, a random image modality discarding strategy was introduced. Considering that B-mode ultrasound is typically an essential basic modality, random discarding was primarily implemented for functional modalities such as elastography and blood flow imaging. For any... Let the random retainable variable be... With probability Take 0 (i.e. discard), with probability Select 1 (i.e., retain). Then, the modal states actually involved in the computation during the training phase are defined as: Among them, when When, mode k is retained in the current training round; when At that time, mode k is randomly masked in the current training round. During training, using Alternative By participating in modality branch selection and fusion weight calculation, the model actively simulates functional modality loss scenarios in real-world clinical settings. This strategy forces the model to learn robust feature representations that do not depend on any specific functional modality, thereby improving the model's generalization ability in real-world clinical environments.

[0071] Let the true label of the sample be ,in This indicates the presence of axillary lymph node metastasis. This indicates the absence of axillary lymph node metastasis. To mitigate the class imbalance problem (in breast cancer patients, negative samples with axillary lymph node metastasis are usually more numerous than positive samples), a weighted binary cross-entropy loss function is used: in, and These represent the loss weights for positive and negative samples, respectively. In this embodiment, the number of positive samples in the training set is assumed to be... The number of negative samples is The weights are then set as follows: , This ensures that positive and negative samples contribute equally to the loss function.

[0072] If parameter constraints are further considered to prevent overfitting, the overall optimization objective can be expressed as: in, λ represents the regularization term. In this embodiment, L2 weight decay is used, and λ represents the regularization coefficient. In this embodiment, λ = 0.0001 is used.

[0073] By minimizing the overall loss function, the Adam optimizer is used for end-to-end joint optimization of the supernetwork and the main network. The initial learning rate is set to 0.001, the batch size to 16, and the number of training epochs to 100. The final result is a breast cancer axillary lymph node metastasis prediction model that combines prediction accuracy with missing robustness.

[0074] During the inference phase, the trained model is used for actual predictions. At this stage, no random discarding is performed; forward propagation is directly performed based on the patient's actual modalities: for existing modalities, shallow features are extracted using real images; for missing modalities, pre-trained learnable baseline features are used to fill the gaps. After dynamic adaptive aggregation and deep feature modulation, the predicted probability of axillary lymph node metastasis risk for the patient is output.

[0075] In other embodiments, the random discarding strategy is further extended: First, the discarding probability adopts a dynamic adjustment strategy, setting a low discarding probability (e.g., 0.1) in the early stage of training and gradually increasing the discarding probability (up to 0.5) as the training rounds increase, so that the model gradually adapts to the lack of modalities; Second, the B-mode ultrasound modality also participates in random discarding with a low probability (e.g., 0.05) to cope with the very few scenarios where the B-mode ultrasound image quality is unacceptable and cannot be used; Third, a structured discarding strategy is adopted, that is, the modality combination is discarded as a whole according to a preset probability distribution, rather than discarding each modality independently.

[0076] The extended random dropout strategy described above can further enhance the model's adaptability to combinations of missing modalities.

[0077] This invention utilizes a baseline feature occupancy mechanism to accommodate arbitrary combinations of 1-3 ultrasound modalities without requiring image-level generation and completion; it achieves deep modulation of the image feature extraction process by generating dynamic parameters through a hypernetwork; it significantly improves the model's robustness to real-world clinical modality loss scenarios through a modality random dropout training strategy; and it designs encoding and compensation mechanisms for missing structured data and missing image modalities to achieve hierarchical processing.

[0078] Example 2 A breast cancer axillary lymph node metastasis risk prediction system is disclosed. This system corresponds to the method described in Example 1. For a detailed description of the method, please refer to the corresponding description in the above method examples, which will not be repeated here. Figure 6 As shown, the system 600 includes: The data acquisition module 610 is used to acquire the patient's structured clinical pathology data and multimodal ultrasound imaging data; the multimodal ultrasound imaging data includes at least one of B-mode ultrasound images, strain elastography images, and color Doppler blood flow imaging images. The structured data encoding module 620 is used to encode the missing state of the structured clinicopathological data and generate a structured representation. Hypernetwork module 630 is used to generate a dynamic parameter set based on the structured representation; The multimodal image fusion module 640 is used to generate basic image representations from the multimodal ultrasound image data through a dynamic adaptive aggregation mechanism; wherein, for existing modalities, shallow features are extracted using real images; for missing modalities, learnable baseline features are used for placeholders; The feature modulation and prediction module 650 is used to perform deep feature modulation on the basic image representation based on the dynamic parameter set and output the predicted probability of transfer risk.

[0079] When the structured data encoding module encodes the missing state of the structured clinical pathology data, it concatenates the numerically filled structured features with the corresponding missing markers to obtain the structured representation.

[0080] The hypernetwork module includes: A structured encoder is used to encode the structured representation to obtain a patient-level clinical prior representation; A parameter generator is used to generate the dynamic parameter set based on the clinical prior characterization.

[0081] The multimodal image fusion module includes: The shallow feature extraction branch is set up separately for B-mode ultrasound images, strain elastic imaging images and color Doppler blood flow imaging images, and is used to extract shallow features from the existing modes. Baseline feature library, which stores learnable baseline features corresponding to each missing modality; The fusion weight calculation unit is used to calculate the fusion response score of each modality based on the shallow features of each modality and the modality existence state vector, and to calculate the fusion weight of each modality based on the normalized exponential mapping. The weighted aggregation unit is used to perform weighted summation of the shallow features of each modality according to the fusion weights to obtain the basic image representation.

[0082] The feature modulation and prediction module includes: The modulated main network consists of L modulated feature extraction modules connected in series, where L≥1; The parameter loading unit is used to load the dynamic parameter set generated by the supernetwork module into the preset modulatorable layer in the modulatorable main network; The classification unit is used to perform global feature aggregation on the deep image representation obtained after deep feature modulation, and then output the predicted probability of the transfer risk through the classification layer.

[0083] The system also includes a training module, which employs an image modality random discarding strategy for training. The image modality random discarding strategy is as follows: during the training phase, at least one of the strain elastic imaging modality and the color Doppler blood flow imaging modality is randomly masked with a preset probability to simulate a functional modality loss scenario in real clinical settings.

[0084] Example 3 A computer device 700, such as Figure 7 As shown, the system includes a memory 710, a processor 720, and a computer program 730 stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a method for predicting the risk of axillary lymph node metastasis in breast cancer. For a detailed description of the method, please refer to the corresponding description in the above method embodiments; it will not be repeated here.

[0085] Example 4 A computer-readable storage medium, such as Figure 8 As shown, a computer program is stored thereon, which, when executed by a processor, implements the steps of a method for predicting the risk of axillary lymph node metastasis in breast cancer. For a detailed description of the method, please refer to the corresponding description in the above method embodiments, which will not be repeated here.

[0086] The number of devices and processing scale described herein are for the purpose of simplifying the description of the invention. Applications, modifications, and variations of the invention will be readily apparent to those skilled in the art.

[0087] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

[0088] The apparatus, computer device, and non-volatile computer storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, computer device, and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, computer device, and non-volatile computer storage medium will not be repeated here.

[0089] Those skilled in the art will also know that, besides implementing the controller in the form of purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller take the form of logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices included within it for implementing various functions can also be considered structures within that hardware component. Alternatively, the devices for implementing various functions can be considered as both software units implementing the method and structures within a hardware component.

[0090] The systems, apparatuses, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above apparatuses are described separately as various units based on their functions. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0091] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0096] This specification may be described in the general context of computer-executable instructions, such as program units, that are executed by a computer. Generally, program units include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification may also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program units may reside in local and remote computer storage media, including storage devices.

[0097] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0098] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A method for predicting the risk of axillary lymph node metastasis in breast cancer, characterized in that, Includes the following steps: Acquire structured clinicopathological data and multimodal ultrasound imaging data of patients; the multimodal ultrasound imaging data includes at least one of B-mode ultrasound images, strain elastography images, and color Doppler flow imaging images; Missing states are encoded in the structured clinicopathological data to generate a structured representation; The structured representation is input into the hypernetwork to generate a dynamic parameter set; The multimodal ultrasound image data is used to generate basic image representations through a dynamic adaptive aggregation mechanism; wherein, for existing modalities, shallow features are extracted using real images; for missing modalities, learnable baseline features are used for placeholders. Based on the dynamic parameter set, deep feature modulation is performed on the basic image representation to output the predicted probability of transfer risk. The step of generating a basic image representation from the multimodal ultrasound image data through a dynamic adaptive aggregation mechanism includes: Shallow features were extracted from the modalities present in B-mode ultrasound images, strain elastography images, and color Doppler blood flow imaging images, respectively, and the corresponding learnable baseline features were used to fill in the missing modalities. The shallow features of each modality are concatenated to obtain the input of the fusion module; The fusion module input and the modal existence state vector input are input to the fusion weight calculation unit to calculate the fusion response score of each modality; The fusion response scores of each modality are converted into fusion weights based on the normalized exponential mapping; The shallow features of each modality are weighted and summed according to the fusion weights to obtain the basic image representation; The step of performing deep feature modulation on the basic image representation based on the dynamic parameter set includes: The basic image representation is input into a modulated main network, which is composed of L modulated feature extraction modules connected in series, where L≥1; The dynamic parameter set is loaded into a preset modulatory layer in the modulatory main network; Feature extraction is performed sequentially through L modulated feature extraction modules to obtain deep image representation; After global feature aggregation of the deep image representation, the predicted probability of migration risk is output through a classification layer.

2. The method for predicting the risk of axillary lymph node metastasis in breast cancer as described in claim 1, characterized in that, The missing state encoding of the structured clinicopathological data specifically includes: The structured features, after numerical imputation, are concatenated with the corresponding missing markers to obtain the structured representation.

3. The method for predicting the risk of axillary lymph node metastasis in breast cancer as described in claim 1, characterized in that, The step of inputting the structured representation into the hypernetwork to generate a dynamic parameter set includes: The structured representation is encoded by a structured encoder in a hypernetwork to obtain a patient-level clinical prior representation; The dynamic parameter set is generated by a parameter generator in the hypernetwork based on the clinical prior characterization.

4. The method for predicting the risk of axillary lymph node metastasis in breast cancer as described in claim 1, characterized in that, The modal existence state vector is used to indicate the presence of each of the B-mode ultrasound image, strain elastic imaging image, and color Doppler blood flow imaging image.

5. The method for predicting the risk of axillary lymph node metastasis in breast cancer as described in claim 1, characterized in that, It also includes training steps: During the training phase, at least one of the strain elastic imaging modal and color Doppler blood flow imaging modal is randomly masked with a preset probability. The state of the randomly masked modal is used to participate in shallow feature extraction and fusion weight calculation to simulate the functional modality loss scenario in real clinical practice.

6. The method for predicting the risk of axillary lymph node metastasis in breast cancer as described in claim 1, characterized in that, When outputting the predicted probability of transfer risk, a weighted binary classification loss function is used for optimization, with different loss weights assigned to positive and negative samples.

7. A breast cancer axillary lymph node metastasis risk prediction system, characterized in that, include: The data acquisition module is used to acquire the patient's structured clinicopathological data and multimodal ultrasound imaging data; the multimodal ultrasound imaging data includes at least one of B-mode ultrasound images, strain elastography images, and color Doppler flow imaging images. The structured data encoding module is used to encode the missing states of the structured clinicopathological data and generate a structured representation. The hypernetwork module is used to generate a dynamic parameter set based on the structured representation; The multimodal image fusion module is used to generate basic image representations from the multimodal ultrasound image data through a dynamic adaptive aggregation mechanism; wherein, for existing modalities, shallow features are extracted using real images; for missing modalities, learnable baseline features are used for placeholders; The feature modulation and prediction module is used to perform deep feature modulation on the basic image representation based on the dynamic parameter set and output the predicted probability of transfer risk. The multimodal image fusion module includes: The shallow feature extraction branch is set up separately for B-mode ultrasound images, strain elastic imaging images and color Doppler blood flow imaging images, and is used to extract shallow features from the existing modes. Baseline feature library, which stores learnable baseline features corresponding to each missing modality; The fusion weight calculation unit is used to calculate the fusion response score of each modality based on the shallow features of each modality and the modality existence state vector, and to calculate the fusion weight of each modality based on the normalized exponential mapping. The weighted aggregation unit is used to perform weighted summation of the shallow features of each modality according to the fusion weights to obtain the basic image representation; The feature modulation and prediction module includes: The modulated main network consists of L modulated feature extraction modules connected in series, where L≥1; The parameter loading unit is used to load the dynamic parameter set generated by the supernetwork module into the preset modulatorable layer in the modulatorable main network; The classification unit is used to perform global feature aggregation on the deep image representation obtained after deep feature modulation, and then output the predicted probability of the transfer risk through the classification layer.

8. The breast cancer axillary lymph node metastasis risk prediction system as described in claim 7, characterized in that, When the structured data encoding module encodes the missing state of the structured clinical pathology data, it concatenates the numerically filled structured features with the corresponding missing markers to obtain the structured representation.

9. A breast cancer axillary lymph node metastasis risk prediction system as described in claim 7, characterized in that, The hypernetwork module includes: A structured encoder is used to encode the structured representation to obtain a patient-level clinical prior representation; A parameter generator is used to generate the dynamic parameter set based on the clinical prior characterization.

10. The breast cancer axillary lymph node metastasis risk prediction system as described in claim 7, characterized in that, The system also includes a training module, which employs an image modality random discarding strategy for training. The image modality random discarding strategy is as follows: during the training phase, at least one of the strain elastic imaging modality and the color Doppler blood flow imaging modality is randomly masked with a preset probability to simulate a functional modality loss scenario in real clinical settings.

11. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

12. A computer-readable storage medium, characterized in that, It stores program instructions that, when executed, implement the method as described in any one of claims 1 to 6.

Images