A liver ultrasound thermogram generation method based on multi-modal learning

Abstract

The application belongs to the technical field of computer vision and medical image recognition, and discloses a liver ultrasound thermal map generation method based on multi-modal learning. Liver ultrasound images are cut into multiple instance image blocks, and bag-level image features are obtained through gated attention multi-instance pooling. Structured clinical indicators are input into a clinical feature encoding module to obtain clinical embedding features, which are fused with the bag-level image features to obtain cascade fusion features. A joint multi-element loss function including a main classification loss, a spatial attention prior loss and a background negative inhibition loss is constructed to complete model training. Based on the trained multi-modal multi-instance learning model, the attention weights of each instance image block in the liver ultrasound image to be detected are obtained, and a liver ultrasound thermal map is generated by mapping, so that the distribution of the potential region of interest in the ultrasound image is directly presented, and the positioning accuracy and interpretability of the computer processing result of the liver ultrasound image are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and medical image recognition technology, and in particular to a method for generating liver ultrasound thermograms based on multimodal learning. Background Technology

[0002] Schistosomiasis-related liver fibrosis is a significant public health burden in endemic areas, and large-scale screening at the grassroots level is crucial for the early detection and control of the disease. In this context, B-mode ultrasound has become the primary means of community screening due to its significant advantages such as being non-invasive, portable, and low-cost. However, the interpretation of traditional ultrasound examination results relies heavily on the clinical experience and subjective judgment of physicians, leading to a serious lack of diagnostic consistency among different operators in primary healthcare institutions, which can easily result in missed diagnoses or misdiagnoses.

[0003] In recent years, with the rapid development of deep learning technology, intelligent auxiliary diagnostic methods for medical images based on convolutional neural networks (CNN) and visual transformers (ViT) have been gradually introduced into the task of liver fibrosis staging. However, in the actual scenario of schistosomiasis screening in primary communities, existing deep learning diagnostic methods still have significant technical bottlenecks.

[0004] First, ultrasound images collected at the grassroots level usually contain a large number of near-field artifacts, gas interference, and irrelevant background noise. Conventional deep learning models tend to capture these background texture features that are irrelevant to the pathological mechanism when there is a lack of strong supervision constraints. This results in a "quick learning" phenomenon, which leads to a significant decrease in the discriminative stability of the model when it is deployed on an independent test set or across centers.

[0005] Secondly, existing strongly supervised methods rely heavily on the intensive manual annotation of regions of interest (ROI) in ultrasound images by clinical experts, which is extremely labor-intensive and impractical in grassroots screening with massive amounts of data. While standard weakly supervised multiple instance learning methods only require sample-level labels, their internal decision-making mechanism is like a "black box," and their high attention weight regions are often outside the actual liver parenchyma lesions, lacking spatial interpretability.

[0006] Furthermore, most existing auxiliary diagnostic methods rely solely on a single ultrasound imaging modality, or simply perform late-stage decision stitching of images and structured clinical indicators (such as serum fibrosis indicators and liver function indicators), failing to effectively achieve deep semantic feature complementarity and fusion of heterogeneous multimodal data.

[0007] Finally, traditional AI models for liver fibrosis typically use a single-stage, multi-classification architecture for staging, blurring the boundaries between routine screening (separation of normal and abnormal) and severity stratification (separation of mild and moderate-to-severe) in the clinical treatment pathway, resulting in a serious disconnect from the actual workflow of two-stage sequential diagnosis in real clinical practice.

[0008] Therefore, how to design a method that can effectively integrate ultrasound images with clinical laboratory indicators and intuitively present the distribution of potential regions of interest in ultrasound images without increasing the burden of manual annotation during the inference period is a key technical problem that urgently needs to be solved. Summary of the Invention

[0009] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of the existing liver ultrasound visualization schemes, such as the inability to integrate images and structured clinical data, poor adaptability of feature aggregation methods, lack of effective background suppression and spatial prior constraints, large background interference in the generated attention heatmap, insufficient accuracy of region of interest localization, and weak model interpretability.

[0010] To address the aforementioned technical problems, this invention provides a method for generating liver ultrasound thermograms based on multimodal learning, comprising: Obtain a training sample set, in which each sample includes a sample-level category label for the same subject, at least one liver ultrasound image, and structured clinical indicators and region of interest annotation information corresponding to each liver ultrasound image; Each sample is input into a multimodal, multi-instance learning model to obtain the binary classification predicted label for each sample, including: The liver ultrasound image is divided into multiple instance image blocks. The instance features of each instance image block are extracted and input into the gated attention multi-instance pooling module to obtain the attention weights of each instance image block. The instance image blocks are then weighted and aggregated to obtain bag-level image features. Structured clinical indicators are input into the clinical feature encoding module to obtain clinical embedded features; The bag-level image features are concatenated with the clinical embedding features to obtain the cascaded fused features; The cascaded fused features are input into the sample-level classifier to obtain the predicted probability of the binary classification label; Based on the predicted probability of binary classification labels and the Focal Loss of sample-level class labels, a main classification loss is constructed; Based on the spatial positional relationship between each instance image patch and the region of interest (ROI) annotation information, the instance-level ROI labels of each instance image patch are obtained, normalized and smoothed, and the instance-level ROI prior distribution is obtained. Based on the instance-level ROI prior distribution and the attention weights of each instance image patch, the cross-entropy is calculated, and the spatial attention prior loss is constructed. From instance image patches whose instance-level ROI labels are non-ROI labels, select a preset proportion of instance image patches as hard negative instance image patches in descending order of attention weight; calculate the average of the attention weights of all hard negative instance image patches, and combine it with a preset interval threshold to construct the background negative suppression loss. The main classification loss, spatial attention prior loss, and background negative inhibition loss are weighted and summed to construct a joint multivariate loss function. This function is then used to train the multimodal multi-instance learning model, resulting in a well-trained multimodal multi-instance learning model. The liver ultrasound image to be detected and its corresponding structured clinical indicators are input into a trained multimodal multi-instance learning model to obtain the attention weights of each instance image patch in the liver ultrasound image to be detected, and then the liver ultrasound heat map is generated.

[0011] The liver ultrasound thermogram generation method based on multimodal learning provided in this application has the following beneficial effects: This invention integrates liver ultrasound imaging features with structured clinical indicators in a multimodal manner, making full use of heterogeneous data information, mitigating feature extraction errors caused by the low signal-to-noise ratio of a single ultrasound image, and improving the accuracy of attention weight calculation. A spatial attention prior constraint mechanism guided by ROI was designed. By establishing a mapping error penalty between the model's adaptive spatial attention distribution and medical priors, the model is strongly guided to gather effective decision information, making the internal black box mechanism transparent and greatly improving spatial interpretability. Meanwhile, a background negative suppression mechanism was proposed. By dynamically filtering and severely penalizing hard negative samples in non-ROI regions during iteration, the "quick learning" path caused by near-field artifacts and irrelevant noise was cut off at the pixel and decision layers, giving the model extremely high robustness and cross-center generalization ability. This invention relies on the output instance attention weights of a pre-trained multimodal multi-instance learning model to generate a liver ultrasound thermogram, which intuitively displays the spatial distribution of potential regions of interest in the ultrasound image of the liver to be detected, thereby improving the interpretability of the ultrasound image analysis process. Attached Figure Description

[0012] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 The flowchart shows the steps of a method for generating liver ultrasound thermograms based on multimodal learning. Figure 2 This is a schematic diagram illustrating the principle of a liver ultrasound thermogram generation method based on multimodal learning. Figure 3 This is a schematic diagram comparing the spatial attention response heatmap and ROI of the method of the present invention with those of the existing ABMIL network and late fusion network on full-view ultrasound images at different pathological stages of liver fibrosis. Detailed Implementation

[0013] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0014] Please see Figure 1 , Figure 1 The diagram shows the steps of the liver ultrasound thermogram generation method based on multimodal learning, including steps S101 to S107.

[0015] S101: Obtain the training sample set, in which each sample includes a sample-level category label for the same subject, at least one liver ultrasound image, and structured clinical indicators and region of interest annotation information corresponding to each liver ultrasound image.

[0016] S102: Input each sample into the multimodal multi-instance learning model to obtain the binary classification prediction label of each sample, including steps S102-1 to S102-4.

[0017] S102-1: Divide the liver ultrasound image into multiple instance image blocks, extract instance features from each instance image block, input them into a gated attention multi-instance pooling module to obtain the attention weights of each instance image block, perform weighted aggregation on each instance image block, and obtain bag-level image features, including: For one or more liver ultrasound images in any sample, the effective fan-shaped imaging region of the liver ultrasound image is extracted, and overlapping sliding sampling is performed using a preset sliding window based on a preset sliding step size to obtain multiple candidate image blocks. The candidate image blocks are selected as instance image blocks, with a proportion greater than a preset proportion value in the effective fan-shaped imaging region and an average gray value greater than a preset gray value. Instance features of each instance image patch are extracted using the ResNet50 backbone network; The features of each instance corresponding to one or more liver ultrasound images belonging to the same sample are input into the gated attention multi-instance pooling module to obtain the attention score of each instance image block. The attention score is then normalized among all instance image blocks in the same sample to obtain the attention weight of each instance image block. The bag-level image features of a sample are obtained by weighting and summing the features of each instance based on the attention weights of each instance image patch.

[0018] Further, in this embodiment, the effective fan-shaped imaging region of the liver ultrasound image is extracted; a 96×96 pixel sliding window is used with a sliding step of 24 pixels to perform overlapping sliding sampling within the effective fan-shaped imaging region to obtain multiple candidate image blocks, and the position coordinates of each candidate image block in the original liver ultrasound image are recorded; the candidate image blocks are filtered according to the proportion of the effective fan-shaped imaging region and the average gray value, and candidate image blocks with an effective fan-shaped imaging region proportion greater than 85% and an average gray value greater than 10 are retained; the filtered instance image blocks are adjusted to 224×224 pixels and input into the ResNet50 backbone network pre-trained on the RadImageNet medical image dataset to obtain the corresponding 2 The initial 2048-dimensional instance features are processed by layer normalization and mapped to 512 dimensions through a fully connected feature transformation layer. These 512-dimensional instance features are then processed by a ReLU activation function and a random inactivation layer with a random inactivation rate of 0.25. The instance features corresponding to one or more liver ultrasound images belonging to the same sample are input into a gated attention multi-instance pooling module to obtain the attention score for each instance image block. The attention score is then normalized across all instance image blocks in the same sample to obtain the attention weight for each instance image block. Based on the attention weight of each instance image block, the corresponding 512-dimensional instance features are weighted and summed to obtain the 512-dimensional bag-level image features of the sample.

[0019] Among them, the The acquisition of attention weights for each instance image patch includes: The first 512-dimensional instance features of an instance image patch Linear transformations and activations are performed on the hyperbolic tangent-gated branch and the sigmoid-gated branch, respectively. In the hyperbolic tangent-gated branch, a learnable linear transformation matrix is ​​utilized. and bias vector For instance features A linear transformation is performed, followed by nonlinear processing using the hyperbolic tangent activation function; in the Sigmoid gated branch, a learnable linear transformation matrix is ​​utilized. and bias vector For instance features Perform a linear transformation and then use the Sigmoid activation function for nonlinear processing; The outputs of the hyperbolic tangent-gated branch and the sigmoid-gated branch are multiplied element-wise, and a learnable weight vector is used. and bias scalar Perform a linear mapping to obtain the first... Attention score for each instance image patch , is represented as: ; Among all instance image patches in the same sample, the attention score of each instance image patch is normalized using Softmax to obtain the first... Attention weights for each instance image patch , is represented as: ; in, Indicates the first The 512-dimensional instance feature row vectors obtained by mapping each instance image patch through a fully connected feature transformation layer , Indicates the sequence number of the instance image patch. , , This indicates the total number of instance image blocks containing one or more liver ultrasound images within the same sample; This represents the hyperbolic tangent activation function. and Let represent the learnable linear transformation matrix and the learnable bias column vector in the hyperbolic tangent gating branch, respectively; This represents the Sigmoid activation function. and These represent the learnable linear transformation matrix and the learnable bias column vector in the Sigmoid gated branch, respectively. and These represent the learnable weight column vector and the learnable bias scalar, respectively, for mapping attention scores. Represents element-wise multiplication. Indicates transpose; .

[0020] S102-2: Input structured clinical indicators into the clinical feature encoding module to obtain clinical embedded features, including: Based on the structured clinical indicators of each sample in the training sample set, the mean and standard deviation of each indicator are obtained. Z-score standardization is performed on each indicator in the structured clinical indicators corresponding to the liver ultrasound image to obtain the standardized structured clinical indicators. Standardized structured clinical indicators The input to the clinical feature encoding module passes through a fully connected mapping layer and a normalization layer in sequence. ReLU activation layer and random deactivation layer Obtain clinical embedding features , is represented as: ; in, and This represents the learnable weight matrix and learnable bias column vector of the fully connected mapping layer; structured clinical indicators include: age, albumin (ALB), laminin (LN), hemoglobin (HGB), hyaluronic acid (HA), - Gamma-glutamyl transferase (GGT), low-density lipoprotein cholesterol (LDL-C), and liver fibrosis index (FIB-4).

[0021] Specifically, the 8-dimensional structured clinical feature vector corresponding to the sample is obtained. Based on the training samples used to train the current multimodal multi-instance learning model, the mean and standard deviation of each structured clinical indicator are calculated, and the mean and standard deviation are used to perform Z-score standardization on each structured clinical indicator of the training sample. When using the current multimodal multi-instance learning model to process the sample to be visualized, the mean and standard deviation determined during the training phase of the model are used to perform Z-score standardization on each structured clinical indicator of the sample to be visualized.

[0022] Specifically, no. The standardized results of each structured clinical indicator are expressed as follows: , ;in, Represents the sample's first The raw values ​​of each structured clinical indicator; and They represent the first and second digits calculated based on the training set, respectively. The mean and standard deviation of each structured clinical indicator; when At that time, it will be used for standardization processing. Set to 1 to avoid division by zero during the standardization process; Indicates the first The standardized results of eight structured clinical indicators; the eight standardized structured clinical indicators constitute an 8-dimensional standardized clinical feature column vector. ; 8-dimensional standardized clinical feature vector Inputting into the clinical feature encoding network yields 128-dimensional clinical embedding features. .

[0023] S102-3: The bag-level image features are concatenated with the clinical embedding features to obtain the cascaded fusion features.

[0024] This embodiment incorporates 128-dimensional clinical embedded features. The 512-dimensional bag-level image features are concatenated along the feature dimensions to obtain 640-dimensional cascaded fused features.

[0025] S102-4: Input the cascaded fused features into the sample-level classifier to obtain the predicted probability of the binary label.

[0026] S103: Based on the predicted probability of the binary classification labels and the Focal Loss of the sample-level class labels, construct the main classification loss, expressed as: ; in, Indicates the primary classification loss. Indicates binary category label The predicted probability; Indicates sample-level category labels. ; It represents the natural logarithm.

[0027] In this embodiment, 640-dimensional cascaded fused features are input into a sample-level classifier. The sample-level classifier sequentially includes a first fully connected layer that maps 640 dimensions to 64 dimensions, a ReLU activation layer, a random inactivation layer with a random inactivation rate of 0.25, and a second fully connected layer that maps 64 dimensions to 2 dimensions, resulting in a sample-level class prediction score vector containing two elements. The sample-level class prediction score vector is then Softmax normalized to obtain the sample-level class prediction probability. Based on the prediction probability corresponding to the real sample-level binary classification label, a Focal Loss with a focusing parameter of 2 is used to construct the main classification loss.

[0028] S104: Based on the spatial positional relationship between each instance image patch and the region of interest (ROI) annotation information, obtain the instance-level ROI labels for each instance image patch, perform normalization and smoothing, and obtain the instance-level ROI prior distribution; calculate the cross-entropy based on the instance-level ROI prior distribution and the attention weights of each instance image patch, and construct the spatial attention prior loss, including: Get the Instance-level ROI labels for each instance image patch , , Indicates the first Each instance image patch is located within the ROI region. Indicates the first Each instance image patch is located in a non-ROI region; Count the number of instance image patches located within the ROI region in the same sample. and in At that time, the instance-level ROI labels of each instance image patch are normalized and smoothed to obtain the first... Instance-level ROI prior probabilities corresponding to each instance image patch , is represented as: ; Based on instance-level ROI prior probability Attention weights relative to the corresponding instance image patch Constructing spatial attention prior loss , is represented as: ; in, , This indicates the total number of instance image patches contained in one or more liver ultrasound images within the same sample. 1; This represents the preset smoothing coefficient, with a value of 0.1. It is a very small positive number, used to avoid the occurrence of The resulting numerical anomaly has a value of ; .

[0029] S105: From instance image patches with non-ROI labels at the instance level, select a preset proportion of instance image patches as hard negative instance image patches in descending order of attention weight; calculate the average of the attention weights of all hard negative instance image patches, and combine it with a preset interval threshold to construct the background negative suppression loss, expressed as: ; When there are no instance image patches located outside the ROI region, the hard negative instance set is empty, and the background negative suppression loss is applied. =0; in, This represents the set of indices for hard negative instance image patches. express Total number of indexes in the database; This represents the preset interval threshold, which is 0.15 in this embodiment.

[0030] Among them, instance-level ROI tags The instance image patch is determined as the background instance image patch, and a background instance set is constructed. , is represented as: And count the number of background instance image patches in the same sample. , is represented as: ;when At that time, the background instance image patches are sorted in descending order of attention weight, and the top ones are selected. Each background instance image patch is used as a hard negative instance image patch to form a hard negative instance set. , , , This represents the preset selection ratio of the hard negative instance image patch, with a value of 0.20. This indicates the rounding operation. Indicates before selection Each element.

[0031] S106: The main classification loss, spatial attention prior loss, and background negative inhibition loss are weighted and summed to construct a joint multivariate loss function. The multimodal multi-instance learning model is then trained to obtain the trained multimodal multi-instance learning model.

[0032] Specifically, the joint loss function is expressed as: ; in, and These represent the preset weight coefficients for the spatial attention prior loss and the background negative suppression loss, respectively; in this embodiment, The value is set to 0.15. The value is set to 0.05.

[0033] S107: Input the ultrasound image of the liver to be detected and its corresponding structured clinical indicators into the trained multimodal multi-instance learning model to obtain the attention weights of each instance image patch in the ultrasound image of the liver to be detected, and map them to generate a liver ultrasound heat map.

[0034] Specifically, the construction of a liver ultrasound thermogram includes: Obtain the spatial coordinates and attention weights of each instance image patch in the ultrasound image of the liver to be detected; Construct an original thermal intensity matrix with the same height and width as the ultrasound image of the liver to be detected, and initialize each pixel value in the original thermal intensity matrix to 0; Based on the spatial coordinates of each instance image patch, the attention weights of each instance image patch are accumulated to the corresponding pixel positions in the original thermal intensity matrix to obtain the weighted thermal intensity matrix. The weighted thermal intensity matrix is ​​normalized to its maximum value to obtain a normalized thermal intensity matrix with values ​​ranging from 0 to 1. The normalized thermal intensity matrix is ​​converted into a color thermal layer, and the color thermal layer is overlaid onto the ultrasound image of the liver to be examined with a preset transparency to obtain a liver ultrasound thermal map.

[0035] The trained multimodal multi-instance learning model includes five sub-models trained using five-fold hierarchical cross-validation. For each sub-model, the mean and standard deviation of each structured clinical indicator are calculated based on the corresponding training subset. The mean and standard deviation are then used to perform Z-score standardization on the samples to be visualized input to that sub-model. Based on this, a weighted heatmap matrix is ​​obtained, including: Using the sub-models of five multimodal multi-instance learning models trained based on five-fold hierarchical cross-validation, the first-order sub-models are obtained respectively. The five attention weights of each instance image patch are averaged to obtain the first attention weight. Integrated attention weights for each instance image patch , is represented as: ; Based on the Integrated attention weights for each instance image patch The pixel position in the ultrasound image of the liver to be tested is obtained by determining its spatial coordinates. Weighted thermal intensity at the point , is represented as: ; A weighted thermal intensity matrix is ​​formed based on the weighted thermal intensity at each pixel location in the ultrasound image of the liver to be detected; in, Indicates the first Sub-models ( The output of the first Attention weights for each instance image patch; Indicates the first Each instance image patch corresponds to a pixel region in the ultrasound image of the liver to be detected. Indicates an indicative function, pixel position. Located in the pixel area The value is 1 if the time condition is met, and 0 otherwise. This indicates the number of instance image patches belonging to the ultrasound image of the liver to be examined. This indicates the image group number of the ultrasound image of the liver to be examined.

[0036] Furthermore, obtain the maximum thermal intensity in the weighted thermal intensity matrix. ,in The original thermal intensity at each pixel location is normalized to its maximum value, and expressed as: , ; in, Indicates pixel position The corresponding normalized thermal intensity; This represents the maximum thermal intensity in the weighted thermal intensity matrix.

[0037] Furthermore, the weighted thermal intensity matrix is ​​normalized to its maximum value, resulting in a normalized thermal intensity matrix with values ​​ranging from 0 to 1; the element values ​​are then... Each element in the normalized thermodynamic intensity matrix is ​​linearly mapped to Obtain the grayscale matrix; use JET color mapping to convert the grayscale matrix into a color heatmap layer, and then apply preset overlay weights. The color thermal layer is weighted and superimposed on the ultrasound image of the liver to be examined to obtain the liver ultrasound thermal map, which is represented as follows: ;in, Indicates the pixel position in the liver ultrasound thermogram. Pixel value at that location, Indicates the pixel position in the ultrasound image of the liver to be examined. Pixel value at that location, Indicates the pixel position in the color heatmap layer The pixel value at that location.

[0038] In this embodiment, the color thermal layer is weighted and superimposed with the corresponding liver ultrasound image to be visualized according to a superposition weight of 0.45 to obtain a liver ultrasound thermal map, represented as follows: .

[0039] This invention integrates liver ultrasound image features with structured clinical indicators through multimodal feature fusion, fully utilizing heterogeneous data information to alleviate feature extraction errors caused by the low signal-to-noise ratio of a single ultrasound image and improve the accuracy of attention weight calculation. A ROI-guided spatial attention prior constraint mechanism is designed, which establishes a mapping error penalty between the model's adaptive spatial attention distribution and medical priors, strongly guiding the model to converge effective decision information and making the internal black-box mechanism transparent, greatly improving spatial interpretability. Simultaneously, a background negative suppression mechanism is proposed, which dynamically filters and heavily penalizes hard negative samples in non-ROI regions during iteration, cutting off the "fast learning" path caused by near-field artifacts and irrelevant noise at the pixel and decision layers, giving the model extremely high robustness and cross-center generalization ability. This invention relies on the output instance attention weights of a pre-trained multimodal multi-instance learning model to generate a liver ultrasound heatmap, intuitively displaying the spatial distribution of potential regions of interest in the liver ultrasound image to be detected, improving the interpretability of the ultrasound image analysis process.

[0040] Based on the above embodiments, the liver ultrasound thermogram generation method based on multimodal learning provided by this invention is applied to the visualization of ultrasound images of schistosomiasis-related liver fibrosis, and further intelligent analysis is performed based on the visualization results. Addressing the technical bottlenecks of existing intelligent diagnostic models for schistosomiasis-related liver fibrosis, such as susceptibility to ultrasound artifacts, poor discrimination stability due to "rapid learning" under weak supervision, lack of spatial interpretability in the decision-making mechanism, and insufficient in-depth fusion of heterogeneous multimodal features, the method provided by this invention significantly enhances the model's spatial focusing ability on key fibrotic lesions in the liver parenchyma without requiring manual annotation during the inference period, and improves the robustness of cross-center deployment.

[0041] Reference Figure 2The diagram illustrates the principle of a liver ultrasound thermal image generation method based on multimodal learning. This embodiment adaptively extracts bag-level features from full-field ultrasound images using linear projection and gated pooling, and employs a multilayer perceptron to process heterogeneous structured clinical indicators, achieving cascaded splicing and fusion of cross-modal isomorphic features. During network training, a dual-domain multivariate joint loss function is constructed, consisting of main classification loss, spatial attention prior loss, and background negative inhibition loss. Through the synergistic operation of positive lesion focusing guidance and negative artifact abnormality gradient penalty, the model's ability to maintain spatial consistency in the solid lesion region is enhanced. In the inference application stage, a two-stage sequential task integration strategy without manual annotation is adopted, sequentially activating fibrosis detection and severity hierarchical classifiers, ultimately generating a three-stage liver fibrosis grading result with high multimodal semantic complementarity and spatial interpretability. Specific visualization steps include S201 to S205.

[0042] S201: Dataset preparation and preprocessing.

[0043] First, this embodiment collects and organizes a multimodal dataset of schistosomiasis-related liver fibrosis required for model training and independent testing. This dataset contains data from 1,280 real, complete cases screened in grassroots communities in 10 schistosomiasis-endemic counties, including full-field ultrasound images (B-mode ultrasound) and corresponding structured clinical laboratory indicators. All cases were clearly classified into three pathological stages using clinical gold standards: normal stage (F0), mild fibrosis stage (F1), and moderate to severe fibrosis stage (F2–F3). Simultaneously, two senior clinical ultrasound experts collaboratively and manually delineated regions of interest (ROIs) on the training set images. These expert ROI annotations were used to construct spatial attention priors during the training phase. Expert ROI annotations on the independent test set were only used for spatial localization consistency evaluation after model training; they were not used as model input and did not participate in model parameter optimization or inference. Manual ROI annotations are not required during the actual deployment phase.

[0044] The development set (training and validation) and the independent test set were randomly divided in a ratio of approximately 7:3, with 895 cases in the development set and 385 cases in the independent external test set, ensuring the balance of each stage across the two data distributions. All input ultrasound full-field-of-view images underwent standardized preprocessing, including pixel value normalization (subtracting the global mean and dividing by the standard deviation) and nonlinear contrast equalization. To enhance the model's cross-center generalization ability and suppress device heterogeneity caused by different ultrasound beam attenuation and gain adjustments, data augmentation strategies were designed, including random horizontal and vertical flipping (both with a probability of 0.5), random rotation (±15 degrees), and random brightness and contrast adjustments (±0.2), enriching the diversity of image instances within the "bag" during multi-instance learning.

[0045] S202: Design the RAIM-MIL model architecture and initialize it.

[0046] A pre-trained ResNet50 backbone network was used as the instance-level feature extraction network. The selected instance image patches were adjusted to 224×224 pixels and then input into the ResNet50 backbone network to obtain 2048-dimensional initial instance features for each instance image patch. These 2048-dimensional initial instance features were then sequentially input into a layer normalization layer, a fully connected feature transformation layer that maps the 2048 dimensions to 512 dimensions, a ReLU activation layer, and a random deactivation layer with a random deactivation rate of 0.25 to obtain 512-dimensional instance features.

[0047] The 512-dimensional instance features corresponding to one or more liver ultrasound images belonging to the same sample are input into the gated attention multi-instance pooling module to obtain the attention weight of each instance image block. Based on the attention weight, the instance features are weighted and aggregated to obtain 512-dimensional bag-level image features.

[0048] After Z-score standardization of the 8-dimensional structured clinical indicators, they are input into the clinical feature encoding module. The inputs are then processed sequentially through a fully connected mapping layer that maps the 8-dimensional indicators to 128-dimensional indicators, a layer normalization layer, a ReLU activation layer, and a random inactivation layer with a random inactivation rate of 0.20, to obtain 128-dimensional clinical embedded features.

[0049] Set the hyperparameters for the joint multivariate loss function and inference strategy: Set the attention prior loss hyperparameter to... The background negative suppression loss hyperparameter is set to... Label smoothing coefficient Preset selection ratio for hard negative instance image patches ; Preset interval threshold for background negative suppression loss .

[0050] S203: Construct a multimodal feature encoding and cascading fusion mechanism.

[0051] This embodiment first analyzes the mapping relationship between heterogeneous clinical variables and the pathological aggravation of liver fibrosis using a multilayer perceptron fusion network of clinical indicators. The input is a structured clinical feature vector. Includes age, albumin (ALB), laminin (LN), hemoglobin (HGB), hyaluronic acid (HA), - Gamma-glutamyl transferase (GGT), low-density lipoprotein cholesterol (LDL-C), and liver fibrosis index (FIB-4).

[0052] Z-score normalization is applied to the heterogeneous clinical feature vectors to eliminate differences in dimensions and data types. The normalized 8-dimensional clinical feature vectors are then input into the clinical feature encoding module, passing sequentially through a fully connected mapping layer that maps the 8-dimensional vectors to 128-dimensional vectors, a layer normalization layer, a ReLU activation layer, and a random inactivation layer with a random inactivation rate of 0.20, resulting in 128-dimensional clinical embedding features. 512-dimensional bag-level image features With 128-dimensional clinical embedding features By concatenating along the feature dimensions, a 640-dimensional cascaded fused feature is obtained. , is represented as: ; In the formula, This represents a direct feature vector concatenation operation, generating cascaded fused features. It effectively captured the synergistic evolution between changes in macroscopic serological indicators (such as elevated serum fibrosis indicators and abnormal liver function) and ultrasound acoustic abnormalities (such as increased and uneven parenchymal echoes).

[0053] S204: Construct a joint multivariate loss function for classification and spatial constraints.

[0054] To address the issues of weakly supervised multi-instance learning easily falling into fast learning traps and lacking spatial interpretability, this embodiment constructs a total loss function jointly optimized by the main classification loss domain, the attention prior constraint domain, and the background suppression domain during the training phase, expressed as: ; Among them, spatial attention prior loss By introducing expert ROI labels, the model's gaze region is forced to the actual liver parenchyma region, as shown below: ; Meanwhile, to address the technical shortcomings of near-field artifacts in ultrasound images and the tendency for abdominal wall fat interference to be misidentified as valid fibrotic nodules by the network, a background negative suppression loss mechanism was constructed. , is represented as: ;in, The construction is the same as the method described in step S105, determining a set of indices of hard negative instance samples from non-ROI instance image patches. The background negative suppression loss is constructed based on the average attention weight of each instance image patch in the index set.

[0055] S205: An unlabeled two-stage sequential integration strategy for network inference.

[0056] In the reasoning phase, this embodiment redefines the diagnosis of schistosomiasis integrating multimodal features as a cascaded sequential conditional decision problem. Multimodal cascaded fusion features are then used. The pre-trained system inference stream is input to sequentially activate a cascaded two-stage classifier, enabling adaptive diagnosis from general screening to severity stratification. Phase 1, Task 1: Fibrosis Detection: Calculate the conditional probability that the liver ultrasound image belongs to the normal phase (F0) or the fibrotic phase (F1–F3); if the interception condition is met... If the flow is terminated, the output liver ultrasound image will be in the normal phase (F0).

[0057] The second stage, Task 2, is a severity stratification detection: If the first stage determines that there is a risk of fibrosis in the liver ultrasound image, the second stage task is immediately cascaded and activated to calculate the classification probability of further stratification; when the probability is less than 0.5, the output is mild fibrosis stage (F1), and when it is greater than or equal to 0.5, the output is moderate to severe fibrosis stage (F2–F3).

[0058] Based on the liver ultrasound thermogram provided by this invention, a two-stage classifier can be used to obtain conditional probabilities for further fibrosis identification.

[0059] Based on the above embodiments, before performing core training, the system's hardware and software training and evaluation environment is first configured and optimized. To ensure both algorithm convergence accuracy and computational energy efficiency, this embodiment uses a consumer-grade high-efficiency graphics card for hardware computing power support. At the software level, since the feature dimension of ultrasound image instances reaches 2048 dimensions, and the clinical structured input has multi-index heterogeneity, to prevent gradient explosion and improve data throughput, the system compilation framework explicitly calls PyTorch's Automatic Mixed Precision Training (AMP) operator and automatic gradient clipping mechanism.

[0060] For the heterogeneous feature fusion mechanism, the hidden layer dimension of the ClinicalMLP is uniformly configured to 128 dimensions, and nested with LayerNorm and an overfitting suppression layer with a Dropout probability of 0.20. The hybrid multimodal classifier uses two fully connected mapping layers with a Dropout probability of 0.25. In addition, the system constructs a generalization dynamic monitoring mechanism based on bootstrap resampling in the evaluation stream. By setting the number of resampling rounds to 2000, the system's AUC confidence interval and multi-objective loss decay curve on the independent validation set are tracked in real time to ensure the stability and convergence effectiveness of the training process under limited hardware resources.

[0061] The specific hardware and software experimental environment for system training and evaluation, as well as the detailed parameters for network model optimization in this embodiment, are shown in Table 1: Table 1 Experimental Environment Configuration Table

[0062] Load the preprocessed development set data and perform iterative training according to the configuration in Table 1. After the model training is completed, in order to comprehensively and quantitatively evaluate the real technical contribution of the RAIM-MIL architecture proposed in this invention to the interpretability of spatial decision-making, a rigorous quantitative evaluation was performed on the independent test set to assess the consistency between the spatial attention distribution learned by the model and the regions of interest (ROIs) manually delineated by medical experts.

[0063] This embodiment introduces three quantitative evaluation indicators specifically used to measure the spatial consistency of the model in focusing on the actual lesion area: ①Attention quality ratio within ROI (AttnMassInROI): This refers to the proportion of the sum of normalized attention weights within the ROI region manually labeled by experts to the total attention weights of the entire field of view image. The higher this value, the more the model can focus its decision-making efforts on the actual lesion area and effectively avoid background artifact interference.

[0064] ② Peak In ROI Accuracy: This refers to the probability that the instance with the highest attention weight in the full-view image (i.e., the strongest core gaze point) falls exactly inside the expert-annotated ROI; this metric can most directly reflect whether the model has "fast learning" behavior caused by artifacts.

[0065] ③ TopKHit: The proportion of multiple local grid block instances with the top K highest attention weights in the whole image that hit the expert ROI (K is set to 10 in this embodiment).

[0066] To ensure the fairness and representativeness of the evaluation, this embodiment selected several classic cutting-edge algorithms in the fields of multi-instance learning and medical image fusion as baseline models for comparison, including the ABMIL algorithm, CLAM algorithm, TransMIL algorithm, and a multimodal network (Late Fusion) with a late fusion architecture. All comparative experiments were performed in the same hardware environment and with independent test set partitioning.

[0067] Reference Figure 3 As shown, this is a schematic diagram comparing the spatial attention response heatmap and ROI of the method of the present invention with those of the existing ABMIL network and late fusion network on full-view ultrasound images at different pathological stages of liver fibrosis. Table 2 shows the quantitative comparison results of the consistency between attention and ROI of different methods in the fibrosis detection task (Task 1) and the severity stratification task (Task 2). Table 2. Quantitative evaluation of the consistency between attention and region of interest (ROI) on the independent test set.

[0068] As shown in Table 2, the quantitative evaluation results of the independent test set demonstrate that the method of this invention has achieved significant technical breakthroughs in the two-stage diagnostic task of schistosomiasis liver fibrosis. Firstly, regarding PeakInROI, a core metric measuring the accuracy of the strongest decision fixation point, traditional ABMIL, CLAM, and Late Fusion networks, lacking explicit constraints on spatial background artifacts, result in a large number of the strongest attention weights in the full-field ultrasound image being incorrectly mapped to near-field acoustic artifacts, leading to localization accuracies hovering in the extremely low range of 0.17 to 0.21. In contrast, the architecture of this invention, in Task 1 and Task 2, powerfully boosts the PeakInROI metric to 0.512 and 0.530 respectively, achieving a spatial localization leap more than twice that of the original baseline model, correcting the "blind fixation" or "blind looking" phenomenon commonly found in traditional multi-instance learning frameworks due to rapid learning. Meanwhile, in terms of the comprehensive convergence of the global high-response decision region, the method of this invention demonstrates superior control over global gaze constraints. Its TopKHit index in the two-stage task reached 0.559 and 0.548 respectively, significantly outperforming all traditional comparative algorithms. For example, compared to the current state-of-the-art single-modal TransMIL model, it achieved absolute increases of 0.165 and 0.126 respectively. Furthermore, the AttnMassInROI index, reflecting the overall lesion focusing quality, maintained the highest mean level and extremely low variance under various complex full-image background noise interferences. These detailed quantitative results profoundly reveal that, in weakly supervised clinical applications without any reliance on manual annotation during model inference, it is precisely the multi-domain synergistic optimization operation of the spatial attention prior loss and background negative inhibition loss creatively combined in this invention that successfully suppressed the abnormal gradients of hard negative samples in non-ROI regions. This forced the neural network to lock more than 55% of its high-response core decision regions firmly within the substantial fibrotic lesions defined by clinical medical experts. This conclusion not only strongly supports the significant inventiveness and substantial technological progress of this invention compared to existing technologies from the perspective of objective experimental data, but also greatly enhances the multimodal interpretability and clinical decision-making credibility of deep learning models in the application of intelligent screening for schistosomiasis liver fibrosis at the grassroots level.

[0069] This invention designs an expert ROI-guided spatial attention prior constraint mechanism (AP Loss), which strongly guides the model to converge effective decision information by establishing a mapping error penalty between the model's adaptive spatial attention distribution and medical priors, making the internal black-box mechanism transparent and greatly improving spatial interpretability. It also proposes a background negative suppression mechanism (NS Loss), which dynamically filters and severely penalizes hard negative samples in non-ROI regions during iteration, cutting off the "quick learning" path caused by near-field artifacts and irrelevant noise at the pixel and decision layers, giving the model extremely high robustness and cross-center generalization ability. Finally, it proposes a multimodal feature encoding and splicing fusion strategy and a two-stage sequential task integration architecture, which effectively realizes the complementary fusion of deep semantic features of heterogeneous multimodal data and perfectly fits the actual workflow from ordinary screening to severe stratification. The method of this invention has achieved a significant performance breakthrough on an independent test set containing 10 epidemic counties, with an AUC of 0.976 for fibrosis detection and an AUC of 0.969 for severity stratification. Moreover, its spatial localization consistency is significantly superior to traditional weakly supervised methods. Compared with existing single-modal or ordinary multi-instance networks, the RAIM-MIL architecture constructed in this invention significantly enhances the reliability of model decision-making and clinical trust while ensuring diagnostic accuracy.

[0070] To further verify the independent contributions of each core spatial constraint component (including spatial attention prior loss and background negative suppression loss) in the joint multivariate loss function proposed in this invention to the model's staging performance and spatial interpretability, this embodiment conducted systematic ablation experiments on different network variants on an independent test set.

[0071] In the experiments, by removing or retaining specific loss constraint modules, we constructed a baseline model containing only the main task classification, a variant model introducing only background negative suppression, a variant model introducing only spatial attention prior, and the complete architecture of this invention integrating both mechanisms. The multi-index quantitative ablation experimental results under the two-stage task are shown in Table 3. Table 3 Comparison of ablation experiments of the proposed model on independent test sets

[0072] A continuous, in-depth logical analysis of the quantitative results of the ablation experiments presented in Table 3 reveals that the two spatial constraint modules designed in this invention exhibit a significant synergistic optimization effect. Firstly, when examining the baseline model lacking any spatial supervision constraints, although the classification AUC remains high due to fitting all high-frequency features, including artifacts, of the entire image, its PeakInROI index, which measures the accuracy of the strongest decision point, is as low as 0.216 and 0.265 in the two-stage tasks, respectively. This indicates that the model exhibits severe rapid learning behavior, overly relying on irrelevant acoustic artifacts or abdominal wall noise in the near field of ultrasound images for classification decisions. Clinically, this is highly prone to false positives and misdiagnosis, lacking basic spatial interpretability.

[0073] Meanwhile, when background negative suppression loss (NS only) is introduced into the baseline model alone, the improvement in spatial localization is relatively limited because gradient penalty is applied only to hard negative samples in the background and there is a lack of positive lesion gaze guidance. The PeakInROI of the two stages is only slightly adjusted to 0.239 and 0.268.

[0074] However, when an expert-guided region of interest spatial attention prior loss (APonly) is applied independently in the network, the direction of feature extraction by the model is strongly corrected due to the explicit construction of a strong spatial gaze constraint. This results in a dramatic increase in PeakInROI localization accuracy in Task 1 and Task 2, reaching 0.506 and 0.522 respectively. This strongly demonstrates that by calculating the cross-entropy constraint between the model's adaptive instance weight distribution and the label-smooth target prior distribution, the model can be precisely guided to shift its decision focus from irrelevant background to the solid liver tissue. Finally, when the complete architecture of this invention is constructed by simultaneously combining the spatial attention prior loss and the background negative inhibition loss (Ours), the model not only successfully inherits its extremely high spatial localization capability, further converging and steadily increasing the PeakInROI of Task 1 and Task 2 to the global peak values ​​of 0.512 and 0.530, but also maintains extremely high classification AUCs (reaching 0.976 and 0.969 respectively) compared to individual variants, with highly overlapping confidence intervals (95% CI).

[0075] The aforementioned series of consecutive ablation comparison results profoundly reveal that simply constraining the pixel classification domain cannot ensure the rationality of the spatial feature distribution of the weakly supervised multi-instance learning model. In contrast, the method proposed in this invention... and The multi-dimensional collaborative operation constructs a precise two-way dynamic balance mechanism between positive region focusing and negative artifact penalty, achieving decoupling and collaborative dual optimization of global classification accuracy and local spatial positioning consistency without sacrificing classification accuracy.

[0076] Based on the above ablation experiments, this embodiment performs computational graph optimization and model lightweighting on the RAIM-MIL model. The optimized intelligent diagnostic model maintains ultra-high diagnostic accuracy with AUC>0.96, making it very suitable for solidification as a core diagnostic algorithm or seamless embedding into primary mobile medical terminals, portable community ultrasound screening instruments, or remote cloud-assisted diagnosis and treatment systems.

[0077] This embodiment utilizes the liver ultrasound thermal map generation method based on multimodal learning provided by the present invention. It extracts ultrasound features by introducing a weakly supervised multi-instance learning network (RAIM-MIL) guided by region of interest (ROI), and employs a multilayer perceptron fusion network for clinical indicators to encode clinical structured variables in a dimensionally compatible manner. During network training, a joint multivariate loss function incorporating attention prior supervision and negative background inhibition is used to optimize the internal feature space distribution and penalize rapid learning. Applying this method, the system can quickly intercept the normal phase (F0) group with extremely high specificity by inputting full-field ultrasound images and corresponding conventional serological indicators into the system without requiring doctors to manually define ROIs or altering existing ultrasound physician clinical workflows. Furthermore, it can accurately stratify ultrasound images exhibiting fibrosis into three stages based on severity, significantly improving the efficiency and accuracy of large-scale ultrasound image screening.

[0078] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for generating liver ultrasound thermograms based on multimodal learning, characterized in that, include: Obtain a training sample set, in which each sample includes a sample-level category label for the same subject, at least one liver ultrasound image, and structured clinical indicators and region of interest annotation information corresponding to each liver ultrasound image; Each sample is input into a multimodal, multi-instance learning model to obtain the binary classification predicted label for each sample, including: The liver ultrasound image is divided into multiple instance image blocks. The instance features of each instance image block are extracted and input into the gated attention multi-instance pooling module to obtain the attention weights of each instance image block. The instance image blocks are then weighted and aggregated to obtain bag-level image features. Structured clinical indicators are input into the clinical feature encoding module to obtain clinical embedded features; The bag-level image features are concatenated with the clinical embedding features to obtain the cascaded fused features; The cascaded fused features are input into the sample-level classifier to obtain the predicted probability of the binary classification label; Based on the predicted probability of binary classification labels and the Focal Loss of sample-level class labels, a main classification loss is constructed; Based on the spatial positional relationship between each instance image patch and the region of interest (ROI) annotation information, the instance-level ROI labels of each instance image patch are obtained, normalized and smoothed, and the instance-level ROI prior distribution is obtained. Based on the instance-level ROI prior distribution and the attention weights of each instance image patch, the cross-entropy is calculated, and the spatial attention prior loss is constructed. From instance image patches whose instance-level ROI labels are non-ROI labels, select a preset proportion of instance image patches as hard negative instance image patches in descending order of attention weight; calculate the average of the attention weights of all hard negative instance image patches, and combine it with a preset interval threshold to construct the background negative suppression loss; The main classification loss, spatial attention prior loss, and background negative inhibition loss are weighted and summed to construct a joint multivariate loss function. This function is then used to train the multimodal multi-instance learning model, resulting in a well-trained multimodal multi-instance learning model. The liver ultrasound image to be detected and its corresponding structured clinical indicators are input into a trained multimodal multi-instance learning model to obtain the attention weights of each instance image patch in the liver ultrasound image to be detected, and then the liver ultrasound heat map is generated.

2. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 1, characterized in that, The liver ultrasound image is divided into multiple instance image patches. Instance features of each instance image patch are extracted and input into a gated attention multi-instance pooling module to obtain the attention weights of each instance image patch. Weighted aggregation of the instance image patches is then performed to obtain bag-level image features, including: For one or more liver ultrasound images in any sample, the effective fan-shaped imaging region of the liver ultrasound image is extracted, and overlapping sliding sampling is performed using a preset sliding window based on a preset sliding step size to obtain multiple candidate image blocks. The candidate image blocks are selected as instance image blocks, with a proportion greater than a preset proportion value in the effective fan-shaped imaging region and an average gray value greater than a preset gray value. Instance features of each instance image patch are extracted using the ResNet50 backbone network; The features of each instance corresponding to one or more liver ultrasound images belonging to the same sample are input into the gated attention multi-instance pooling module to obtain the attention score of each instance image block. The attention score is then normalized among all instance image blocks in the same sample to obtain the attention weight of each instance image block. The bag-level image features of a sample are obtained by weighting and summing the features of each instance based on the attention weights of each instance image patch.

3. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 2, characterized in that, The features of each instance corresponding to one or more liver ultrasound images belonging to the same sample are input into a gated attention multi-instance pooling module to obtain the attention score of each instance image patch. The attention scores are then normalized across all instance image patches in the same sample to obtain the attention weight of each instance image patch, including: The first Instance features of an instance image patch Linear transformations and activations are performed on the hyperbolic tangent-gated branch and the sigmoid-gated branch respectively. The outputs of the branches are then multiplied element-wise, and the linear transformation is used to obtain the nth... Attention score for each instance image patch , represented as: ； Among all instance image patches in the same sample, the attention score of each instance image patch is normalized using Softmax to obtain the first... Attention weights for each instance image patch , represented as: ； in, , Indicates the sequence number of the instance image patch. , , This indicates the total number of instance image blocks containing one or more liver ultrasound images within the same sample; This represents the hyperbolic tangent activation function. and Let represent the learnable linear transformation matrix and the learnable bias column vector in the hyperbolic tangent gating branch, respectively; This represents the Sigmoid activation function. and These represent the learnable linear transformation matrix and the learnable bias column vector in the Sigmoid gated branch, respectively. and These represent the learnable weight column vector and the learnable bias scalar, respectively, for mapping attention scores. Represents element-wise multiplication. Indicates transpose; .

4. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 1, characterized in that, The structured clinical indicators corresponding to the liver ultrasound images are input into the clinical feature encoding module to obtain clinical embedded features, including: Based on the structured clinical indicators of each sample in the training sample set, the mean and standard deviation of each indicator are obtained. Z-score standardization is performed on each indicator in the structured clinical indicators corresponding to the liver ultrasound image to obtain the standardized structured clinical indicators. Standardized structured clinical indicators The input to the clinical feature encoding module passes through a fully connected mapping layer and a normalization layer in sequence. ReLU activation layer and random deactivation layer Obtain clinical embedding features , represented as: ； in, and This represents the learnable weight matrix and learnable bias column vector of the fully connected mapping layer; structured clinical indicators include: age, albumin (ALB), laminin (LN), hemoglobin (HGB), hyaluronic acid (HA), - Gamma-glutamyl transferase (GGT), low-density lipoprotein cholesterol (LDL-C), and liver fibrosis index (FIB-4).

5. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 1, characterized in that, Based on the predicted probabilities of binary classification labels and the Focal Loss of sample-level class labels, the main classification loss is constructed, expressed as: ； in, Indicates the primary classification loss. Indicates binary category label The predicted probability; Indicates sample-level category labels. ; It represents the natural logarithm.

6. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 1, characterized in that, Constructing the spatial attention prior loss includes: Get the Instance-level ROI labels for each instance image patch , , Indicates the first Each instance image patch is located within the ROI region. Indicates the first Each instance image patch is located in a non-ROI region; Count the number of instance image patches located within the ROI region in the same sample. and in At that time, the instance-level ROI labels of each instance image patch are normalized and smoothed to obtain the first instance. Instance-level ROI prior probabilities corresponding to each instance image patch , represented as: ; Based on instance-level ROI prior probability Attention weights relative to the corresponding instance image patch Constructing spatial attention prior loss , represented as: ； in, , This indicates the total number of instance image patches contained in one or more liver ultrasound images within the same sample. 1; This indicates the preset smoothing coefficient. It is a very small positive number.

7. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 6, characterized in that, Calculate the average attention weights of all hard negative instance image patches, and combine them with a preset interval threshold to construct a background negative suppression loss. , represented as: ； When there are no instance image patches located outside the ROI region, the hard negative instance set is empty, and the background negative suppression loss is applied. =0; in, The set of indices representing hard negative instance image patches is denoted as: ; express Total number of indexes in , This represents the set of instance image patches with an instance-level ROI label of 0. This indicates the preset selection ratio of the hard negative instance image patch. This indicates the rounding operation. Indicates before selection One element; This indicates the preset interval threshold.

8. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 1, characterized in that, The attention weights of each instance image patch in the ultrasound image of the liver to be detected are obtained and mapped to generate a liver ultrasound thermogram, including: Obtain the spatial coordinates and attention weights of each instance image patch in the ultrasound image of the liver to be detected; Construct an original thermal intensity matrix with the same height and width as the ultrasound image of the liver to be detected, and initialize each pixel value in the original thermal intensity matrix to 0; Based on the spatial coordinates of each instance image patch, the attention weights of each instance image patch are accumulated to the corresponding pixel positions in the original thermal intensity matrix to obtain the weighted thermal intensity matrix. The weighted thermal intensity matrix is ​​normalized to its maximum value to obtain a normalized thermal intensity matrix with values ​​ranging from 0 to 1. The normalized thermal intensity matrix is ​​converted into a color thermal layer, and the color thermal layer is overlaid onto the ultrasound image of the liver to be examined with a preset transparency to obtain a liver ultrasound thermal map.

9. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 8, characterized in that, Obtain the weighted thermal intensity matrix, including: Using the sub-models of five multimodal multi-instance learning models trained based on five-fold hierarchical cross-validation, the first-order sub-models are obtained respectively. The five attention weights of each instance image patch are averaged to obtain the first attention weight. Integrated attention weights for each instance image patch; Based on the Integrated attention weights for each instance image patch The pixel position in the ultrasound image of the liver to be tested is obtained by determining its spatial coordinates. Weighted thermal intensity at the point , represented as: ； A weighted thermal intensity matrix is ​​formed based on the weighted thermal intensity at each pixel location in the ultrasound image of the liver to be detected; in, Indicates the first Each instance image patch corresponds to a pixel region in the ultrasound image of the liver to be detected. Indicates an indicative function, pixel position. Located in the pixel area The value is 1 if the time condition is met, and 0 otherwise. This indicates the number of instance image patches belonging to the ultrasound image of the liver to be examined. This indicates the image group number of the ultrasound image of the liver to be examined.

10. The method for generating liver ultrasound thermograms based on multimodal learning according to claim 8, characterized in that, The normalized thermal intensity matrix is ​​converted into a color thermal layer, and this color thermal layer is overlaid onto the ultrasound image of the liver to be examined with a preset transparency, resulting in a liver ultrasound thermal map, including: The element value is Each element in the normalized thermodynamic intensity matrix is ​​linearly mapped to Obtain the grayscale interval matrix; The grayscale matrix is ​​converted into a color heatmap layer using the JET color mapping method, and then overlaid according to preset weights. The color thermal layer is weighted and superimposed on the ultrasound image of the liver to be examined to obtain the liver ultrasound thermal map, which is represented as follows: ； in, Indicates the pixel position in the liver ultrasound thermogram. Pixel value at that location, Indicates the pixel position in the ultrasound image of the liver to be examined. Pixel value at that location, Indicates the pixel position in the color heatmap layer The pixel value at that location.

Images