Liver nodule detection box and benign and malignant probability prediction method and device based on ct and mri multi-modal images

Abstract

The application discloses a liver nodule detection frame and benign and malignant probability prediction method and device based on CT and MRI multi-modal images, and relates to the technical field of medical image processing, and aims to improve the accuracy of liver nodule detection frame and benign and malignant probability prediction.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a method and apparatus for detecting liver nodules and predicting the probability of benign or malignant nodules based on CT and MRI multimodal images. Background Technology

[0002] In liver imaging, phase is a crucial carrier of diagnostic information: the enhancement / washout appearance of the same nodule in different phases often determines the diagnostic tendency. Clinically and academically, systems such as LI-RADS are commonly used to standardize and stratify the technical aspects, phase acquisition, and imaging characteristics of CT / MRI examinations. Therefore, in the design of artificial intelligence models, multi-phase images are not simply "multiple similar pictures," but rather complementary evidence carrying different physiological / pathological meanings. Furthermore, multi-center, multi-device, and multi-protocol approaches can lead to differences in phase quality and consistency, thus affecting the robustness of the algorithm.

[0003] With the development of machine learning, deep neural networks are gradually integrating more information, from single-modality to multi-phase, and then to multimodal fusion. There are already studies on using multi-phase CT deep learning for liver lesion classification / diagnosis. However, current comprehensive analysis shows that the real engineering challenge for multimodal (CT+MRI) implementation lies in the fact that, compared to "single-modality multi-phase," "CT+MRI multimodal" suffers from even stronger inconsistencies. These inconsistencies stem from cross-modal appearance differences (different grayscale / contrast mechanisms); spatial inconsistencies caused by differences in acquisition time / position / respiration; and the common occurrence of modality / phase loss in real-world data (e.g., some cases lack a certain phase or sequence). This makes simple "semblage input / average fusion" difficult to operate stably. Summary of the Invention

[0004] In view of this, the main objective of the embodiments of the present invention is to provide a method and apparatus for predicting the detection frame and benign / malignant probability of liver nodules based on CT and MRI multimodal images, in order to solve at least one of the problems of the prior art. The present invention can improve the accuracy of the detection frame and the prediction of benign / malignant probability of liver nodules.

[0005] To achieve the above objectives, one aspect of the present invention provides a method for predicting the detection bounding box and benign / malignant probability of liver nodules based on CT and MRI multimodal images, the method comprising: Obtain the original training set and preprocess the original training set to obtain the preprocessed training set; the original training set includes the original single-group samples, and the preprocessed training set includes the preprocessed single-group samples; each single-group sample includes a first slice image of three phases under CT modality and a second slice image of four phases under MRI modality. For each preprocessed single-group sample, a random single-period phase missing construct is performed to generate training sample pairs; Based on the training sample pairs, the initial conditional diffusion model is trained to obtain the target conditional diffusion model; Using the target conditional diffusion model, a multi-missing mode image generation operation is performed on the original training set to obtain an augmented training set; By using a target multimodal multi-phase hierarchical fusion network, the augmented training set is independently encoded, intramodal feature aggregation is performed, cross-modal feature alignment and fusion is performed, multi-scale fusion is performed, and feature decoding is performed to obtain the prediction results of liver nodule detection boxes and the prediction results of benign and malignant probabilities.

[0006] In some embodiments, obtaining the original training set and preprocessing the original training set to obtain the preprocessed training set includes the following steps: Spatial normalization is performed on the first slice image and the second slice image of each of the single samples to obtain the first aligned image and the second aligned image. Intramodal grayscale normalization is performed on the first aligned image and the second aligned image respectively to obtain the first grayscale image and the second grayscale image. The liver nodule detection box is mapped to the coordinate system of the first grayscale image and the coordinate system of the second grayscale image to generate a lesion mask. Based on each of the aforementioned phases, a standardized text prompt vocabulary library is constructed; The preprocessed training set is obtained based on the original training set, the lesion mask, and the standardized text prompt vocabulary.

[0007] In some embodiments, the step of constructing a random single-phase missing data structure for each preprocessed single-group sample to generate training sample pairs includes the following steps: Obtain the preprocessed slice image from the single group of samples after preprocessing; Random sampling is performed on all phases to determine the missing and non-missing phases; The preprocessed slice image corresponding to the non-missing period is subjected to intramodal feature stitching and cross-modal feature stitching to obtain the first image conditional feature map; Based on a standardized text prompt vocabulary, extract the feature vector of the first text prompt word corresponding to the missing period; The first image conditional feature map and the first text prompt word feature vector are fused to obtain a joint conditional input. The preprocessed slice image corresponding to the missing period is used as the initial label; The lesion mask in the preprocessed training set is incorporated into the initial label to obtain the generated label; The training sample pairs are generated based on the joint conditional input and the generated labels.

[0008] In some embodiments, training an initial conditional diffusion model based on the training sample pairs to obtain a target conditional diffusion model includes the following steps: Stepwise Gaussian noise is added to the generated labels in the training sample pairs to obtain noisy images with different noise step sizes; The intermediate feature vector of the noise step is concatenated with the joint conditional input in the training sample pair to obtain the input features of the target denoising diffusion backbone network. A modality-phase attention layer is added to the initial encoder and initial decoder of the initial denoising and diffusion backbone network, and a lesion region constraint layer is added to the initial decoder to obtain the target denoising and diffusion backbone network. Based on the target denoising diffusion backbone network, the denoising diffusion probability model is improved to obtain the initial conditional diffusion model; Obtain the denoised prediction image under the noise step size, and construct a first loss function based on the noisy image and the denoised prediction image; The initial conditional diffusion model is trained based on the input features and the first loss function to obtain the target conditional diffusion model.

[0009] In some embodiments, the step of performing a multi-missing mode image generation operation on the original training set using the target conditional diffusion model to obtain an augmented training set includes the following steps: The target conditional diffusion model is used to process the original training set by removing a single phase, resulting in the first enhanced sample. By applying the target conditional diffusion model to the original training set through multiple phase loss processing, a second enhanced sample is obtained. The first augmented sample, the second augmented sample, and the original single-group sample are merged to obtain the augmented training set.

[0010] In some embodiments, the step of using a target multimodal multi-phase hierarchical fusion network to independently encode, aggregate intra-modal features, align and fuse cross-modal features, fuse at multiple scales, and decode features on the augmented training set to obtain liver nodule detection box prediction results and benign / malignant probability prediction results includes the following steps: The first slice image and the second slice image in the augmented training set are independently encoded to obtain a first feature map group in the CT modality and a second feature map group in the MRI modality; the first feature map group includes 3 first independent features and the second feature map group includes 4 second independent features. Intramodal feature aggregation is performed on the first feature map group and the second feature map group respectively to obtain intramodal aggregated features; Cross-modal feature alignment and fusion are performed on the first feature map group and the second feature map group to obtain cross-modal fused features; Multi-scale fusion is performed on the first independent feature, the second independent feature, the intra-modal aggregated feature, and the cross-modal fusion feature to generate a globally shared feature map; After decoding the global shared feature map, it is input into the dual-task backbone head, and the predicted results of liver nodule detection boxes and the predicted probabilities of benign and malignant nodules are output.

[0011] In some embodiments, performing intra-modal feature aggregation on the first feature map group and the second feature map group respectively to obtain intra-modal aggregated features includes the following steps: The first feature map group is input into the phase attention aggregation layer of the CT modality, and the three first independent features are weighted and fused to obtain the first aggregated feature map of the CT modality. The second feature map group is input into the phase attention aggregation layer of the MRI modality, and the four second independent features are weighted and fused to obtain the second aggregated feature map of the MRI modality. The intramodal aggregation features include the first aggregation feature map and the second aggregation feature map.

[0012] In some embodiments, the cross-modal feature alignment and fusion of the first feature map group and the second feature map group to obtain cross-modal fused features includes the following steps: Spatial feature registration is performed on the first feature map group and the second feature map group to obtain the first aligned feature map group and the second aligned feature map group; A dual-path attention fusion mechanism is used to perform cross-modal semantic fusion on the first aligned feature map group and the second aligned feature map group to obtain the cross-modal fused features.

[0013] In some embodiments, the step of decoding the globally shared feature map and inputting it into the dual-task backbone head to output the liver nodule detection box prediction result and the benign / malignant probability prediction result includes the following steps: The decoded global shared feature map is subjected to convolutional dimensionality reduction to extract the detection box prediction task features; The decoded global shared feature map is subjected to convolutional dimensionality reduction and global average pooling to extract probabilistic prediction task features; Input the detection box prediction task features into the detection box prediction branch, and output the detection box coordinates and the detection box confidence score; The lesion region features are obtained by cropping the detection box feature region from the predicted task features of the detection box. The probability prediction task features and the lesion area features are input into the benign / malignant probability prediction branch, and the benign / malignant prediction probability value is output.

[0014] To achieve the above objectives, another aspect of the present invention proposes a liver nodule detection frame and benign / malignant probability prediction device based on CT and MRI multimodal images, the device comprising: The first module is used to acquire the original training set and preprocess the original training set to obtain the preprocessed training set; the original training set includes the original single-group samples, and the preprocessed training set includes the preprocessed single-group samples; each single-group sample includes a first slice image of three phases in the CT modality and a second slice image of four phases in the MRI modality. The second module is used to construct random single-period phase missing for each preprocessed single group of samples to generate training sample pairs. The third module is used to train the initial conditional diffusion model based on the training sample pairs to obtain the target conditional diffusion model. The fourth module is used to perform multi-missing mode image generation operation on the original training set through the target conditional diffusion model to obtain an augmented training set. The fifth module is used to perform independent encoding, intra-modal feature aggregation, cross-modal feature alignment and fusion, multi-scale fusion and feature decoding on the augmented training set through a target multimodal multi-phase hierarchical fusion network to obtain the liver nodule detection box prediction results and benign / malignant probability prediction results.

[0015] To achieve the above objectives, another aspect of the present invention provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described above.

[0016] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.

[0017] To achieve the above objectives, another aspect of the present invention provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the aforementioned method.

[0018] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a method and apparatus for predicting liver nodule detection bounding boxes and benign / malignant probabilities based on CT and MRI multimodal images. This scheme obtains an original training set and preprocesses it to obtain a preprocessed training set, ensuring consistency between subsequent generation and training; it constructs random single-phase deletions for each preprocessed single-group sample to generate training sample pairs; it trains an initial conditional diffusion model based on the training sample pairs to obtain a target conditional diffusion model; it performs multi-deletion mode image generation operations on the original training set through the target conditional diffusion model to obtain an augmented training set, improving the robustness of the model to multi-phase deletions; through a target multimodal multi-phase hierarchical fusion network, it performs independent encoding, intramodal feature aggregation, cross-modal feature alignment and fusion, multi-scale fusion, and feature decoding on the augmented training set, realizing mutual promotion of detection and diagnosis tasks, solving the problems of inconsistent multimodal multi-phase image features and cumulative errors in serial tasks, and obtaining liver nodule detection bounding box prediction results and benign / malignant probability prediction results, thereby improving the accuracy of liver nodule detection bounding boxes and benign / malignant probability prediction. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the liver nodule detection frame and benign / malignant probability prediction method based on CT and MRI multimodal images provided in this embodiment of the invention; Figure 2 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0022] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification, claims, and the foregoing drawings may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to a determination," or "in the event of a determination."

[0023] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0025] Before providing a detailed description of the embodiments of the present invention, some of the nouns and terms involved in the embodiments of the present invention will be explained first. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0026] Liver nodules: refer to lesions that appear as focal space-occupying lesions or abnormal signal / density changes in liver imaging, including but not limited to benign and malignant nodules.

[0027] Multimodal imaging data / multiphase imaging data: refers to a collection of image data from different imaging modalities (such as CT and MRI) and / or different time phases (enhanced phase) within the same modality. In this invention, multimodal imaging includes CT and MRI, and multiphase imaging includes CT three-phase and MRI four-phase.

[0028] Three-phase CT imaging: refers to CT images of the liver of the same subject acquired in different phases of contrast-enhanced CT scans, including at least the CT arterial phase, CT portal venous phase, and CT delayed phase (or CT equilibrium phase). It should be noted that different institutions may have different names for the delayed phase / equilibrium phase.

[0029] MRI four-phase imaging: refers to MRI images of the liver of the same subject acquired in different phases of enhanced MRI examination, including at least the late arterial phase, portal venous phase, transitional phase, and hepatobiliary phase.

[0030] Single sample: refers to a set of paired data from a single subject. In this invention, each sample contains one two-dimensional slice image of each of the seven phase modalities (CT three-phase image + MRI four-phase image), and includes corresponding annotation information.

[0031] This invention provides a method and device for predicting the benign and malignant probabilities of liver nodules based on CT and MRI multimodal images. This approach addresses seven modalities of liver nodule image data, comprising three phases (arterial phase, portal venous phase, and delayed / equilibrium phase) in CT and four phases (late arterial phase, portal venous phase, transitional phase, and hepatobiliary phase) in MRI. Data augmentation solves problems such as insufficient multimodal image sample size, missing phases / modalities, and cross-center distribution bias. A customized multi-task network structure enables precise localization of liver nodules (outputting target bounding boxes, BBoxes) and estimation of benign and malignant probabilities across the seven modalities. The augmented images are directly used for training the multi-task network, with cross-modal spatial consistency and phase feature complementarity constraints applied during training. This ultimately achieves the output of BBoxes for each of the seven images in a single sample, along with benign and malignant probability estimations, significantly improving the robustness and accuracy of the model in real-world clinical multimodal and multi-phase imaging scenarios.

[0032] Figure 1 This is an optional flowchart of a liver nodule detection frame and benign / malignant probability prediction method based on CT and MRI multimodal images provided in an embodiment of the present invention. Figure 1 The method may include, but is not limited to, steps S100 to S500: Step S100: Obtain the original training set and preprocess the original training set to obtain the preprocessed training set; the original training set includes the original single-group samples, and the preprocessed training set includes the preprocessed single-group samples; each single-group sample includes the first slice image of 3 phases under CT modality and the second slice image of 4 phases under MRI modality. Step S200: For each preprocessed single-group sample, a random single-period phase missing construction is performed to generate training sample pairs; Step S300: Train the initial conditional diffusion model based on the training sample pairs to obtain the target conditional diffusion model; Step S400: Using the target conditional diffusion model, perform multi-missing mode image generation operation on the original training set to obtain the augmented training set. Step S500: Through the target multimodal multi-phase hierarchical fusion network, the augmented training set is independently encoded, intramodal feature aggregation is performed, cross-modal feature alignment and fusion is performed, multi-scale fusion is performed, and feature decoding is performed to obtain the liver nodule detection box prediction results and benign / malignant probability prediction results.

[0033] In steps S100 to S400 of some embodiments, a multimodal, multi-phase image data augmentation method driven by a conditional diffusion model is applied. By training a modality-phase-aware conditional diffusion generation model, the method can accurately complete and generate any phase image that is randomly missing in a single sample, thereby completing the large-scale augmentation of the training set. The generated augmented sample has the same format as the original sample (one two-dimensional slice image of each of the seven modalities in a single sample), and retains the spatial structure and pathological features of liver nodule lesions. The original target detection bounding box and benign / malignant annotations can be directly reused (after spatial consistency verification).

[0034] In step S100 of some embodiments, each single sample in the original training set (including the first slice image of 3 phases under CT modality and the second slice image of 4 phases under MRI modality, a total of 7 two-dimensional original slice images, liver nodule BBox annotation, benign and malignant annotation) is uniformly preprocessed to ensure the consistency between subsequent generation and training.

[0035] In some embodiments, step S100 may include, but is not limited to, steps S110 to S150: Step S110: Perform spatial normalization on the first slice image and the second slice image of each single sample to obtain the first aligned image and the second aligned image. Step S120: Perform intramodal grayscale normalization on the first aligned image and the second aligned image respectively to obtain the first grayscale image and the second grayscale image. Step S130: Map the liver nodule detection box to the coordinate system of the first grayscale image and the coordinate system of the second grayscale image to generate a lesion mask. Step S140: Construct a standardized text prompt vocabulary for each phase; Step S150: Based on the original training set, lesion mask, and standardized text prompt vocabulary, a preprocessed training set is obtained.

[0036] In step S110 of some embodiments, the first slice image and the second slice image of each single sample are spatially normalized. For example, for each single sample, the seven two-dimensional original slice images are aligned at the same level (based on the registration of liver anatomical feature points) and uniformly cropped to a fixed size (such as 512×512) to eliminate the spatial offset caused by the acquisition position, breathing and equipment, so as to obtain the first aligned image of three phases in the CT modality and the second aligned image of four phases in the MRI modality.

[0037] In step S120 of some embodiments, the first aligned image and the second aligned image are respectively subjected to intramodal gray-level normalization. For example, for each single sample, the CT modality and MRI modality are subjected to intramodal gray-level stretching and standardization (CT uses HU value window width and window level adjustment, and MRI uses quantile normalization). While preserving the intramodal gray-level features, the generation interference caused by gray-level differences between modalities is avoided, and the first gray-level image of three phases in the CT modality and the second gray-level image of four phases in the MRI modality are obtained.

[0038] In step S130 of some embodiments, the liver nodule detection box (BBox) is mapped to the coordinate system of the first grayscale image and the coordinate system of the second grayscale image to generate a lesion mask, which is used for lesion region constraints in the subsequent model generation.

[0039] In step S140 of some embodiments, a standardized text cue word library is constructed for the seven phase modalities, uniquely identifying each phase. For example, the format of the text cue words in the standardized text cue word library is [modality type]-[phase number], specifically: CT-1 (arterial phase), CT-2 (portal venous phase), CT-3 (delayed / equilibrium phase), MRI-1 (late arterial phase), MRI-2 (portal venous phase), MRI-3 (transitional phase), and MRI-4 (hepatobiliary phase). After passing through an embedding layer, the text cue words can be converted into fixed-dimensional text feature vectors, which serve as the conditional input for the subsequent conditional diffusion model.

[0040] In step S150 of some embodiments, to allow for a more intuitive view of the annotation effect, the lesion mask can be overlaid back onto the original slice image of the original training set in a semi-transparent color or other form to generate a slice image with a mask. Based on the slice image with the mask, the lesion mask, and the standardized text prompt dictionary, a preprocessed training set can be generated.

[0041] In some embodiments, step S200 may include, but is not limited to, steps S210 to S280: Step S210: Obtain the preprocessed slice image from a single group of preprocessed samples; Step S220: Randomly sample all phases to determine the missing phases and the non-missing phases; Step S230: Perform intra-modal feature stitching and cross-modal feature stitching on the preprocessed slice images corresponding to the non-missing period to obtain the first image conditional feature map; Step S240: Extract the feature vector of the first text prompt word corresponding to the missing period based on the standardized text prompt word library; Step S250: The first image conditional feature map and the first text prompt word feature vector are fused to obtain the joint conditional input; Step S260: Use the preprocessed slice image corresponding to the missing period as the initial label; Step S270: Incorporate the lesion mask from the preprocessed training set into the initial label to obtain the generated label; Step S280: Generate training sample pairs based on the joint condition input and generated labels.

[0042] In step S210 of some embodiments, seven preprocessed slice images can be obtained for each preprocessed single sample.

[0043] In step S220 of some embodiments, random sampling is performed on the seven phase modes of a single sample to determine one missing phase (denoted as...). , (corresponding to 7 phases respectively), the remaining 6 non-missing phases are denoted as .

[0044] In step S230 of some embodiments, the preprocessed slice images corresponding to the six non-missing phases are subjected to intramodal feature stitching (the preprocessed slice images corresponding to the non-missing phases in the three phases of CT are stitched in the channel dimension, and the preprocessed slice images corresponding to the non-missing phases in the four phases of MRI are stitched in the channel dimension), and then cross-modal feature stitching is performed to obtain a first image conditional feature map with 6 channels. .

[0045] In steps S240 to S250 of some embodiments, the missing period phase is extracted. Corresponding text prompt word feature vector , and the first image condition feature map Feature fusion is performed to obtain joint conditional input. Specifically, this involves extracting the feature vectors of text prompt words. The image is mapped to the conditional feature map of the first image through a 1×1 convolution. With the same number of channels, feature fusion is achieved by adding them element by element.

[0046] In steps S260 to S270 of some embodiments, the missing period is... Preprocessed slice images As the initial label, it also incorporates a mask for liver nodules. Generate labels with lesion constraints. (⊙ represents element-wise multiplication, and ⊙ represents...) (as weights), ensuring that the generated model focuses on feature restoration of the lesion area.

[0047] In step S280 of some embodiments, training sample pairs are generated by constructing random single-phase missing values ​​for each preprocessed single-group sample. The training sample pairs consist of joint conditional inputs and generated labels.

[0048] In step S300 of some embodiments, the denoising diffusion probability model is improved by introducing a modality-phase attention mechanism and a lesion region constraint module to obtain an initial conditional diffusion model. The initial conditional diffusion model is trained through a forward diffusion process, a reverse denoising process, and loss function design to obtain a target conditional diffusion model. The target conditional diffusion model structure includes a text embedding layer, an image feature encoder, a joint conditional fusion layer, a denoising diffusion backbone network, and an image decoder.

[0049] In some embodiments, step S300 may include, but is not limited to, steps S310 to S360: Step S310: Add progressive Gaussian noise to the generated labels in the training sample pairs to obtain noisy images with different noise step sizes; Step S320: The intermediate feature vector of the noise step is concatenated with the joint conditional input in the training sample pair to obtain the input features of the target denoising diffusion backbone network. Step S330: Add a modality-phase attention layer to the initial encoder and initial decoder of the initial denoising diffusion backbone network, and add a lesion region constraint layer to the initial decoder to obtain the target denoising diffusion backbone network; Step S340: Based on the target denoising diffusion backbone network, the denoising diffusion probability model is improved to obtain the initial conditional diffusion model; Step S350: Obtain the denoised prediction image under the noise step size, and construct the first loss function based on the noisy image and the denoised prediction image. Step S360: Train the initial conditional diffusion model based on the input features and the first loss function to obtain the target conditional diffusion model.

[0050] In steps S310 to S320 of some embodiments, during the forward diffusion process, the generated tag is... By applying progressively increasing Gaussian noise, different noise step sizes were obtained. Noisy images Noise step size eigenvectors and joint conditional inputs The splicing serves as the input feature for the target denoising diffusion backbone network.

[0051] In steps S330 to S340 of some embodiments, during the reverse denoising process, the initial denoising diffusion backbone network uses U-Net+++ as its basic architecture. A modality-phase attention layer is added to each layer of the initial encoder and initial decoder of the U-Net. This modality-phase attention layer utilizes the feature vectors of text prompt words. Attention weighting is applied to image features to enable the model to accurately perceive the phase modality type to be generated; simultaneously, a lesion region constraint layer is added to the initial decoder to mask the liver nodule lesions. As an attention mask, the model is forced to learn higher feature weights in the lesion region than in the background region, resulting in the target denoising diffusion backbone network. Based on the target denoising diffusion backbone network, the denoising diffusion probability model is improved by introducing the modality-phase attention mechanism and the lesion region constraint module into the denoising diffusion probability model, thus obtaining the initial conditional diffusion model.

[0052] In step S350 of some embodiments, in the loss function design, mean squared error loss with lesion weights (FW-MSE) is used as the model training loss. Weights are assigned to the pixel loss in the lesion area. ( ,like The background area is assigned a weight of 1, and the formula is as follows: ,in For the model at the noise step size Denoising prediction image.

[0053] In step S360 of some embodiments, the AdamW optimizer is used to optimize the model based on the input features and the first loss function, and the model is iteratively trained until the first loss function converges, thus obtaining the trained modality-phase-aware target conditional diffusion model. .

[0054] In some embodiments, step S400 may include, but is not limited to, steps S410 to S430: Step S410: Using the target conditional diffusion model, perform single-phase missing processing on the original training set to obtain the first enhanced sample; Step S420: Using the target conditional diffusion model, perform multiple phase loss processing on the original training set to obtain the second enhanced sample; Step S430: The first augmented sample, the second augmented sample, and the original single sample are merged to obtain the augmented training set.

[0055] In step S410 of some embodiments, the original slice images in the original single-group sample are obtained; in the original single-group sample, one original slice image is designated as the missing image, and the remaining original slice images are designated as non-missing images; the non-missing images are subjected to intra-modal feature stitching and cross-modal feature stitching to obtain a second image conditional feature map; according to a standardized text prompt word library, the second text prompt word feature vector corresponding to the non-missing image is extracted; the second image conditional feature map and the second text prompt word feature vector are input into the target conditional diffusion model to obtain the completed image corresponding to each missing image; the completed image is combined with the non-missing image to obtain the first enhanced sample. For example, for each original single-group sample, seven missing modes are executed sequentially (each of the seven phases is missing once), and the 6-channel image conditional feature map and the corresponding text prompt word feature vector corresponding to each missing mode are input into... Seven completed images are generated and combined with six original unmissing images in the corresponding missing mode to form seven sets of enhanced samples (each set still consists of seven complete images in seven modalities, one of which is a generated completed image and the other six are original unmissing images).

[0056] In step S420 of some embodiments, in the original single-group sample, two original slice images are sequentially designated as missing images, and the remaining original slice images are designated as non-missing images; the non-missing images are subjected to intra-modal feature stitching and cross-modal feature stitching to obtain a third image conditional feature map; based on a standardized text prompt word library, the third text prompt word feature vector corresponding to the non-missing images is extracted; the third image conditional feature map and the third text prompt word feature vector are input into the target conditional diffusion model to obtain the completed image corresponding to each missing image; the completed image is combined with the non-missing image to obtain the second enhanced sample. By performing random two-phase deletion on a single sample to form additional enhanced samples, the robustness of the model to multi-phase deletion is further improved.

[0057] In some embodiments, the generated first enhanced sample and second enhanced sample can directly reuse the liver nodule BBox annotation and benign / malignant annotation of the original single sample. The spatial consistency of the lesion space between the generated image and the original image is verified by spatial mutual information verification, and invalid enhanced samples with consistency below a threshold (e.g., 0.9) are removed.

[0058] In step S430 of some embodiments, all valid first augmented samples, valid second augmented samples, and the original single-group samples are merged to obtain an augmented training set. The sample size can be increased from the original N level to the kN level (k≥10, such as 100 cases can be expanded to 1000+ cases), and it covers all scenarios with single-phase missing and some multi-phase missing scenarios, greatly enriching the data distribution.

[0059] In step S500 of some embodiments, a multi-task joint learning method for liver nodule detection and benign / malignant diagnosis is applied. Based on an augmented training set, a multi-modal multi-phase hierarchical fusion network (MMP-HFN) is designed to achieve end-to-end multi-task learning of liver nodules. The core output is the BBox coordinates (including confidence) of the liver nodules corresponding to the seven images of a single sample, and the benign / malignant probability (malignant probability / benign probability) of the corresponding liver nodules. The network adopts a feature learning strategy of intra-modal aggregation-cross-modal alignment-global fusion, and constructs a phase-independent feature encoder, a global feature sharing decoder, a dual-task head for detection boxes and benign / malignant probability prediction, and a phase-modal perception subheading. At the same time, cross-phase consistency constraints and lesion feature sharing constraints are applied to achieve mutual promotion of detection and diagnosis tasks and solve the problems of inconsistent multi-modal and multi-phase image features and error accumulation in serial tasks.

[0060] In some embodiments, a multimodal multi-phase hierarchical fusion network (MMP-HFN) was constructed. The MMP-HFN network is divided into five main modules: multimodal multi-phase image independent encoder, intramodal phase feature aggregation module, cross-modal feature alignment and fusion module, global feature sharing decoder, and detection-diagnosis dual task head (including 7 phase modality sensing sub-heads). The architecture logic is as follows: 7 images are independently encoded, intramodal aggregation is performed for CT / MRI, cross-modal alignment and fusion is performed for CT-MRI, global feature sharing is decoded, and BBox and benign / malignant probability prediction results are output for each of the 7 phases. The core design is to design independent feature encoders for the 7 phase modalities (to ensure the independence of features for each phase), while designing a shared feature decoder (to realize feature sharing for detection and diagnosis tasks), and configuring a dedicated task sub-head for each phase (to realize independent output of results for a single image).

[0061] In some embodiments, step S500 may include, but is not limited to, steps S510 to S550: Step S510: Independently encode the first slice image and the second slice image in the augmented training set to obtain the first feature map group in the CT modality and the second feature map group in the MRI modality; the first feature map group includes 3 first independent features and the second feature map group includes 4 second independent features. Step S520: Perform intra-modal feature aggregation on the first feature map group and the second feature map group respectively to obtain intra-modal aggregated features; Step S530: Perform cross-modal feature alignment and fusion on the first feature map group and the second feature map group to obtain cross-modal fused features; Step S540: Perform multi-scale fusion on the first independent feature, the second independent feature, the intra-modal aggregated feature, and the cross-modal fusion feature to generate a globally shared feature map; Step S550: After decoding the global shared feature map, input it into the dual-task backbone head, and output the liver nodule detection box prediction result and the benign / malignant probability prediction result.

[0062] In step S510 of some embodiments, lightweight convolutional encoders are designed for each of the seven phases, including three phases of CT and four phases of MRI. ~ All encoders have a consistent structure but their parameters are updated independently, ensuring that images from each phase are not affected by other phase modalities during feature extraction, while also adapting to the modal feature differences of CT / MRI. The encoder uses ResNet-18 as its basic backbone, removing fully connected layers and retaining convolutional layers, residual blocks, and pooling layers. A single 2D slice image and a single-channel second slice image are input into the corresponding encoder, which are then encoded independently, outputting 1024-channel phase-independent feature maps. (Each corresponds to one of the 7 phases), the feature maps preserve the spatial structure information of the image, providing spatial feature support for subsequent BBox detection. Optionally, the independent feature maps of the 7 phases are divided into two groups according to CT modality and MRI modality, resulting in the first feature map group under CT modality ( ) and the second feature map group in MRI modality ( ),in There are three first independent features. There are four second independent features.

[0063] In step S520 of some embodiments, the independent feature maps of the seven phases are divided into two groups, and intramodal phase feature aggregation is performed to mine the dynamic enhancement features of liver nodules in different phases under the same modality (such as the malignant features of CT arterial phase enhancement and portal venous phase washout). The core of the module is the Phase Attention Aggregation Layer (PAAL).

[0064] In some embodiments, step S520 may include, but is not limited to, steps S521 to S522: Step S521: Input the first feature map group into the phase attention aggregation layer of the CT modality, and perform weighted fusion of the three first independent features to obtain the first aggregated feature map of the CT modality; Step S522: Input the second feature map group into the phase attention aggregation layer of the MRI modality, and perform weighted fusion of the four second independent features to obtain the second aggregated feature map of the MRI modality; The intramodal aggregation features include a first aggregation feature map and a second aggregation feature map.

[0065] In step S521 of some embodiments, the CT three-phase feature map is... Inputting the CT-specific PAAL layer, this layer learns the weight allocation among the three phase features through a self-attention mechanism (e.g., arterial phase features of malignant nodules have higher weights), and performs weighted fusion of the three phase features to obtain the first aggregated feature map of the CT modality. At the same time, it retains the independent characteristics of the three phases. .

[0066] In step S522 of some embodiments, the MRI four-phase feature map is... Inputting the MRI-specific PAAL, and employing the same self-attention mechanism as CT, it learns the weight allocation of features across four phases (e.g., malignant features with low signal intensity in the hepatobiliary phase have higher weights), to obtain the second aggregated feature map of the MRI modality. At the same time, it retains the independent characteristics of the four phases. .

[0067] In some embodiments, during the intramodal aggregation process, a phase feature residual constraint is applied, meaning that the aggregated feature map must retain the residual information of each phase's independent features to avoid feature loss caused by excessive fusion of phase features.

[0068] In step S530 of some embodiments, the core issue of inconsistency between CT and MRI cross-modal features is addressed through a cross-modal feature alignment and fusion module, achieved via a cross-modal feature alignment layer (CFA) and a cross-modal fusion layer (CMF). and Spatial feature alignment and semantic feature fusion are achieved, while preserving modality aggregation features and phase-independent features.

[0069] In some embodiments, step S530 may include, but is not limited to, steps S531 to S532: Step S531: Spatial feature registration is performed on the first feature map group and the second feature map group to obtain the first aligned feature map group and the second aligned feature map group. Step S532: Using a dual-path attention fusion mechanism, cross-modal semantic fusion is performed on the first aligned feature map group and the second aligned feature map group to obtain cross-modal fused features.

[0070] In step S531 of some embodiments, the liver anatomical feature points are used to perform [the following] through the CFA layer. and Spatial feature registration is performed by adjusting the spatial position of the feature maps using deformable convolutional layers to eliminate spatial offset between CT and MRI during acquisition, resulting in a spatially aligned aggregated CT feature map. (First aligned feature map group) with MRI aggregated feature map (Second alignment feature map group).

[0071] In step S532 of some embodiments, a dual-path attention fusion mechanism is used through the CMF layer to calculate separately. right Attention weights and right Attention weights are used to achieve mutual enhancement of cross-modal semantic features, resulting in a cross-modal global fusion feature map. This refers to cross-modal fusion features.

[0072] In step S540 of some embodiments, the three-layer feature branches are preserved to provide multi-scale feature support for subsequent decoding, including phase-independent feature branches for each period. Modal aggregation feature branches Cross-modal global fusion feature branch Multi-scale fusion of the first independent feature, the second independent feature, intra-modal aggregated features, and cross-modal fused features can yield a globally shared feature map. .

[0073] In step S550 of some embodiments, the global feature sharing decoder uses an improved FPN (Feature Pyramid Network) as its basic architecture, introduces a multi-branch feature fusion layer, and fuses the above three feature branches at multiple scales to generate a multi-scale global shared feature map. (Including 8 scales, from 1×1 to 512×512), adaptable to feature extraction of liver nodules of different sizes (such as fine-grained features for small nodules and global features for large nodules). The core design is that the decoder shares all feature extraction parameters for both the bounding box prediction task and the benign / malignant probability prediction task, enabling the two tasks to mutually promote each other during the feature learning stage. The spatial features of the bounding box prediction task provide lesion localization support for the benign / malignant probability prediction task, while the pathological features of the benign / malignant probability prediction task provide lesion region screening criteria for the bounding box prediction task.

[0074] In some embodiments, step S550 may include, but is not limited to, steps S551 to S555: Step S551: Perform convolutional dimensionality reduction on the decoded global shared feature map to extract the detection box prediction task features; Step S552: Perform convolutional dimensionality reduction and global average pooling on the decoded global shared feature map to extract probabilistic prediction task features; Step S553: ​​Input the detection box prediction task features into the detection box prediction branch, and output the detection box coordinates and detection box confidence. Step S554: Crop the detection box feature region from the detection box prediction task features to obtain the lesion region features; Step S555: Input the probability prediction task features and lesion area features into the benign / malignant probability prediction branch, and output the benign / malignant prediction probability value.

[0075] In steps S551 to S552 of some embodiments, a dual-task backbone is constructed at the output of the shared decoder, and seven dedicated phase-aware subheadings are designed for the seven phase modes. ~ Each sub-header corresponds to a single image and independently outputs the liver nodule bounding box (BBox) and benign / malignant prediction results for that image, achieving accurate output from 7 images to 7 sets of BBoxes, and then to benign / malignant results. The dual-task backbone head includes a detection box prediction task backbone layer and a probability prediction task backbone layer, both constructed based on convolutional layers and fully connected layers.

[0076] In step S551 of some embodiments, in the backbone layer of the detection box prediction task, the shared feature map is... Perform convolutional dimensionality reduction to extract detection task-specific features. It includes features such as the spatial location, size, and shape of liver nodules, providing a basis for BBox prediction.

[0077] In step S552 of some embodiments, in the backbone layer of the probabilistic prediction task, the shared feature map is... Convolutional dimensionality reduction and global average pooling are performed to extract diagnostic task-specific features. It includes pathological features of liver nodules such as grayscale, texture, and enhancement patterns, providing a basis for predicting benign or malignant conditions.

[0078] In steps S553 to S555 of some embodiments, for the phase modality perception subhead, the seven subheads have the same structure, all of which include a detection box prediction branch and a benign / malignant probability prediction branch. Each subhead is equipped with a phase modality feature gating layer, which introduces the text prompt word feature vector of the corresponding phase as a gating signal, so that the subhead can accurately perceive the modality features of the corresponding phase and avoid cross-phase interference.

[0079] In step S553 of some embodiments, task-specific features are detected. The input subheading predicts bounding box branches, generates multi-scale anchor boxes through an anchor mechanism, and after non-maximum suppression (NMS) filtering, outputs the precise BBox coordinates of the liver nodules. , The output is the coordinates of the top left and bottom right corners of the image and the BBox confidence score (0~1, representing the probability that the location is a liver nodule); for images without liver nodules, an empty BBox and a confidence score of 0 are output.

[0080] In steps S554 to S555 of some embodiments, diagnostic task-specific features are... The lesion region features in the output of the detection box prediction branch (from The feature regions corresponding to the BBox are cropped out and fused. The benign and malignant probability prediction branch is input, and the malignant probability of liver nodules (0~1) and benign probability (1-malignant probability) are output through a fully connected layer and a sigmoid activation function. For images without liver nodules, the benign and malignant probabilities are both 0.

[0081] In some embodiments, the gating layer of each sub-head uses the text prompt word feature vector of the corresponding phase to weight the input features, so that the sub-head only focuses on the feature information of the corresponding phase, ensuring that the output results of the 7 sub-heads are independent of each other and match the corresponding images.

[0082] In some embodiments, a dual-task loss function was designed, and the multimodal, multi-phase hierarchical fusion network was trained and inferred. For example, a multi-task joint loss function was used to weightedly fuse detection and diagnostic losses, while simultaneously applying a cross-phase consistency constraint loss to ensure consistency between detection and diagnostic results for the same liver nodule across seven phases. The total loss function... The formula is: ,in Weighting coefficients (e.g.) The design of each loss term is as follows: (1) Detection loss A combined loss function of Focal Loss and GIoU Loss is used. Focal Loss addresses the class imbalance problem in liver nodule samples (background is much larger than lesions), while GIoU Loss improves the accuracy of BBox prediction. The formula is as follows: .

[0083] (2) Diagnostic loss We employ weighted cross-entropy loss to address the imbalance between good and bad samples, assigning higher weights to minority class samples. The formula is as follows: in The label represents the true benign or malignant nature (1 for malignant, 0 for benign). To predict the probability of malignancy, , Weights for benign and malignant samples.

[0084] (3) Intertemporal consistency constraint loss For the same liver nodule, the mean square error of the BBox coordinate deviation and the benign / malignant probability deviation of the output of the seven phases is calculated to force the detection and diagnosis results of each phase to be consistent. The formula is as follows: ,in , For the first The BBox coordinates of the head and the probability of malignancy. , The coordinates of the 7 child boxes and the mean of the probability of malignancy.

[0085] For example, the training process for a multimodal, multi-phase hierarchical fusion network includes: dividing the augmented training set into training, validation, and test sets in an 8:1:1 ratio; using the AdamW optimizer to perform end-to-end training on the MMP-HFN network, setting a learning rate decay strategy (cosine annealing), iteratively training until the total loss function converges, and selecting the optimal model based on the mean accuracy (mAP) of the BBox on the validation set and the accuracy of benign / malignant probability prediction (ACC); and using random data augmentation (such as rotation, flipping, and Gaussian blur) during training to further improve the model's generalization ability.

[0086] For example, a new clinical single-group sample (7 two-dimensional slice images from CT phase III and MRI phase IV) is input into the trained MMP-HFN network after spatial normalization and grayscale normalization consistent with the training phase. The inference process is as follows: the 7 images are input into their respective independent encoders to extract phase-independent features; after intramodal aggregation and cross-modal alignment and fusion, globally shared features are obtained; after decoding by the shared decoder, they are input into the 7 phase-sensing subheads of the dual task head; each subhead independently outputs the BBox coordinates (including confidence) and benign / malignant probability of the corresponding liver nodule in the image, and finally outputs 7 sets of results to complete the detection and benign / malignant diagnosis of the liver nodule in the sample.

[0087] This invention also provides a liver nodule detection frame and benign / malignant probability prediction device based on CT and MRI multimodal images, which can realize the above-mentioned liver nodule detection frame and benign / malignant probability prediction method based on CT and MRI multimodal images. The device includes: The first module is used to acquire the original training set and preprocess the original training set to obtain the preprocessed training set. The original training set includes the original single-group samples, and the preprocessed training set includes the preprocessed single-group samples. Each single-group sample includes the first slice image of three phases in the CT modality and the second slice image of four phases in the MRI modality. The second module is used to construct random single-period phase missing for each preprocessed single-group sample to generate training sample pairs. The third module is used to train the initial conditional diffusion model based on the training sample pairs to obtain the target conditional diffusion model; The fourth module is used to generate multi-missing mode images from the original training set using the target conditional diffusion model, thereby obtaining an augmented training set. The fifth module is used to independently encode, aggregate intra-modal features, align and fuse cross-modal features, fuse multiple scales, and decode features on the augmented training set through a multimodal, multi-phase hierarchical fusion network to obtain the prediction results of liver nodule detection boxes and the prediction results of benign and malignant probabilities.

[0088] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0089] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including a tablet computer, an in-vehicle computer, or similar device.

[0090] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0091] refer to Figure 2 , Figure 2 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention. The memory 602 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601. The input / output interface 603 is used to implement information input and output; The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604); The processor 601, memory 602, input / output interface 603, and communication interface 604 are connected to each other within the device via bus 605.

[0092] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0093] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0094] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions to cause the computer device to perform the aforementioned method.

[0095] In summary, the liver nodule detection frame and benign / malignant probability prediction method and device based on CT and MRI multimodal images of the present invention have the following advantages: 1. The embodiments of the present invention are based on the hierarchical fusion feature learning mechanism of CT three-phase + MRI four-phase seven-modal images. First, the phase feature aggregation within the CT modality and within the MRI modality is completed. Then, the feature alignment and fusion of CT and MRI across modalities are performed to achieve effective coupling of multimodal and multiphase image features and solve the problem of inconsistency between cross-phase and cross-modal features.

[0096] 2. The present invention combines the target phase identifier text and the remaining phase images in the same group under two conditions to generate a controllable diffusion data augmentation method. It generates a complete image by randomly constructing a single phase missing scene, and uses the same-level constraint to realize the direct reuse of the original BBox annotation. It expands the training samples and covers the phase missing distribution without additional annotation.

[0097] 3. The embodiments of the present invention construct an integrated framework of shared decoder and dual task heads through an end-to-end joint optimization network structure of liver nodule detection box localization detection and benign / malignant probability prediction. This enables spatial information of localization and qualitative pathological information to promote each other, reduce the accumulation of errors in serial tasks, and output liver nodule BBox detection results and benign / malignant classification results.

[0098] 4. The multi-task joint training constraint strategy designed for 7-modality images in this embodiment of the invention includes consistency constraints on lesion regions, consistency constraints on cross-phase detection / diagnosis results, and spatial structure constraints on the generated completed image and the original image, thereby improving the stability of model training and the performance of downstream tasks.

[0099] 5. The embodiments of the present invention provide a robust processing scheme for multimodal and multiphase image input. By using diffusion generation to enhance and cover different modal / phase missing configurations, the model can still stably complete liver nodule detection and benign / malignant diagnosis under multi-center, multi-device, different acquisition conditions and modal combination input.

[0100] 6. The embodiments of the present invention realize the deep coupling of generative data augmentation and joint learning of detection-diagnosis, and directly integrate the completed images generated by the diffusion model into the joint training process of multimodal and multi-phase images, ensuring the stable improvement of downstream detection and diagnosis performance by image synthesis / completeness from the method structure.

[0101] 7. The embodiments of the present invention are based on an integrated processing scheme for a single set of samples (one two-dimensional slice image for each of the seven phase modalities), which can perform collaborative analysis of CT and MRI multi-phase images of the same subject, output liver nodule detection and diagnosis results corresponding to each phase image, and match the clinical multi-phase image diagnosis process.

[0102] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0103] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0104] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0106] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0107] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0108] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0109] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0110] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A method for detecting liver nodules and predicting benign / malignant probabilities based on CT and MRI multimodal images, characterized in that, Includes the following steps: Obtain the original training set and preprocess the original training set to obtain the preprocessed training set; the original training set includes the original single-group samples, and the preprocessed training set includes the preprocessed single-group samples; each single-group sample includes a first slice image of three phases under CT modality and a second slice image of four phases under MRI modality. For each preprocessed single-group sample, a random single-period phase missing construct is performed to generate training sample pairs; Based on the training sample pairs, the initial conditional diffusion model is trained to obtain the target conditional diffusion model; Using the target conditional diffusion model, a multi-missing mode image generation operation is performed on the original training set to obtain an augmented training set; By using a target multimodal multi-phase hierarchical fusion network, the augmented training set is independently encoded, intramodal feature aggregation is performed, cross-modal feature alignment and fusion is performed, multi-scale fusion is performed, and feature decoding is performed to obtain the prediction results of liver nodule detection boxes and the prediction results of benign and malignant probabilities.

2. The method according to claim 1, characterized in that, The process of obtaining the original training set and preprocessing it to obtain the preprocessed training set includes the following steps: Spatial normalization is performed on the first slice image and the second slice image of each of the single samples to obtain the first aligned image and the second aligned image. Intramodal grayscale normalization is performed on the first aligned image and the second aligned image respectively to obtain the first grayscale image and the second grayscale image. The liver nodule detection box is mapped to the coordinate system of the first grayscale image and the coordinate system of the second grayscale image to generate a lesion mask. Based on each of the aforementioned phases, a standardized text prompt vocabulary library is constructed; The preprocessed training set is obtained based on the original training set, the lesion mask, and the standardized text prompt vocabulary.

3. The method according to claim 1, characterized in that, The step of constructing random single-phase missing data for each preprocessed single-group sample to generate training sample pairs includes the following steps: Obtain the preprocessed slice image from the single group of samples after preprocessing; Random sampling is performed on all phases to determine the missing and non-missing phases; The preprocessed slice image corresponding to the non-missing period is subjected to intramodal feature stitching and cross-modal feature stitching to obtain the first image conditional feature map; Based on a standardized text prompt vocabulary, extract the feature vector of the first text prompt word corresponding to the missing period; The first image conditional feature map and the first text prompt word feature vector are fused to obtain a joint conditional input. The preprocessed slice image corresponding to the missing period is used as the initial label; The lesion mask in the preprocessed training set is incorporated into the initial label to obtain the generated label; The training sample pairs are generated based on the joint conditional input and the generated labels.

4. The method according to claim 1, characterized in that, Based on the training sample pairs, the initial conditional diffusion model is trained to obtain the target conditional diffusion model, including the following steps: Stepwise Gaussian noise is added to the generated labels in the training sample pairs to obtain noisy images with different noise step sizes; The intermediate feature vector of the noise step is concatenated with the joint conditional input in the training sample pair to obtain the input features of the target denoising diffusion backbone network. A modality-phase attention layer is added to the initial encoder and initial decoder of the initial denoising and diffusion backbone network, and a lesion region constraint layer is added to the initial decoder to obtain the target denoising and diffusion backbone network. Based on the target denoising diffusion backbone network, the denoising diffusion probability model is improved to obtain the initial conditional diffusion model; Obtain the denoised prediction image under the noise step size, and construct a first loss function based on the noisy image and the denoised prediction image; The initial conditional diffusion model is trained based on the input features and the first loss function to obtain the target conditional diffusion model.

5. The method according to claim 1, characterized in that, The step of generating an augmented training set by performing a multi-missing mode image generation operation on the original training set using the target conditional diffusion model includes the following steps: The target conditional diffusion model is used to process the original training set by removing a single phase, resulting in the first enhanced sample. By applying the target conditional diffusion model to the original training set through multiple phase loss processing, a second enhanced sample is obtained. The first augmented sample, the second augmented sample, and the original single-group sample are merged to obtain the augmented training set.

6. The method according to claim 1, characterized in that, The process involves using a multimodal, multi-phase hierarchical fusion network to independently encode, aggregate intra-modal features, align and fuse cross-modal features, fuse at multiple scales, and decode features from the augmented training set to obtain liver nodule detection box prediction results and benign / malignant probability prediction results. This includes the following steps: The first slice image and the second slice image in the augmented training set are independently encoded to obtain a first feature map group in the CT modality and a second feature map group in the MRI modality; the first feature map group includes 3 first independent features and the second feature map group includes 4 second independent features. Intramodal feature aggregation is performed on the first feature map group and the second feature map group respectively to obtain intramodal aggregated features; Cross-modal feature alignment and fusion are performed on the first feature map group and the second feature map group to obtain cross-modal fused features; Multi-scale fusion is performed on the first independent feature, the second independent feature, the intra-modal aggregated feature, and the cross-modal fusion feature to generate a globally shared feature map; After decoding the global shared feature map, it is input into the dual-task backbone head, and the predicted results of liver nodule detection boxes and the predicted probabilities of benign and malignant nodules are output.

7. The method according to claim 6, characterized in that, The step of performing intra-modal feature aggregation on the first feature map group and the second feature map group respectively to obtain intra-modal aggregated features includes the following steps: The first feature map group is input into the phase attention aggregation layer of the CT modality, and the three first independent features are weighted and fused to obtain the first aggregated feature map of the CT modality. The second feature map group is input into the phase attention aggregation layer of the MRI modality, and the four second independent features are weighted and fused to obtain the second aggregated feature map of the MRI modality. The intramodal aggregation features include the first aggregation feature map and the second aggregation feature map.

8. The method according to claim 6, characterized in that, The process of performing cross-modal feature alignment and fusion on the first feature map group and the second feature map group to obtain cross-modal fused features includes the following steps: Spatial feature registration is performed on the first feature map group and the second feature map group to obtain the first aligned feature map group and the second aligned feature map group; A dual-path attention fusion mechanism is used to perform cross-modal semantic fusion on the first aligned feature map group and the second aligned feature map group to obtain the cross-modal fused features.

9. The method according to claim 6, characterized in that, The process of decoding the globally shared feature map and inputting it into the dual-task backbone head to output the liver nodule detection box prediction result and the benign / malignant probability prediction result includes the following steps: The decoded global shared feature map is subjected to convolutional dimensionality reduction to extract the detection box prediction task features; The decoded global shared feature map is subjected to convolutional dimensionality reduction and global average pooling to extract probabilistic prediction task features; Input the detection box prediction task features into the detection box prediction branch, and output the detection box coordinates and the detection box confidence score; The lesion region features are obtained by cropping the detection box feature region from the predicted task features of the detection box. The probability prediction task features and the lesion area features are input into the benign / malignant probability prediction branch, and the benign / malignant prediction probability value is output.

10. A liver nodule detection frame and benign / malignant probability prediction device based on CT and MRI multimodal images, characterized in that, include: The first module is used to acquire the original training set and preprocess the original training set to obtain the preprocessed training set. The original training set includes original single-group samples, and the preprocessed training set includes preprocessed single-group samples; each single-group sample includes a first slice image of three phases in the CT modality and a second slice image of four phases in the MRI modality. The second module is used to construct random single-period phase missing for each preprocessed single group of samples to generate training sample pairs. The third module is used to train the initial conditional diffusion model based on the training sample pairs to obtain the target conditional diffusion model. The fourth module is used to perform multi-missing mode image generation operation on the original training set through the target conditional diffusion model to obtain an augmented training set. The fifth module is used to perform independent encoding, intra-modal feature aggregation, cross-modal feature alignment and fusion, multi-scale fusion and feature decoding on the augmented training set through a target multimodal multi-phase hierarchical fusion network to obtain the liver nodule detection box prediction results and benign / malignant probability prediction results.

Images