Non-invasive biopsy multimodal image fusion system and method

Abstract

This invention relates to the field of medical image processing, specifically to a multimodal medical image fusion system and method, comprising a data preprocessing module, a domain manifold embedding module, a hierarchical adaptive domain alignment network module, a multimodal feature fusion module, and a diagnostic decision module. The system establishes a topological correspondence between EBUS ultrasound, OCT tomography, and DWI functional images through a multidimensional domain manifold embedding structure; employs a hierarchical adaptive domain alignment network to achieve multi-scale feature extraction and precise alignment; designs a topology-preserving multimodal feature fusion mechanism to ensure the complete preservation of key diagnostic information from each modality during the fusion process; and utilizes a dual-path cross-validation decision strategy to improve the stability and reliability of diagnosis. The system inputs the domain-aligned multimodal features into a Transformer network and generates nodule detection and benign / malignant classification results through dual-threshold judgment, achieving accurate and non-invasive diagnosis of small lesions such as pulmonary nodules and early gastrointestinal lesions.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, specifically to a multimodal medical image fusion system and method, and particularly to a non-invasive biopsy multimodal image fusion technology combining endoscopic ultrasound (EBUS), optical coherence tomography (OCT), and diffusion-weighted imaging (DWI). Background Technology

[0002] Currently, medical imaging technology plays an increasingly important role in clinical diagnosis. Different imaging techniques have different imaging mechanisms and characteristics, and can provide complementary physiological and pathological information. For example, endoscopic ultrasound (EBUS) can provide information on the acoustic properties of tissues, optical coherence tomography (OCT) can provide high-resolution images of tissue microstructures, while diffusion-weighted imaging (DWI) can provide functional information of tissues.

[0003] However, due to significant differences in imaging principles, resolution, contrast, and signal-to-noise ratio among different modalities of medical images, the fusion of multimodal medical images faces enormous challenges. Existing technologies for multimodal medical image fusion mainly suffer from the following problems: First, there are significant domain differences between images of different modalities, which often prevent direct fusion from yielding satisfactory results. Existing registration methods mainly rely on the geometric features of the images, making it difficult to handle situations with large modal differences.

[0004] Secondly, traditional fusion methods often employ simple techniques such as weighted average or principal component analysis, which make it difficult to preserve the key diagnostic information contained in each modality, resulting in a decrease in the diagnostic value of the fused images.

[0005] Third, existing multimodal imaging diagnostic systems usually require subjective judgment from professional doctors and lack objective and reliable automated diagnostic support. Diagnostic results often depend on doctors' experience and subjective judgment.

[0006] In addition, traditional biopsy methods are often invasive, causing pain and risks to patients, while existing non-invasive biopsy techniques have problems with insufficient accuracy.

[0007] Therefore, there is an urgent need to develop a system and method that can effectively integrate multimodal medical images and provide accurate non-invasive biopsy diagnosis. Summary of the Invention

[0008] The purpose of this invention is to provide a non-invasive biopsy multimodal image fusion system and method, which aims to solve the problems of difficulty in multimodal medical image fusion, information loss after fusion, and insufficient diagnostic accuracy in the prior art.

[0009] This invention proposes a non-invasive biopsy multimodal image fusion system, comprising: Domain manifold embedding module, used for: It receives EBUS ultrasound images, OCT tomographic images, and DWI functional images as input; Construct a three-domain data structure that includes the EBUS ultrasound dataset, the OCT tomographic dataset, and the DWI functional dataset; Generate a domain mapping table, which is used to establish the topological correspondence between the EBUS ultrasound images, OCT tomographic images and DWI functional images; The hierarchical adaptive domain alignment network module, connected to the domain manifold embedding module, is used for: Receive the three-domain data structure and the domain mapping table; Multiscale features of the EBUS ultrasound image, OCT tomography image and DWI functional image are extracted using a three-level feature extraction structure. The multi-scale features are encoded using an autoencoder discriminator network to obtain a feature vector; Based on the feature vector, conditional adversarial optimization is performed to achieve domain alignment between the EBUS ultrasound images, OCT tomographic images, and DWI functional images. The multimodal feature fusion module, connected to the hierarchical adaptive domain alignment network module, is used for: Receive domain-aligned EBUS ultrasound image features, OCT tomographic image features, and DWI functional image features. A shared coding space is constructed, and the EBUS ultrasound image features, OCT tomographic image features, and DWI functional image features are mapped to the shared coding space. Calculate the importance weights of modal features, and perform multi-scale feature fusion based on the weights; Through topology verification, fusion features that preserve the topology are generated; The diagnostic decision module, connected to the multimodal feature fusion module, is used for: Receive the fusion feature; The fused features are processed by a Transformer network to generate pixel-level classification maps; The pixel-level classification mapping is judged based on a preset threshold to obtain nodule detection results and benign / malignant classification results; Output fused images and diagnostic analysis results.

[0010] Preferably, the domain manifold embedding module includes: A domain mapping construction unit is used to construct a four-dimensional domain mapping structure, which includes an EBUS ultrasound dataset U, an OCT tomography dataset T, a DWI functional dataset F, and a multimodal mapping reference dataset M. The local neighborhood preservation unit is used to define a set of k nearest neighbors for each modal data point and preserve the relative positional relationships of the points within the set of k nearest neighbors in the embedding space. The inter-domain mapping bridging unit is used to establish a bidirectional mapping relationship between the EBUS ultrasound dataset U, the OCT tomographic dataset T, and the DWI functional dataset F, and selects a set of anchor points as the reference points for cross-domain correspondence based on the similarity of image features. The domain embedding verification unit is used to calculate the intra-domain distance metric and the inter-domain distance metric, and to determine the validity of the domain embedding structure based on the ratio of the intra-domain distance metric to the inter-domain distance metric.

[0011] Preferably, the hierarchical adaptive domain alignment network module includes: The three-level feature extraction unit is used to extract low-level features, intermediate features, and high-level features from EBUS ultrasound images, OCT tomographic images, and DWI functional images, respectively. The resolution of the low-level features is 128×128, the resolution of the intermediate features is 64×64, and the resolution of the high-level features is 32×32. An autoencoder discriminator unit is used to encode and decode the features of each domain. The encoding part maps the input features to an h1-dimensional feature vector, and the decoding part reconstructs the h1-dimensional feature vector back to the original input size. The conditional adversarial optimization unit is used to construct a conditional adversarial learning framework based on the domain identifier vector, and to distinguish between real domain features and transformed domain features. The progressive training control unit controls the training process in four stages: training each encoder individually, introducing adversarial learning, enabling cycle consistency constraints, and jointly optimizing all components.

[0012] Preferably, the multimodal feature fusion module includes: A shared coding space construction unit is used to design a coding network h2 as a shared mapping space for different modal features. The structure of the coding network h2 consists of an input layer, a 128×128×m-dimensional intermediate layer, a 64×64×2m-dimensional intermediate layer, a 32×32×4m-dimensional intermediate layer, a 16×16×8m-dimensional intermediate layer, and a shared feature space output layer, where m is the number of input channels. The feature importance weighting unit is used to calculate the importance weights w_U, w_T, and w_F of three modal features: EBUS ultrasound images, OCT tomography images, and DWI functional images, and dynamically adjusts the weights based on the contribution of each modality to the current diagnostic task. The multi-scale feature fusion unit is used to perform feature fusion at three scales: global, regional, and local. It integrates the fusion results of different scales through skip connections to form a multi-resolution feature representation. The topology verification unit is used to evaluate the degree of preservation of the topology before and after fusion, and to trigger the topology optimization process when the structural similarity is lower than a preset threshold.

[0013] Preferably, the diagnostic decision module includes: A Transformer processing unit is used to receive fused features and process the fused features through a Transformer network configured with 8 attention heads and 6 encoder layers; The pixel-level classification unit is used to perform pixel-level classification on the output of the Transformer network and generate predicted values ​​ranging from 0 to 100. The dual threshold judgment unit is used to determine the category of each pixel based on the nodule detection threshold τ1=20 and the benign / malignant classification threshold τ2=40. Pixels with a predicted value greater than τ1 are identified as nodules, and nodules with a predicted value greater than τ2 are classified as malignant nodules. The dual-path cross-validation unit is used to improve the reliability of diagnosis by cross-validating the results of two independent diagnostic paths. The two diagnostic paths are: a first path combining enhanced OCT images and DWI images with original EBUS images, and a second path combining original EBUS images and enhanced OCT images with enhanced DWI images.

[0014] Preferably, a data preprocessing module is also included, connected to the domain manifold embedding module, for: Temporal denoising and median filtering are performed on the EBUS ultrasound images, wherein the window size of the median filtering is 3 pixels, and the processed EBUS ultrasound images are downsampled at multiple scales to generate multi-scale EBUS ultrasound images with resolutions of 128×128, 64×64 and 32×32. The RGB three-channel matrix was extracted from the OCT tomographic images based on DICOM information, and each channel was independently processed into a standardized 128×128 image. Extract sequential images from DWI images based on DICOM information and process them into uniform 128×128 images; The processed EBUS ultrasound images, OCT tomographic images, and DWI functional images are converted into NumPy file format and used as input to the domain manifold embedding module.

[0015] Preferably, the autoencoder discriminator network of the hierarchical adaptive domain alignment network module adopts the following staged training strategy: During the initialization phase, 10 rounds of independent pre-training are performed for each encoder. During the alignment initiation phase, 15 rounds of coarse alignment training were performed to introduce adversarial learning. In the fine-tuning phase, 20 rounds of fine-tuning training with cycle consistency constraints are performed; During the global joint optimization phase, 40 rounds of joint training of all components are performed. During training, an adaptive learning rate adjustment mechanism is used. The initial learning rate is 0.001. When the ratio of the intra-domain distance metric to the inter-domain distance metric is greater than 0.8 times the preset threshold, the learning rate is multiplied by 0.8. When there is no improvement in performance for 5 consecutive rounds, the learning rate is multiplied by 0.5. The minimum learning rate is limited to 0.00001.

[0016] Preferably, the fusion process of the multi-scale feature fusion unit of the multi-modal feature fusion module includes: Global feature fusion: For features with a resolution of 32×32, the overall semantic information is fused. Regional feature fusion: For features with a resolution of 64×64, local structural information is fused. Local feature fusion: For features at 128×128 resolution, micro-texture information is fused. Feature integration integrates global features, regional features, and local features into a multi-resolution feature representation through skip connections; Feature selection involves eliminating redundancy in the integrated features based on the information gain principle.

[0017] Preferably, the dual-path cross-validation unit of the diagnostic decision module is further used for: When the results of two diagnostic paths are inconsistent, a weighted decision is made based on the confidence level of the predicted values. Regional connectivity analysis is performed on diagnostic results from different locations to eliminate isolated abnormal pixels; Generate a structured diagnostic report that includes nodule location, size, shape, and benign / malignant classification.

[0018] The method based on the aforementioned non-invasive biopsy multimodal image fusion system includes the following steps: It receives EBUS ultrasound images, OCT tomographic images, and DWI functional images as input; The EBUS ultrasound images, OCT tomographic images, and DWI functional images are preprocessed to generate standardized multi-scale image data. Construct a domain manifold embedding structure, establish the topological correspondence between the EBUS ultrasound images, OCT tomographic images and DWI functional images, and generate a domain mapping table. Based on the domain mapping table, domain alignment between the EBUS ultrasound images, OCT tomography images and DWI functional images is achieved through a hierarchical adaptive domain alignment network. The domain-aligned EBUS ultrasound image features, OCT tomographic image features, and DWI functional image features are mapped to a shared coding space, and multi-scale feature fusion is performed to generate fused features that preserve the topological structure. The fused features are processed by a Transformer network to generate pixel-level classification maps; Based on preset nodule detection thresholds and benign / malignant classification thresholds, the pixel-level classification mapping is judged to obtain nodule detection results and benign / malignant classification results; Output fused images and diagnostic analysis results.

[0019] The present invention has the following beneficial effects: 1. By designing a multidimensional domain manifold embedding structure, a topological correspondence between different modal medical images was established, which effectively solved the problem of alignment difficulties when there are large modal differences and improved the accuracy of fusion.

[0020] 2. A hierarchical adaptive domain alignment network is adopted to achieve multi-scale feature extraction and accurate alignment, which overcomes the limitation of insufficient information in a single modality and enhances the information integrity of the fused image.

[0021] 3. A topology-preserving multimodal feature fusion mechanism was designed to ensure the complete preservation of key diagnostic information of each modality during the fusion process, thereby improving the diagnostic value of the fused images.

[0022] 4. By adopting a dual-path cross-validation decision-making strategy, the risk of misdiagnosis caused by a single judgment is effectively reduced, and the stability and reliability of the system diagnosis are improved.

[0023] 5. It reduces the need for invasive diagnostics, provides highly accurate non-invasive diagnosis through multimodal image fusion, reduces the risk of biopsy-related complications, and alleviates patient suffering.

[0024] 6. It improves early diagnostic capabilities, significantly enhancing the detection rate of small nodules and the accuracy of distinguishing between benign and malignant early lesions, which helps to achieve early detection and intervention of malignant tumors. Attached Figure Description

[0025] Figure 1 This is a structural framework diagram of the non-invasive biopsy multimodal image fusion system of the present invention.

[0026] Figure 2 This is a schematic diagram of the structure of the domain manifold embedding module of the present invention.

[0027] Figure 3 This is a schematic diagram of the hierarchical adaptive domain alignment network module of the present invention.

[0028] Figure 4 This is a schematic diagram of the structure of the multimodal feature fusion module of the present invention.

[0029] Figure 5 This is a schematic diagram of the diagnostic decision module of the present invention.

[0030] Figure 6 This is a flowchart of the non-invasive biopsy multimodal image fusion method of the present invention.

[0031] Figure 7 This is a flowchart of the data preprocessing module of the present invention.

[0032] Figure 8 This is a schematic diagram of the structure of the three-level feature extraction unit of the present invention.

[0033] Figure 9 This is a schematic diagram of the fusion process of the multi-scale feature fusion unit of the present invention.

[0034] Figure 10 This is a flowchart of the dual-path cross-validation unit of the present invention. Detailed Implementation

[0035] Please refer to the attached document. Figure 1-10 The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0036] like Figure 1 As shown, the non-invasive biopsy multimodal image fusion system provided by the present invention includes a domain manifold embedding module 1, a hierarchical adaptive domain alignment network module 2, a multimodal feature fusion module 3, a diagnostic decision module 4, and a data preprocessing module 5.

[0037] The data preprocessing module 5 is connected to the domain manifold embedding module 1 and is used to preprocess EBUS ultrasound images, OCT tomographic images, and DWI functional images. The domain manifold embedding module 1 is connected to the hierarchical adaptive domain alignment network module 2 and is used to construct domain mapping relationships. The hierarchical adaptive domain alignment network module 2 is connected to the multimodal feature fusion module 3 and is used to achieve domain alignment. The multimodal feature fusion module 3 is connected to the diagnostic decision module 4 and is used to generate fused features. The diagnostic decision module 4 is used to output diagnostic results.

[0038] In a preferred embodiment of the present invention, the system workflow is as follows: First, the system receives EBUS ultrasound images, OCT tomographic images, and DWI functional images as input; then, the data preprocessing module 5 preprocesses these images; next, the domain manifold embedding module 1 constructs a domain mapping table to establish topological correspondences between images of different modalities; subsequently, the hierarchical adaptive domain alignment network module 2 realizes domain alignment of images of different modalities; then, the multimodal feature fusion module 3 performs feature fusion to generate fused features that maintain the topological structure; finally, the diagnostic decision module 4 processes the fused features and outputs diagnostic results.

[0039] like Figure 2 As shown, the domain manifold embedding module 1 includes a domain mapping construction unit 11, a local neighborhood preservation unit 12, an inter-domain mapping bridging unit 13, and a domain embedding verification unit 14.

[0040] Domain mapping building unit 11 is used to construct a four-dimensional domain mapping structure, including the EBUS ultrasound dataset U, the OCT tomography dataset T, the DWI functional dataset F, and the multimodal mapping reference dataset M. Each domain data retains its original 128×128×n-dimensional features (where n is the number of channels) and is appended with a domain identifier vector. The domain identifier vectors maintain similar values ​​within the same modality and exhibit significant differences between different modalities.

[0041] For example, in one embodiment of the present invention, the domain identifier vector of the EBUS ultrasound dataset U can be set to [1,0,0], the domain identifier vector of the OCT tomography dataset T can be set to [0,1,0], the domain identifier vector of the DWI functional dataset F can be set to [0,0,1], and the domain identifier vector of the multimodal mapping reference dataset M is set according to its corresponding modal combination.

[0042] The local neighborhood preservation unit 12 is used to define a set of k nearest neighbors for each modal data point and preserve the relative positional relationships of points within the k nearest neighbors set in the embedding space. Specifically, for each modal data point x, its set of k nearest neighbors N(x) is defined, and the neighborhood preservation loss function is calculated as follows. Ensure topological invariance in the embedded space: , in: The neighborhood-preserving loss function is used; D represents the dataset; x represents a data point in the dataset; Describe the set of k nearest neighbors of x; Represents the Euclidean distance between x and y in the original space; This represents the distance of x to its farthest neighbor; This represents the mapping of x in the embedding space; Indicates embedding space and The Euclidean distance; express The distance between it and its farthest neighbor in the embedded space.

[0043] In a preferred embodiment of the present invention, the value of k is set to 10, because this has been verified through numerous experiments. It can achieve a good balance between preserving local structural information and reducing computational complexity.

[0044] Inter-domain mapping bridging unit I3 is used to establish a bidirectional mapping relationship between the EBUS ultrasound dataset U, the OCT tomographic dataset T, and the DWI functional dataset F, and selects a set of anchor points as the reference points for cross-domain correspondence based on image feature similarity. Specifically, a bidirectional mapping relationship is established between the three source domains: ; ; Establish a set of "anchor points" for each pair of domains. , serving as the benchmark for cross-domain correspondence.

[0045] The selection of anchor points is based on a similarity metric of image features to ensure the reliability of the mapping. In one embodiment of the invention, the similarity metric can be calculated using the following formula: , in: For data points and A measure of similarity between them; and These represent data points from two different domains; This represents the feature representation of data point x; This represents the squared Euclidean distance between the feature representations; This is a parameter that controls the rate of similarity decay; in this embodiment, it is set to 0.5. When When the value exceeds a preset threshold (e.g., 0.8), Add to the anchor point set.

[0046] The domain embedding verification unit 14 is used to calculate the intra-domain distance metric and the inter-domain distance metric, and to determine the validity of the domain embedding structure based on the ratio of the intra-domain distance metric to the inter-domain distance metric. Specifically, the intra-domain distance metric is defined. Inter-domain distance metric : , , in: Indicates the distance metric within the domain; This represents the distance metric between domains; Indicates the size of the dataset (number of samples); Indicates and A set of data points belonging to the same domain; This represents the representation of data point x in the embedding space; This represents the squared Euclidean distance between two points in the embedding space; This indicates that data point j belongs to the... Same domain; This indicates that data point j does not belong to the... Same domain.

[0047] when Below the preset threshold (In a preferred embodiment of the invention) When the value is set to 0.3), the domain embedding structure is considered valid. The domain embedding structure is continuously optimized through a feedback loop until the verification criteria are met.

[0048] like Figure 3 As shown, the hierarchical adaptive domain alignment network module 2 includes a three-level feature extraction unit 21, an autoencoder discriminator unit 22, a conditional adversarial optimization unit 23, and a progressive training control unit 24.

[0049] The three-level feature extraction unit 21 is used to extract low-level, mid-level, and high-level features from EBUS ultrasound images, OCT tomographic images, and DWI functional images, respectively. For example... Figure 8 As shown, the low-level feature layer captures pixel-level texture and edge information with a resolution of 128×128; the mid-level feature layer extracts structural and morphological features with a resolution of 64×64; and the high-level feature layer identifies semantic-level diagnostic features with a resolution of 32×32. Each level has an independent feature extraction unit, forming a feature pyramid structure.

[0050] In a preferred embodiment of the present invention, feature extraction is implemented using a multi-layer convolutional neural network. For example, low-level feature extraction can use 3 convolutional layers, each using a 3×3 convolutional kernel, with a stride of 1 and padding of 1; mid-level feature extraction adds 2 convolutional layers to the low-level features and uses downsampling with a stride of 2; high-level feature extraction adds 2 more convolutional layers to the mid-level features, also using downsampling with a stride of 2.

[0051] The autoencoder discriminator unit 22 is used to encode and decode the features of each domain. Specifically, an autoencoder discriminator is configured for each domain (U, T, F). The encoding part maps the input features to an h1-dimensional feature vector, and the decoding part reconstructs the h1-dimensional feature vector back to the original input size.

[0052] In one embodiment of the present invention, the structure of the encoding part is: input → 128n → 64n → 32n → 16n → h1-dimensional feature vector (where n is the number of channels, and the initial h1 value is 32); the structure of the decoding part is: h1-dimensional feature vector → 16n → 32n → 64n → 128n → reconstructed output. The encoding dimension h1 is dynamically adjusted: the initial value is 32, and it is adaptively adjusted according to the alignment difficulty.

[0053] The conditional adversarial optimization unit 23 is used to construct a conditional adversarial learning framework based on the domain identifier vector, distinguishing between real domain features and transformed domain features. Specifically, the discrimination objective is to distinguish between real domain features and transformed domain features, and the generation objective is to generate domain-transformed features that the discriminator cannot distinguish.

[0054] In a preferred embodiment of the present invention, the loss function for conditional adversarial learning can be expressed as: , in: The adversarial loss function is represented by D; the discriminator by G; the generator by x; the real data sample by z; the random noise input by c; and the conditional domain identifier vector by c. Represents the actual data distribution; Indicates noise distribution. This represents the discriminator's judgment result on the real data x under given condition c (the output is a scalar between 0 and 1, representing the probability that the sample is the real data); This represents the sample generated by the generator from noise z under given condition c; This indicates the discriminator's judgment result on the generated sample; It represents the mathematical expectation.

[0055] In addition, a cycle consistency constraint is introduced to ensure the reversibility of feature transformations: , in: This represents the loss of cycle consistency. This represents a generator that can be used to generate data from domain A to domain B. This represents a generator that generates data from domain B to domain A; x represents a data sample in domain A; y represents a data sample in domain B. express Norm (sum of absolute values); This represents the result of first converting sample y from domain B to domain A and then back to domain B; This represents the result of first converting sample x from domain A to domain B and then back to domain A.

[0056] The progressive training control unit 24 is used to control the training process to proceed in four stages: training each encoder individually, introducing adversarial learning, enabling cycle consistency constraints, and jointly optimizing all components.

[0057] Specifically, the progressive alignment training process includes four stages: the first stage (initialization stage) performs 10 rounds of independent pre-training for each encoder; the second stage (alignment initiation stage) performs 15 rounds of coarse alignment training with adversarial learning; the third stage (fine optimization stage) performs 20 rounds of fine alignment training with cycle consistency constraints enabled; and the fourth stage (global joint optimization stage) performs 40 rounds of joint training for all components.

[0058] During training, an adaptive learning rate adjustment mechanism is adopted. The initial learning rate is 0.001. When the ratio of the intra-domain distance metric to the inter-domain distance metric is greater than 0.8 times the preset threshold, the learning rate is multiplied by 0.8. When there is no improvement in performance for 5 consecutive rounds, the learning rate is multiplied by 0.5. The minimum learning rate is limited to 0.00001.

[0059] In addition, regarding the batch processing strategy, the initial batch size is set to 16, which is adaptively adjusted according to the GPU memory. Each batch contains an equal amount of data from three modalities, and an anchor sample priority sampling mechanism is adopted to ensure the stability of inter-domain mapping.

[0060] like Figure 4 As shown, the multimodal feature fusion module 3 includes a shared coding space construction unit 31, a feature importance weighting unit 32, a multi-scale feature fusion unit 33, and a topology verification unit 34.

[0061] The shared coding space construction unit 31 is used to design the coding network h2 as a shared mapping space for different modal features. In a preferred embodiment of the present invention, the structure of the coding network h2 is as follows: an input layer, a 128×128×m dimensional intermediate layer, a 64×64×2m dimensional intermediate layer, a 32×32×4m dimensional intermediate layer, a 16×16×8m dimensional intermediate layer, and a shared feature space output layer, where m is the number of input channels.

[0062] The dimension of the shared feature space is adaptively set according to the complexity of the diagnostic task, typically between 1024 and 4096. Orthogonal constraints are introduced into the shared space to ensure the independence between features of different modalities: , in: Indicates the orthogonal constraint loss; This represents the representation of the i-th modal feature after mapping through the coding network h2; This represents the representation of the j-th modal feature after mapping through the coding network h2; This represents the inner product (dot product) between two feature representations. Feature representation L2 norm (Euclidean length); Feature representation The L2 norm; This indicates summing over all distinct modal pairs.

[0063] Feature importance weighting unit 32 is used to calculate the importance weights of three modal features: EBUS ultrasound images, OCT tomography images, and DWI functional images. , and The weights are dynamically adjusted based on the contribution of each modality to the current diagnostic task.

[0064] In one embodiment of the present invention, modal weights can be calculated using the following formula: , in: This represents the importance weight of the i-th mode; This represents the importance score of the i-th mode; express The exponential function; Represents a set of three modes; This represents the sum of the importance scores for all modalities. Importance Score It can be calculated through attention mechanisms: , in: Let v represent the importance score of the i-th modality; v is the attention vector (learnable parameter); W is the weight matrix (learnable parameter); b is the bias term (learnable parameter). is the encoded representation of the i-th modality feature; tanh is the hyperbolic tangent activation function, which maps the input to the range [-1, 1]. This represents the transpose of vector v.

[0065] The multi-scale feature fusion unit 33 is used to perform feature fusion at three scales: global, regional, and local. It integrates the fusion results of different scales through skip connections to form a multi-resolution feature representation.

[0066] like Figure 9 As shown, the multi-scale feature fusion process includes five steps: global feature fusion, regional feature fusion, local feature fusion, feature integration, and feature selection. Global feature fusion targets features at 32×32 resolution and fuses overall semantic information; regional feature fusion targets features at 64×64 resolution and fuses local structural information; and local feature fusion targets features at 128×128 resolution and fuses microscopic texture information.

[0067] In a preferred embodiment of the present invention, multi-scale weighted fusion can be expressed as: , in: Indicated in scale The following fusion features; Indicates different scale levels (global, regional, local); Indicates the i-th mode in Features at a given scale; Indicates the corresponding weight; Represents the coding network; It represents a set of three modalities (EBUS ultrasound, OCT tomography, and DWI function).

[0068] Feature integration combines global, regional, and local features into a multi-resolution feature representation through skip connections: , in: This represents the integrated multi-resolution features; Concat represents the feature concatenation operation, which connects multiple features along the channel dimension. Represents the fusion features at the global scale; Indicates the fusion characteristics at the regional scale; This represents the fusion features at a local scale; Upsample represents the upsampling operation, used to adjust features of different resolutions to the same spatial size.

[0069] Feature selection, based on the information gain principle, eliminates redundancy in the integrated features. The information gain for each feature dimension can be calculated using the following formula: , in: Representing feature dimension Information gain for label Y; The entropy of label Y represents the uncertainty of label distribution; Representing a given feature The conditional entropy of label Y under given conditions measures the conditional entropy of the label Y under given conditions. The uncertainty of label distribution under certain circumstances.

[0070] entropy and conditional entropy Calculate using the following formulas respectively: , , in: This represents the probability of label y; Representation of features The probability of; This indicates that the eigenvalue is Given the condition, the conditional probability of label y is calculated. Based on the information gain value, feature dimensions with high information content are retained, while redundant dimensions are eliminated.

[0071] The topology verification unit 34 is used to evaluate the degree of preservation of the topology before and after fusion, and to trigger the topology optimization process when the structural similarity is lower than a preset threshold.

[0072] In one embodiment of the present invention, the structural similarity index can be calculated using the following formula: , in: A structural similarity index representing the relationship between original features and fused features; Indicates the original characteristics before fusion; Indicates the characteristics after fusion; express The mean; express The mean; express Standard deviation; express Standard deviation; express and Covariance between them; and It is a small constant to prevent the denominator from being zero, usually , Where L is the dynamic range of the eigenvalues. , .

[0073] When the SSIM value falls below a preset threshold (e.g., 0.75), a topology optimization process is triggered, which adjusts the fusion weights and connection strategies. The optimized fusion features are then verified by a unified discriminator (h3 encoding) to ensure information integrity.

[0074] like Figure 5 As shown, the diagnostic decision module 4 includes a Transformer processing unit 41, a pixel-level classification unit 42, a dual threshold judgment unit 43, and a dual-path cross-validation unit 44.

[0075] Transformer processing unit 41 is used to receive fused features and process the fused features through a Transformer network configured with 8 attention heads and 6 encoder layers. In a preferred embodiment of the present invention, the attention mechanism of the Transformer network can be expressed as follows: , Where: Attention(Q, K, V) represents the output of the attention mechanism; Q represents the query matrix, with dimensions [batch_size, seq_length, ... ]; K represents the key matrix, with dimensions [batch_size, seq_length, V represents the value matrix, with dimensions [batch_size, seq_length, d_v]. This represents a matrix multiplication of the query matrix and the transpose of the key matrix, with the result having dimensions [batch_size, seq_length, seq_length]. The dimension representing the key is used to normalize the scaling factor; softmax is the softmax normalization function applied to the last dimension to ensure that the sum of the attention weights is 1.

[0076] The multi-head attention mechanism can be represented as: , , Where: MultiHead(Q, K, V) represents the output of the multi-head attention mechanism; head_i represents the output of the i-th attention head; , , These are the linear projection matrices of the query, key, and value of the i-th attention head, with dimensions respectively. , , ,in It is the hidden layer dimension of the model; It is the output projection matrix, with dimensions of ,in It refers to the number of heads; Concat indicates a concatenation operation along the last dimension.

[0077] The pixel-level classification unit 42 is used to perform pixel-level classification on the output of the Transformer network, generating predicted values ​​ranging from 0 to 100. In one embodiment of the present invention, pixel-level classification can be achieved by the following formula: , Where: P(x, y) represents the predicted value at position (x, y), ranging from 0 to 100; f(x, y) represents the output feature of the Transformer network at position (x, y); σ represents the sigmoid activation function, defined as σ(z) = 1 / (1 + e^(-z)), ensuring that the output range is between [0, 1]; multiplying by 100 adjusts the range to [0, 100].

[0078] The dual-threshold judgment unit 43 is used to determine the category of each pixel based on the nodule detection threshold τ1 and the benign / malignant classification threshold τ2. In a preferred embodiment of the present invention, τ1 is set to 20 and τ2 is set to 40. Specifically, pixels with a predicted value greater than τ1 are identified as nodules, and nodules with a predicted value greater than τ2 are classified as malignant nodules.

[0079] These thresholds were determined based on extensive clinical data analysis. τ1=20 was determined based on the contrast difference between nodules and normal tissue, a value that can control the false positive rate while ensuring the detection rate. τ2=40 was determined based on the differences between benign and malignant nodules in multimodal fusion features, a value that can better balance sensitivity and specificity.

[0080] The dual-path cross-validation unit 44 is used to cross-validate the results of two independent diagnostic paths, thereby improving the reliability of the diagnosis. For example... Figure 10 As shown, the two diagnostic pathways are: the first pathway, which combines enhanced OCT images and DWI images with the original EBUS images, and the second pathway, which combines the original EBUS images and enhanced OCT images with enhanced DWI images.

[0081] Furthermore, when the results of the two diagnostic paths are inconsistent, the system will make a weighted decision based on the confidence level of the predicted values: , in: This indicates the final diagnosis result; and These represent the diagnostic results for the two paths (which can be binary values ​​or probability values). and The corresponding confidence weight can be calculated from the difference between the predicted value and the threshold. , in: This represents the confidence weight of the i-th path; This represents the predicted value for the i-th path; The threshold value represents the corresponding threshold (nodule detection threshold T1 or benign / malignant classification threshold T2); exp represents the natural exponential function. This represents the absolute value of the difference between the predicted value and the threshold; the larger the difference, the higher the confidence level.

[0082] At the same time, the system will also perform regional connectivity analysis on the diagnostic results at different locations, eliminate isolated abnormal pixels, and generate a structured diagnostic report that includes nodule location, size, shape, and benign / malignant classification.

[0083] like Figure 7 As shown, the data preprocessing module 5 is connected to the domain manifold embedding module 1 and is used to preprocess EBUS ultrasound images, OCT tomographic images and DWI functional images to generate standardized data formats.

[0084] Specifically, the data preprocessing module 5 is used to perform temporal denoising and median filtering on the EBUS ultrasound images. The median filtering window size is 3 pixels. The processed EBUS ultrasound images are then downsampled at multiple scales to generate multi-scale EBUS ultrasound images with resolutions of 128×128, 64×64, and 32×32.

[0085] For OCT tomographic images, data preprocessing module 5 extracts the RGB three-channel matrix based on DICOM information and processes each channel independently into a standardized 128×128 image.

[0086] For DWI functional images, the data preprocessing module 5 extracts the sequence images based on DICOM information and processes them into uniform 128×128 images.

[0087] Finally, the data preprocessing module 5 converts the processed EBUS ultrasound images, OCT tomographic images, and DWI functional images into NumPy file format, which is then used as input to the domain manifold embedding module 1.

[0088] In a preferred embodiment of the present invention, the processing path of EBUS ultrasound images is as follows: raw EBUS data → temporal domain denoising (window = 3px) → median filtering → multi-scale downsampling (128×128 → 64×64 → 32×32) → nodule / non-nodule classification → NumPy format storage.

[0089] The processing path for OCT tomographic images is as follows: raw OCT data → DICOM information extraction → RGB three-channel separation → independent processing of each channel → standardization (128×128) → channel-specific storage in NumPy format.

[0090] The processing path for DWI functional images is as follows: raw DWI sequence → DICOM sequence information parsing → target sequence selection → sequence extraction → standardization (128×128) → storage in NumPy format.

[0091] Data preprocessing is a fundamental step in the entire system. Effective preprocessing can significantly improve the results of subsequent domain embedding and domain alignment.

[0092] like Figure 6 As shown, the non-invasive biopsy multimodal image fusion method based on the above system includes the following steps: 1. Receives EBUS ultrasound images, OCT tomographic images, and DWI functional images as input; 2. Preprocess EBUS ultrasound images, OCT tomographic images, and DWI functional images to generate standardized multi-scale image data; 3. Construct a domain manifold embedding structure, establish the topological correspondence between EBUS ultrasound images, OCT tomographic images and DWI functional images, and generate a domain mapping table; 4. Based on the domain mapping table, domain alignment between EBUS ultrasound images, OCT tomography images, and DWI functional images is achieved through a hierarchical adaptive domain alignment network. 5. Map the domain-aligned EBUS ultrasound image features, OCT tomographic image features, and DWI functional image features to the shared coding space, and perform multi-scale feature fusion to generate fused features that preserve the topological structure. 6. Process the fused features using a Transformer network to generate pixel-level classification maps; 7. Based on the preset nodule detection threshold and benign / malignant classification threshold, the pixel-level classification mapping is judged to obtain the nodule detection result and the benign / malignant classification result; 8. Output fused images and diagnostic analysis results.

[0093] In practice, the preprocessing in step 2 includes performing time-domain noise reduction and median filtering on EBUS ultrasound images, extracting the RGB three-channel matrix from OCT tomographic images based on DICOM information, extracting sequential images from DWI functional images based on DICOM information, and converting all processed images into a standardized data format.

[0094] In step 3, the construction of the domain manifold embedding structure includes four sub-steps: domain mapping construction, local neighborhood preservation, inter-domain mapping bridging, and domain embedding verification.

[0095] In step 4, the hierarchical adaptive domain alignment network achieves domain alignment through three-level feature extraction, autoencoder discriminator encoding, conditional adversarial optimization, and progressive training.

[0096] In step 5, multimodal feature fusion includes four sub-steps: shared coding space construction, feature importance weighting, multi-scale feature fusion, and topological structure verification.

[0097] In steps 6 and 7, diagnostic decisions are made through Transformer network processing, pixel-level classification, dual threshold judgment, and dual-path cross-validation.

[0098] Preferably, in step 7, the nodule detection threshold τ1 is set to 20, and the benign / malignant classification threshold τ2 is set to 40.

[0099] Furthermore, the dual-path cross-validation includes two independent diagnostic pathways: a first pathway combining enhanced OCT and DWI images with the original EBUS image, and a second pathway combining the original EBUS image with enhanced OCT and DWI images. When the results of the two pathways are inconsistent, a weighted decision is made based on the confidence level of the predicted values.

[0100] In summary, the non-invasive biopsy multimodal image fusion system and method provided by this invention effectively solves the domain difference problem between different modalities of medical images by innovatively designing domain manifold embedding and representation space alignment techniques, achieving efficient and accurate multimodal image fusion. This invention combines advanced deep learning technology to provide a non-invasive or minimally invasive biopsy solution that can significantly improve the accuracy and reliability of diagnosis, and has significant clinical application value.

[0101] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A non-invasive biopsy multimodal image fusion system, characterized in that, include: Domain manifold embedding module, used for: It receives EBUS ultrasound images, OCT tomographic images, and DWI functional images as input; Construct a three-domain data structure that includes the EBUS ultrasound dataset, the OCT tomographic dataset, and the DWI functional dataset; Generate a domain mapping table, which is used to establish the topological correspondence between the EBUS ultrasound images, OCT tomographic images and DWI functional images; The hierarchical adaptive domain alignment network module, connected to the domain manifold embedding module, is used for: Receive the three-domain data structure and the domain mapping table; Multiscale features of the EBUS ultrasound image, OCT tomography image and DWI functional image are extracted using a three-level feature extraction structure. The multi-scale features are encoded using an autoencoder discriminator network to obtain a feature vector; Based on the feature vector, conditional adversarial optimization is performed to achieve domain alignment between the EBUS ultrasound images, OCT tomographic images, and DWI functional images. The multimodal feature fusion module, connected to the hierarchical adaptive domain alignment network module, is used for: Receive domain-aligned EBUS ultrasound image features, OCT tomographic image features, and DWI functional image features. A shared coding space is constructed, and the EBUS ultrasound image features, OCT tomographic image features, and DWI functional image features are mapped to the shared coding space. Calculate the importance weights of modal features, and perform multi-scale feature fusion based on the weights; Through topology verification, fusion features that preserve the topology are generated; The diagnostic decision module, connected to the multimodal feature fusion module, is used for: Receive the fusion feature; The fused features are processed by a Transformer network to generate pixel-level classification maps; The pixel-level classification mapping is judged based on a preset threshold to obtain nodule detection results and benign / malignant classification results; Output fused images and diagnostic analysis results.

2. The system according to claim 1, characterized in that, The domain manifold embedding module includes: A domain mapping construction unit is used to construct a four-dimensional domain mapping structure, which includes an EBUS ultrasound dataset U, an OCT tomography dataset T, a DWI functional dataset F, and a multimodal mapping reference dataset M. The local neighborhood preservation unit is used to define a set of k nearest neighbors for each modal data point and preserve the relative positional relationships of the points within the set of k nearest neighbors in the embedding space. The inter-domain mapping bridging unit is used to establish a bidirectional mapping relationship between the EBUS ultrasound dataset U, the OCT tomographic dataset T, and the DWI functional dataset F, and selects a set of anchor points as the reference points for cross-domain correspondence based on the similarity of image features. The domain embedding verification unit is used to calculate the intra-domain distance metric and the inter-domain distance metric, and to determine the validity of the domain embedding structure based on the ratio of the intra-domain distance metric to the inter-domain distance metric.

3. The system according to claim 1, characterized in that, The hierarchical adaptive domain alignment network module includes: The three-level feature extraction unit is used to extract low-level features, intermediate features, and high-level features from EBUS ultrasound images, OCT tomographic images, and DWI functional images, respectively. The resolution of the low-level features is 128×128, the resolution of the intermediate features is 64×64, and the resolution of the high-level features is 32×32. An autoencoder discriminator unit is used to encode and decode the features of each domain. The encoding part maps the input features to an h1-dimensional feature vector, and the decoding part reconstructs the h1-dimensional feature vector back to the original input size. The conditional adversarial optimization unit is used to construct a conditional adversarial learning framework based on the domain identifier vector, and to distinguish between real domain features and transformed domain features. The progressive training control unit controls the training process in four stages: training each encoder individually, introducing adversarial learning, enabling cycle consistency constraints, and jointly optimizing all components.

4. The system according to claim 1, characterized in that, The multimodal feature fusion module includes: A shared coding space construction unit is used to design a coding network h2 as a shared mapping space for different modal features. The structure of the coding network h2 consists of an input layer, a 128×128×m-dimensional intermediate layer, a 64×64×2m-dimensional intermediate layer, a 32×32×4m-dimensional intermediate layer, a 16×16×8m-dimensional intermediate layer, and a shared feature space output layer, where m is the number of input channels. The feature importance weighting unit is used to calculate the importance weights w_U, w_T, and w_F of three modal features: EBUS ultrasound images, OCT tomography images, and DWI functional images, and dynamically adjusts the weights based on the contribution of each modality to the current diagnostic task. The multi-scale feature fusion unit is used to perform feature fusion at three scales: global, regional, and local. It integrates the fusion results of different scales through skip connections to form a multi-resolution feature representation. The topology verification unit is used to evaluate the degree of preservation of the topology before and after fusion, and to trigger the topology optimization process when the structural similarity is lower than a preset threshold.

5. The system according to claim 1, characterized in that, The diagnostic decision module includes: A Transformer processing unit is used to receive fused features and process the fused features through a Transformer network configured with 8 attention heads and 6 encoder layers; The pixel-level classification unit is used to perform pixel-level classification on the output of the Transformer network and generate predicted values ​​ranging from 0 to 100. The dual threshold judgment unit is used to determine the category of each pixel based on the nodule detection threshold τ1=20 and the benign / malignant classification threshold τ2=40. Pixels with a predicted value greater than τ1 are identified as nodules, and nodules with a predicted value greater than τ2 are classified as malignant nodules. The dual-path cross-validation unit is used to improve the reliability of diagnosis by cross-validating the results of two independent diagnostic paths. The two diagnostic paths are: a first path combining enhanced OCT images and DWI images with original EBUS images, and a second path combining original EBUS images and enhanced OCT images with enhanced DWI images.

6. The system according to claim 1, characterized in that, It also includes a data preprocessing module, connected to the domain manifold embedding module, for: Temporal denoising and median filtering are performed on the EBUS ultrasound images, wherein the window size of the median filtering is 3 pixels, and the processed EBUS ultrasound images are downsampled at multiple scales to generate multi-scale EBUS ultrasound images with resolutions of 128×128, 64×64 and 32×32. The RGB three-channel matrix was extracted from the OCT tomographic images based on DICOM information, and each channel was independently processed into a standardized 128×128 image. Extract sequential images from DWI images based on DICOM information and process them into uniform 128×128 images; The processed EBUS ultrasound images, OCT tomographic images, and DWI functional images are converted into NumPy file format and used as input to the domain manifold embedding module.

7. The system according to claim 1, characterized in that, The autoencoder discriminator network of the hierarchical adaptive domain alignment network module adopts the following staged training strategy: During the initialization phase, 10 rounds of independent pre-training are performed for each encoder. During the alignment initiation phase, 15 rounds of coarse alignment training were performed to introduce adversarial learning. In the fine-tuning phase, 20 rounds of fine-tuning training with cycle consistency constraints are performed; During the global joint optimization phase, 40 rounds of joint training of all components are performed. During training, an adaptive learning rate adjustment mechanism is used. The initial learning rate is 0.

001. When the ratio of the intra-domain distance metric to the inter-domain distance metric is greater than 0.8 times the preset threshold, the learning rate is multiplied by 0.

8. When there is no improvement in performance for 5 consecutive rounds, the learning rate is multiplied by 0.

5. The minimum learning rate is limited to 0.00001.

8. The system according to claim 4, characterized in that, The fusion process of the multi-scale feature fusion unit of the multimodal feature fusion module includes: Global feature fusion: For features with a resolution of 32×32, the overall semantic information is fused. Regional feature fusion: For features with a resolution of 64×64, local structural information is fused. Local feature fusion: For features at 128×128 resolution, micro-texture information is fused. Feature integration integrates global features, regional features, and local features into a multi-resolution feature representation through skip connections; Feature selection involves eliminating redundancy in the integrated features based on the information gain principle.

9. The system according to claim 5, characterized in that, The dual-path cross-validation unit of the diagnostic decision module is also used for: When the results of two diagnostic paths are inconsistent, a weighted decision is made based on the confidence level of the predicted values. Regional connectivity analysis is performed on diagnostic results from different locations to eliminate isolated abnormal pixels; Generate a structured diagnostic report that includes nodule location, size, shape, and benign / malignant classification.

10. A method based on the non-invasive biopsy multimodal image fusion system according to any one of claims 1-9, characterized in that, Includes the following steps: It receives EBUS ultrasound images, OCT tomographic images, and DWI functional images as input; The EBUS ultrasound images, OCT tomographic images, and DWI functional images are preprocessed to generate standardized multi-scale image data. Construct a domain manifold embedding structure, establish the topological correspondence between the EBUS ultrasound images, OCT tomographic images and DWI functional images, and generate a domain mapping table. Based on the domain mapping table, domain alignment between the EBUS ultrasound images, OCT tomography images and DWI functional images is achieved through a hierarchical adaptive domain alignment network. The domain-aligned EBUS ultrasound image features, OCT tomographic image features, and DWI functional image features are mapped to a shared coding space, and multi-scale feature fusion is performed to generate fused features that preserve the topological structure. The fused features are processed by a Transformer network to generate pixel-level classification maps; Based on preset nodule detection thresholds and benign / malignant classification thresholds, the pixel-level classification mapping is judged to obtain nodule detection results and benign / malignant classification results; Output fused images and diagnostic analysis results.

Images