A virtual modal imaging calculation method based on multi-level consistency

Abstract

The present invention belongs to the technical field of image processing, and relates to a virtual modality imaging calculation method based on multi-level consistency for medical images, including: obtaining paired data of original B-ultrasound images and real elastography ultrasound images; extracting feature maps through feature encoding; decoding the extracted feature maps to generate virtual elastography ultrasound images; in the training stage, through multi-level consistency constraints, supervised learning is performed on the virtual elastography ultrasound images to optimize the image modality conversion model; in the inference stage, the trained virtual modality generation model is applied to convert new B-ultrasound images into virtual elastography ultrasound images and perform nodule diagnosis; the present invention can, when the hardware does not support multi-modal medical image imaging, realize the acquisition of virtual modality medical images through neural network calculation, and can realize the imaging of elastograms without changing the ultrasound probe hardware, providing more sufficient imaging information for clinical diagnosis.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a virtual modal imaging calculation method based on multi-level consistency. Background Art

[0002] With the development of modern medicine, in order to non-invasively observe the actual situation inside the patient's body, various medical imaging technologies are widely used in clinical practice. As a non-invasive and low-cost soft tissue imaging method, ultrasound imaging is the most commonly used examination method for parts such as the thyroid and breast. Currently, elastography is developing rapidly. It can more accurately evaluate the biomechanical properties of soft tissues. The multi-modal diagnostic method based on B-ultrasound and elasticity has gradually received attention in the thyroid and breast imaging diagnosis guidelines and is playing an increasingly important role in clinical diagnosis.

[0003] However, compared with the most commonly used B-ultrasound imaging, elastography is difficult to obtain in some cases. First, elastography relies on high-end ultrasound equipment, which makes it difficult to apply the multi-modal ultrasound diagnostic method to the portable device of B-ultrasound single modality. Second, elastography is difficult to perform when the patient cannot tolerate a large pressure at the lesion site. Third, elastography has high requirements for techniques and experience, which limits its popularization in areas with underdeveloped medical resources.

[0004] Moreover, most of the existing image modality conversion technical solutions are based on the encoding-decoding network or diffusion model to establish the mapping relationship between the original image and the target image, and then use the GAN loss to narrow the distance between the color and texture distributions of the generated image and the target image, and establish pixel-level supervision based on the L2 loss. These methods show good performance in some image modality conversion tasks with similar feature distributions, such as the modality conversion from CT to MRI, or the modality conversion between different MRI sequences. However, in the BUS-to-EUS virtual imaging task, they show the following deficiencies:

[0005] 1. The GAN loss focuses on the distribution difference between the generated image and the real image, but ignores the measurement of the consistency and difference between samples, which makes it difficult for the model to learn the key feature differences of the elastograms corresponding to different B-ultrasound images and difficult to provide effective information for subsequent diagnostic tasks.

[0006] 2. The pixel-level loss is limited by the receptive field and is difficult to effectively model the local structure information of EUS, which is crucial for achieving accurate BUS-to-EUS virtual imaging.

[0007] Therefore, how to promote multi-modal ultrasound diagnosis to portable devices and underdeveloped medical areas, and make the virtual modal images closer to the real images, so as to provide more accurate information for diagnosis has always been the concern of medical workers. Summary of the Invention

[0008] To solve the above technical problems, the present invention is specifically implemented through the following technical solutions.

[0009] A virtual modal imaging calculation method based on multi-level consistency, the following steps:

[0010] Step 1, obtain paired data of the original B-ultrasound image BUS and the real elastography ultrasound image R-EUS;

[0011] Step 2, use an encoder to encode the original B-ultrasound image BUS and extract a feature map;

[0012] Step 3, use a decoder to decode the feature map extracted in Step 2 to generate a virtual elastography ultrasound image G-EUS;

[0013] Step 4, in the training stage, through sample-level, region-level, pixel-level, and domain-level consistency constraints, perform supervised learning on the virtual elastography ultrasound image G-EUS generated in Step 3 to optimize the image modality conversion model;

[0014] Step 5, in the inference stage, apply the virtual modal generation model trained in Step 4 to convert a new original B-ultrasound image BUS into a virtual elastography ultrasound image G-EUS, and combine the original B-ultrasound image BUS and the virtual elastography ultrasound image G-EUS for nodule diagnosis.

[0015] Preferably, in Step 2, the encoder includes: an encoder using a fusion convolutional neural network (CNN) and a Transformer architecture; in Step 3, the decoder includes: using a transposed convolutional decoder.

[0016] More preferably, Step 1 includes the following sub-steps:

[0017] Step 1-1, obtain paired data of the original B-ultrasound image BUS and the real elastography ultrasound image R-EUS from high-end equipment;

[0018] Step 1-2, intercept the lesion areas of the original B-ultrasound image BUS and the real elastography ultrasound image R-EUS;

[0019] Step 1-3, organize the obtained samples into a training set and a test set.

[0020] Preferably, in step 2, an encoder integrating a convolutional neural network (CNN) and a Transformer architecture is used to encode the original B-mode ultrasound image (BUS) to extract a feature map, expressed as:

[0021] z B = G Enc (x B ) (1)

[0022] In formula (1), z B represents the extracted feature map, G Enc represents the encoder, and x B represents the original B-mode ultrasound image (BUS).

[0023] Preferably, in step 3, a transposed convolutional decoder is used to decode the extracted feature map to generate a virtual elastography ultrasound image (G-EUS), expressed as:

[0024]

[0025] In formula (2), represents the virtual elastography ultrasound image (G-EUS), and G Dec represents the decoder.

[0026] Preferably, step 4 includes the following sub-steps:

[0027] Step 4-1: Sample-level consistency constraint, which is achieved by contrastive learning between samples to implement the sample-level consistency constraint;

[0028] Step 4-2: Region-level consistency constraint, which is achieved by performing region-level comparison between the generated elastogram and the real elastogram x E to implement the region-level consistency constraint;

[0029] Step 4-3: Pixel-level consistency constraint, which is achieved by calculating the pixel-level L1 loss between the real elastogram x E and the generated elastogram to implement the pixel-level constraint; the calculation is as follows:

[0030]

[0031] In formula (8), w and h are the width and height of the image;

[0032] Step 4-4, Domain-level consistency constraint: Construct a domain discriminator D composed of a convolutional neural network and a fully connected layer classifier. Use the domain discriminator to distinguish between the generated elastogram and the real elastogram. If the domain discriminator can easily distinguish between the generated elastogram and the real elastogram, it indicates that the domain-level consistency between the generated image and the real image is weak; otherwise, it indicates that the domain-level consistency is strong. In this way, a domain-level consistency constraint is established, and the calculation is as follows:

[0033]

[0034] In Equation (9), D is the domain discriminator, and G is the generation network;

[0035] Step 4-5, Overall optimization objective: The overall optimization objective of the virtual imaging model training is obtained by adaptively weighted summation of the above four levels of constraints. Among them, the adaptive weighting parameter is calculated driven by the causal chain of nodule diagnosis, and the calculation is as follows:

[0036]

[0037] In Equation (10), y c is the probability that the sample belongs to the c-th class, C is the number of possible classes, and f Diag is the diagnostic model, represents the loss function with weight parameters w1, w2, and w3.

[0038] The final overall loss is calculated as follows:

[0039]

[0040] In Equation (11) and are the weighting coefficients.

[0041] More preferably, the said Step 4-1 includes the following sub-steps:

[0042] Step 4-1-1, Construct an encoder with the same structure as the encoder in the generation network for extracting the feature map z of the real elastographic ultrasound image R-EUS E , and through global pooling operation, convert the feature maps of the original B-mode ultrasound image BUS and the real elastographic ultrasound image R-EUS into query vector q and key vector k respectively;

[0043]

[0044] Step 4-1-2: Calculate the distance between the features corresponding to the original B-mode ultrasound image BUS samples and their corresponding real elastography ultrasound image R-EUS samples within a mini-batch with a sample size of N, and minimize the distance between the feature vectors q and k corresponding to the paired samples + , and maximize the distance between the feature vectors q and k other than k + except i . Taking this as the optimization objective, establish a sample set consistency constraint, and this optimization objective can be expressed as:

[0045]

[0046] In Equation (4), d k is the dimension of vector k, and μ is used for scale transformation to stabilize gradient update;

[0047] Step 4-1-3: During the training process, when updating the parameters, the parameters of the feature encoder G Enc and are updated by the momentum update method. When updating, the parameters θ t and θ t E are calculated as follows:

[0048]

[0049]

[0050] In Equations (5) and (6), η t represents the learning rate, set to 0.001, L represents the loss function, and m is the momentum coefficient, set to 0.99.

[0051] The beneficial effects of the present invention are:

[0052] 1. The virtual modality imaging technology proposed by the present invention can, when the hardware does not support multi-modal medical image imaging, achieve the imaging of virtual modality images through neural network calculation. For example, a portable ultrasound device only supports the B-mode modality, and through the present invention, elastogram imaging can be achieved without changing the ultrasound probe hardware, providing more sufficient information for clinical diagnosis.

[0053] 2. The present invention further proposes a virtual modality generation method based on multi-level consistency constraints. Compared with existing medical image modality conversion methods, firstly, by establishing sample-level constraints, the association between samples is established, enhancing the acquisition of sample-specific features, making the generated virtual modality images more conducive to diagnosis. Secondly, by establishing regional-level constraints, compared with pixel-level losses, the extraction of texture and color features at the regional scale is enhanced, making the generated virtual modality images more accurate.

[0054] 3. Compared with the current cutting-edge technologies in the academic field, the virtual modal images generated by the method of the present invention are closer to real images and can provide more accurate information for diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the process and system of a virtual modal imaging calculation method based on multi-level consistency of the present invention;

[0056] Figure 2 It is a diagram of the virtual modal imaging method of the present invention;

[0057] Figure 3 It is a virtual elasticity comparison diagram generated by the present invention and other methods. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] Next, the relevant technologies in the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0059] This embodiment is a virtual modal imaging calculation method based on multi-level consistency. As Figure 1 shown, it includes the following steps:

[0060] 1. Obtain paired data of the original B-ultrasound image (BUS) and the real elastography ultrasound image (R-EUS).

[0061] 2. Use an encoder that combines a convolutional neural network (CNN) and a Transformer architecture to encode the BUS and extract the feature map.

[0062] 3. Use a transposed convolutional decoder to decode the extracted feature map to generate a virtual elastography ultrasound image (G-EUS).

[0063] 4. In the training stage, through consistency constraints at the sample level, region level, pixel level, and domain level, supervise and learn the generated G-EUS to optimize the image modality conversion model.

[0064] 5. In the inference stage, apply the trained virtual modal generation model to convert a new BUS image into a G-EUS image, and combine the BUS and G-EUS for nodule diagnosis.

[0065] A more specific virtual modal imaging calculation method based on multi-level consistency is as Figure 2 shown and is described as follows:

[0066] 1. Data acquisition

[0067] 1.1 Obtain paired data of the original B-ultrasound image (BUS) and the real elastography ultrasound image (R-EUS) from high-end equipment

[0068] 1.2 Crop the lesion regions of the B-ultrasound image and the elastogram

[0069] 1.3 Organize the obtained samples into a training set and a test set.

[0070] 2. Virtual imaging model training

[0071] 2.1 As shown in Figure 2 (b), use the encoder G that fuses the convolutional neural network CNN and the Transformer architecture Enc to encode the B-ultrasound image x B and extract the feature map z B , expressed as

[0072] z B = G Enc (x B ) (1)

[0073] 2.2 Use the transposed convolutional decoder G Dec to decode the extracted feature map and generate a virtual elastography ultrasound image

[0074]

[0075] 2.3 In the training stage of the generation network, perform supervised learning on the generated virtual elastography ultrasound image through the following steps:

[0076] 1) Sample-level consistency constraint

[0077] As shown in Figure 2 (c), achieve sample-level consistency constraint through contrastive learning between samples. First, construct an encoder with the same structure as the encoder in the generation network in Figure 2 (b) to extract the feature z of the real elastogram . Through global pooling operations, convert the feature maps of BUS and R-EUS into query vector q and key vector k respectively. The calculation of z E is expressed as: E

[0078]

[0079] Then, within a mini-batch with a sample size of N, calculate the distance between the features corresponding to the BUS sample and its corresponding R-EUS sample, and minimize the pairwise samples (q and k + ​The distance between them is maximized for the distance between non-paired samples (q and k except + k other than i ), and with this as the optimization objective, a consistency constraint for the sample set is established. This optimization objective can be expressed as:

[0080]

[0081] where d k is the dimension of vector k, and μ is used for scale transformation to stabilize gradient update.

[0082] In addition, during the training process, when the parameters are updated, the parameters of the feature encoder G Enc and are updated by the momentum update method. For example, when the parameters are updated at the t-th step, the parameters θ t and θ t E are calculated as follows:

[0083]

[0084]

[0085] where η t represents the learning rate, set to 0.001, L represents the loss function, and m is the momentum coefficient, set to 0.99.

[0086] 2) Regional consistency constraint

[0087] As shown in Figure 2 (d), regional consistency constraint is achieved by making a regional comparison between the generated elastic map and the real elastic map x E . First, the real elastic map x E is regionally partitioned through pixel clustering, and the calculation is:

[0088] P = SLIC(x E ) (16)

[0089]

[0090] where P represents the region segmentation function, SLIC represents the simple linear iterative pixel clustering method, represents the set of each sub-region obtained by the real elastic map x E through the region segmentation function.

[0091] Then, the generated elastic map is regionally partitioned in the same regional partitioning manner, and the calculation is:

[0092]

[0093] Each region is flattened and resampled to a uniform size ( Norm ). will flatten and unify the size of x E and The images of each region are input into the multi-layer perceptron (MLP) to obtain the feature vector corresponding to each region, which is expressed as:

[0094] v=MLP(Norm(p R )) (14)

[0095] u=MLP(Norm(p G )) (15)

[0096] Among them, x E The corresponding feature vector of each region is represented by v. The corresponding feature vector of each region is represented by u. By comparing u and v, we can get closer to x. E and The distance between the images of each corresponding region makes the generated image have local structural information similar to the real image, which can be expressed as:

[0097]

[0098] 3) Pixel-level consistency constraints

[0099] like Figure 2 As shown in (e), by calculating the true elasticity graph x E and generate elasticity graphs The pixel-level L1 loss between and is used to implement pixel-level constraints. The calculation is as follows:

[0100]

[0101] Where w and h are the width and height of the image.

[0102] 4) Domain-level consistency constraints

[0103] like Figure 2 As shown in (e), a domain discriminator D consisting of a convolutional neural network and a fully connected layer classifier is constructed. The generated elasticity map and the real elasticity map are discriminated by the domain discriminator. If the domain discriminator can easily discriminate the generated elasticity map and the real elasticity map, it means that the domain-level consistency between the generated image and the real image is weak. Otherwise, it means that the domain-level consistency is strong. In this way, the domain-level consistency constraint is established, which is calculated as follows:

[0104]

[0105] Where D is the domain discriminator and G is the generator network.

[0106] 5) Overall optimization objective

[0107] The overall optimization objective of virtual imaging model training is obtained by adaptively weighted summation of the above four levels of constraints. Among them, the adaptive weighting parameters are driven and calculated by the causal chain of nodule diagnosis, and the calculation is as follows:

[0108]

[0109] Among them, y c is the probability that the sample belongs to the c-th category, C is the number of possible categories, and f Diag is the diagnostic model. represents the loss function with weight parameters w1, w2, and w3.

[0110] The final overall loss is calculated as follows:

[0111]

[0112] Among them, and are the weighting coefficients.

[0113] 3. Deployment and diagnostic application of the virtual imaging model

[0114] The trained virtual imaging model is transplanted to a mobile device that can only obtain B-ultrasound images. A virtual elastogram is generated through the input B-ultrasound image, and the B-ultrasound image and the generated elastogram are jointly input into the diagnostic network to realize the diagnosis of the lesion.

[0115] The medical image virtual modality imaging method of this embodiment: Learn the mapping relationship between modalities on high-end multi-modal medical imaging equipment, train the virtual modality imaging model, and deploy the trained virtual modality imaging model to a portable device to realize the generation of images of high-cost modalities through images of low-cost modalities (for example: a mobile ultrasound device can only obtain B-ultrasound images and cannot obtain elastograms, while our method can realize the generation of virtual elastograms).

[0116] The medical image virtual modality imaging method based on multi-level consistency of this embodiment: Guide the model to generate more accurate virtual modality images through multi-level consistency constraints. The multi-level consistency constraints include sample level, region level, pixel level, and domain level. Among them, the constraints at the pixel level and domain level can be implemented by existing methods, while the constraints at the sample level and region level are the key technical points protected by this patent.

[0117] Sample-level constraints for medical image virtual modality imaging in this embodiment: By performing contrastive learning between samples, sample-level consistency constraints are achieved, associations between samples are established, the acquisition of sample-specific features is enhanced, and the generated virtual modality images are more conducive to diagnosis.

[0118] Region-level constraints for medical image virtual modality imaging in this embodiment: By comparing the generated elastogram with the real elastogram x E at the region level, region-level consistency constraints are achieved. Compared with pixel-level loss, the extraction of texture and color features at the region scale is enhanced, and the generated virtual modality images are more accurate.

[0119] In summary, the virtual modality imaging technology proposed by the present invention can achieve virtual modality imaging through neural network calculation when the hardware does not support multi-modal medical image imaging. Through the present invention, elastogram imaging can be achieved without changing the ultrasound probe hardware, providing more sufficient information for diagnosis.

[0120] It should be emphasized that: The above are only preferred embodiments of the present invention, and do not impose any form of limitation on the present invention. Any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A virtual modal imaging calculation method based on multi-level consistency, characterized in that, The following steps are involved: Step 1: Acquire paired data of an original B-ultrasound image BUS and a true elastic ultrasound image R-EUS; Step 2: Encode the original B ultrasound image BUS using an encoder to extract a feature map; Step 3: Decode the feature map extracted in step 2 using a decoder to generate a virtual elastic ultrasound image G-EUS; Step 4: During the training phase, supervised learning is performed on the virtual elastic ultrasound image (G-EUS) generated in step 3 through consistency constraints at the sample level, region level, pixel level, and domain level to optimize the image modality conversion model. Step 5: In the inference phase, the virtual modality generation model trained in step 4 is applied to convert the new original B-ultrasound image BUS into a virtual elastic ultrasound image G-EUS, and the original B-ultrasound image BUS and the virtual elastic ultrasound image G-EUS are combined for nodule diagnosis; In step 2, the encoder includes: an encoder integrating a convolutional neural network and a Transformer architecture; in step 3, the decoder includes: a decoder constructed based on deconvolution; Described step 4 comprises the following sub-steps: Step 4-1: Sample-level consistency constraint, which is achieved through comparative learning between samples; Step 4-1-1: Construct an encoder with the same structure as the encoder in the generation network to extract the feature z of the real elastography ultrasound image R-EUS E , and through global pooling operations, convert the feature maps of the original B-mode ultrasound image BUS and the real elastography ultrasound image R-EUS into query vector q and key vector k respectively; Step 4-1-2: Calculate the distance between the features corresponding to the original B-mode ultrasound image BUS samples and their corresponding real elastography ultrasound image R-EUS samples within a mini-batch with a sample size of N, and minimize the distance between the feature vectors q and k corresponding to the paired samples + , and maximize the distance between the feature vectors q and k other than k corresponding to the unpaired samples + other than k i , and take this as the optimization objective to establish a sample-level consistency constraint. This optimization objective can be expressed as: In Equation (4), d k is the dimension of the vector k, and μ is used for scale transformation to stably update the gradient; Step 4-1-3: During the training process, when the parameters are updated, the feature encoder G Enc and parameters are updated by the momentum update method. When updating, the parameter θ t and are calculated as follows: In formulas (5) and (6), η t represents the learning rate, which is set to 0.001, L represents the loss function, and m is the momentum coefficient, which is set to 0.99; Step 4-2, Regional consistency constraint, by comparing the generated elastic graph with the real elastic graph x E to perform regional comparison and achieve regional consistency constraint; Step 4-2-1: For the real elastic graph x E Perform region division through pixel clustering, and the calculation is as follows: P = SLIC(x E ) (16) Wherein, P represents a region segmentation function, and SLIC represents a simple linear iterative pixel clustering method. represents the true elasticity map x E a set of sub-regions obtained through the region segmentation function; Step 4-2-2: Divide the generated elasticity diagram according to the same regional division method and perform regional division. The calculation is as follows: Flatten each region and resample it to a unified size Norm; for the flattened and uniformly sized x E and input the regional images of each into the multi-layer perceptron MLP, and respectively obtain the feature vectors corresponding to each region as: v = MLP(Norm(p R )) (14) u = MLP(Norm(p G )) (15) Among them, x E The feature vector of each corresponding region is represented by v, The feature vector of each corresponding region is represented by u; By comparing u and v, the distance between x E and the images of each corresponding region is reduced, so that the generated image has local structural information similar to that of the real image, expressed as:

2. The virtual modal imaging calculation method based on multi-level consistency according to claim 1, wherein, Step 1 The following steps are included: Step 1-1, obtaining paired data of original B-ultrasound images (BUS) and real elastic ultrasound images (R-EUS) from a high-end multimodal device; Step 1-2, intercepting the lesion area of the original B-ultrasound image BUS and the real elastic ultrasound image R-EUS; Steps 1-3: Organize the obtained samples into training sets and test sets.

3. A virtual modal imaging calculation method based on multi-level consistency according to claim 1, characterized in that In step 2, the encoder integrating CNN and Transformer architecture is used to encode the original B ultrasound image BUS and extract the feature map, which is expressed as: z B = G Enc (x B ) (1) In formula (1), z B represents the extracted feature map, G Enc represents the encoder, and x B represents the original B-mode ultrasound image BUS.

4. A virtual modal imaging calculation method based on multi-level consistency according to claim 3, characterized in that In step 3, the extracted feature map is decoded using a deconvolution decoder to generate a virtual elastic ultrasound image G-EUS, which is expressed as: In formula (2), represents the generated elastic graph, G Dec represents the decoder.

5. A virtual modal imaging calculation method based on multi-level consistency according to claim 1, characterized in that The step 4 comprises: Step 4-3, Pixel-level consistency constraint, by calculating the ground truth elasticity map x E and the generated elasticity map to achieve pixel-level constraints by calculating the pixel-level L1 loss between them; the calculation is as follows: In formula (8), w and h are the width and height of the image; Step 4-4: Domain-level consistency constraint. Construct a domain discriminator D consisting of a convolutional neural network and a fully connected layer classifier. Use the domain discriminator to discriminate between the generated elasticity map and the true elasticity map. If the domain discriminator can easily distinguish between the generated elasticity map and the true elasticity map, it means that the domain-level consistency between the generated image and the true image is weak. Otherwise, it means that the domain-level consistency is strong. In this way, the domain-level consistency constraint is established. Its calculation is as follows: In formula (9), D is the domain discriminator and G is the generative network; Step 4-5. Overall optimization objective. The overall optimization objective of virtual imaging model training is obtained by adaptively weighted summation of the above four levels of constraints. Among them, the adaptive weighting parameter is driven and calculated by the causal chain of nodule diagnosis, and the calculation is as follows: In Equation (10), y c is the probability that the sample belongs to the c-th class, C is the number of possible classes, and f Diag is the diagnostic model, represents the loss function with weight parameters w1, w2, and w3; The final overall loss is calculated as follows: In formula (11) and are weighting coefficients.

Images