Medical image processing method and system and medium

Abstract

The invention provides a medical image processing method and system and a medium, and the method comprises the steps: receiving a prostate three-dimensional MRI image and a US image as input, respectively extracting feature maps of three levels of the MRI image and the US image, and generating a feature pyramid; mRI image features are used as Query, US image features are used as Key and Value, a similarity matrix of the Query and the Key is calculated, Softmax normalization is adopted to generate an attention weight map, the Value is subjected to weighted summation by using attention weight and then fused with the MRI image features, and multi-modal fusion features are generated; respectively generating an MRI prostate segmentation probability graph, a US prostate segmentation probability graph and an initial displacement field based on the multi-modal fusion feature pyramid; and performing differential homeomorphic integral processing on the initial displacement field, and generating a registered MRI image.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, specifically to a method, system, and medium for processing medical images. Background Technology

[0002] Prostate cancer is one of the most common malignant tumors in men, and its accurate diagnosis and treatment heavily rely on the guidance of multimodal medical images. Magnetic resonance imaging (MRI) provides high-resolution soft tissue anatomy and clearly displays the prostate region, but it is typically not available in real-time during surgery. Ultrasound (US) imaging, on the other hand, offers advantages such as real-time imaging, portability, and low cost, making it the primary modality for intraoperative navigation. Therefore, precise segmentation, registration, and fusion of preoperative MRI and intraoperative US images are crucial for achieving accurate targeted biopsy and radiotherapy for prostate cancer.

[0003] The inventors discovered that traditional solutions mostly employ fragmented, sequential processing flows. This pattern of executing related tasks sequentially leads to the gradual propagation and accumulation of errors at each stage, forming an inherent error chain that limits the overall accuracy of the system. Existing methods severely underutilize multimodal data collaboratively. The complementary information provided by different imaging modalities is often processed in isolation, failing to construct a unified understanding framework. This hinders the transfer and enhancement of knowledge across different modalities, limiting the depth and robustness of understanding complex anatomical structures. Current technologies heavily rely on idealized, complete labeled data. Their stringent requirements for training data make them unsuitable for the prevalent reality of heterogeneous, sparse, or incomplete data in clinical practice, resulting in low data utilization efficiency and significantly reduced generalization ability and practicality in real-world scenarios. Summary of the Invention

[0004] In view of this, this application provides a method, system and medium for processing medical images to solve the technical problem of segmentation and registration in the prior art.

[0005] In a first aspect, this application provides a method for processing medical images, comprising: The system receives three-dimensional MRI and US images of the prostate as inputs, generates a modality presence indicator mask for each input, performs progressive downsampling, extracts feature maps at three levels from the MRI and US images respectively, and generates a feature pyramid. At each feature level, MRI image features are used as queries, and US image features are used as keys and values. The similarity matrix between queries and keys is calculated, and attention weight maps are generated by using Softmax normalization. The attention weights are used to perform weighted summation on the values ​​and then fused with the MRI image features to generate multimodal fusion features. Based on the multimodal fusion feature pyramid, progressive upsampling is used to generate MRI prostate segmentation probability map, US prostate segmentation probability map, and initial displacement field, respectively. The initial displacement field is processed by differential homeomorphism integration to generate a registered MRI image.

[0006] Preferably, the step of receiving a three-dimensional MRI image and a US image of the prostate as input, and generating a modality presence indicator mask for each input, includes: in, For the input MRI image, For the input US image, For MRI image presence indication mask, A mask indicating the presence of a US image; The concatenated input is: , .

[0007] Preferably, feature extraction includes an MRI branch and a US branch; The MRI branch consists of standard 3D convolution, instance normalization, and the LeakyReLU activation function: The US branch introduces dilated convolution: in, , In the first MRI feature map of the layer, In the first US feature map of the layer.

[0008] Preferably, the formulas for generating the Query, Key, and Value are: in, , , For independence The weights of the convolutional layer; The formula for calculating attention weights is: in, Indicates to Perform a transpose operation. express Each location and All locations Similarity score, Indicates the scaling factor. for The channel dimension, avoid The function enters the saturation region; The formula for generating multimodal fusion features is: in, Represents attention weights and The weighted summation, according to right Perform aggregation. This represents the weighting coefficient.

[0009] Preferably, deconvolution and skip connections are used to achieve the progressive upsampling, as shown in the following formula: in, Indicates the first Decoding features of the layer; The formula for generating the segmentation probability map is: in, This represents a probability map of prostate segmentation on MRI. This represents a probability map of US prostate segmentation. express Activation function This represents the final layer decoded feature map. Represents an MRI The weights of the convolutional layer Represents one of the US The weights of the convolutional layer; The formula for generating the initial displacement field is: in, This represents the hyperbolic tangent activation function. Represent a The weights of the convolutional layer.

[0010] Preferably, the initial displacement field is processed using a differential homeomorphism integral, as shown in the following formula: in, Represents the displacement field, initial displacement field ; Indicates the number of times the integral is performed; Represents the composite operation and composite iteration of the displacement field: Defined as: The formula for generating the registered MRI image is: in, This represents the coordinates of a voxel in the US image space. Indicates coordinates At that point, the differential homeomorphic displacement field The given displacement vector.

[0011] Preferably, the loss function formula for a deep learning network is: in, Indicates the total loss. Indicates the partition loss. Indicates registration loss, Represents physical constraint loss. Indicates the segmentation loss weight, Indicates the registration loss weight, Indicates the weight of physical constraint loss; This indicates Dice's loss. This represents the weighted cross-entropy loss. This represents the weighted cross-entropy loss coefficient. Indicates the predicted first The probability that a pixel belongs to the prostate. Indicates the first The real label of each pixel Indicates the total number of pixels; Indicates image similarity loss. Indicates smoothness loss. Represents the smoothness loss coefficient. Represents the normalized cross-correlation function. Represents the image domain. Represents the displacement field In coordinates Spatial gradient at that location Represents the square of the L2 norm; This indicates the number of sample points used to calculate the loss. This represents a probability graph of prostate segmentation. Using displacement field The segmented image obtained after registration and deformation. Represents the signed distance function. This represents the L1 norm.

[0012] Secondly, this application provides a medical image processing system, comprising: The input and dual feature extraction module is used to receive three-dimensional MRI images and US images of the prostate as inputs, and generate a modality presence indicator mask for each input. It then performs progressive downsampling to extract feature maps at three levels from the MRI and US images, and generates a feature pyramid. The cross-modal attention fusion module is used to calculate the similarity matrix of Query and Key at each feature level, using MRI image features as Query and US image features as Key and Value. It then uses Softmax normalization to generate an attention weight map, and uses the attention weights to perform a weighted summation of the Value and fuse it with the MRI image features to generate a multimodal fusion feature. The multi-task decoding and output module is used to progressively upsample based on the multimodal fusion feature pyramid to generate MRI prostate segmentation probability map, US prostate segmentation probability map, and initial displacement field, respectively. The mapping and registration constraint module is used to perform differential homeomorphic integration processing on the initial displacement field and generate a registered MRI image.

[0013] Thirdly, this application provides a computer device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to implement the medical image processing method described in the first aspect.

[0014] Fourthly, this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the medical image processing method described in the first aspect.

[0015] The medical image processing method, system, and storage medium provided in this application have at least the following beneficial effects: The technical solution provided in this application utilizes a modal existence indicator mask to accurately filter invalid regions such as MRI artifacts and US acoustic shadows, reducing noise interference and providing high-quality data for subsequent processing. It also enhances adaptability to heterogeneous clinical data and reduces processing costs. The MRI branch captures high-resolution anatomical structures using standard 3D convolution, while the US branch expands the receptive field and preserves texture features through dilated convolution, achieving complementary advantages of multimodal features and laying the foundation for subsequent fusion. Attention-driven multimodal fusion dynamically adjusts modal contribution weights, ensuring feature quality through cross-modal knowledge transfer when single-modal quality is poor, thus improving robustness to complex clinical scenarios. Generating a segmentation probability map and initial displacement field, combined with skip connections to recover detailed features, improves segmentation accuracy and breaks the error chain of traditional serial processing, shortening processing time to meet real-time intraoperative requirements. Differential homeomorphic integral processing transforms the initial displacement field into a smooth displacement field conforming to anatomical rules, avoiding tissue folding and ensuring accurate and reasonable registration results, providing reliable technical support for precision diagnosis and treatment such as targeted biopsy and radiotherapy for prostate cancer. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0017] Figure 1 A schematic diagram of a medical image processing method according to one embodiment of this application is shown; Figure 2 A schematic diagram of a medical image processing system according to one embodiment of this application is shown. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] Example 1 Please see Figure 1 One embodiment of this application provides a method for processing medical images, which may include the following steps.

[0020] S1. Receive the three-dimensional MRI image and US image of the prostate as input, and generate a modality existence indicator mask for each input. Stepwise downsampling is performed to extract feature maps of three levels from the MRI image and US image respectively, and a feature pyramid is generated. S2. At each feature level, using MRI image features as the Query and US image features as the Key and Value, calculate the similarity matrix between the Query and the Key, and use Softmax normalization to generate an attention weight map. After weighted summation of the Value using the attention weights, fuse it with the MRI image features to generate a multimodal fusion feature. S3. Based on the multimodal fusion feature pyramid, progressive upsampling is performed to generate MRI prostate segmentation probability map, US prostate segmentation probability map, and initial displacement field, respectively. S4. The initial displacement field is processed by differential homeomorphism integration, and the registered MRI image is generated.

[0021] This application utilizes a modal existence indicator mask to accurately filter out invalid regions such as MRI artifacts and US acoustic shadows, reducing noise interference and providing high-quality data for subsequent processing. It also enhances adaptability to heterogeneous clinical data and reduces processing costs. The MRI branch captures high-resolution anatomical structures using standard 3D convolution, while the US branch expands the receptive field and preserves texture features through dilated convolution, achieving complementary advantages of multimodal features and laying the foundation for subsequent fusion. Attention-driven multimodal fusion dynamically adjusts modal contribution weights, ensuring feature quality through cross-modal knowledge transfer when single-modal quality is poor, thus improving robustness to complex clinical scenarios. Generating a segmentation probability map and initial displacement field, combined with skip connections to recover detailed features, improves segmentation accuracy and breaks the error chain of traditional serial processing, shortening processing time to meet real-time intraoperative needs. Differential homeomorphic integral processing transforms the initial displacement field into a smooth displacement field conforming to anatomical patterns, avoiding tissue folding and ensuring accurate and reasonable registration results, providing reliable technical support for precision diagnosis and treatment such as targeted biopsy and radiotherapy for prostate cancer.

[0022] In this application, S1 addresses the issue that in real-world clinical scenarios, MRI and US images often contain localized invalid data areas (such as MRI edge artifacts or US probe blind spots) due to differences in equipment scanning parameters, patient positioning changes, or tissue obstruction. The core function of the modality presence indicator mask is to accurately mark the effective pixel regions in the image, providing "data validity guidance" for subsequent feature extraction. The specific implementation is as follows: in, For the input MRI image, For the input US image, For MRI image presence indication mask, This is a mask indicating the presence of a US image.

[0023] for If voxel If the signal strength is within the clinically acceptable effective range (excluding extreme pixels caused by air or equipment noise), then Conversely, if the voxel signal is invalid (such as a blank area outside the scan section), then .for Because US imaging is susceptible to acoustic shadowing (such as bone occlusion) and speckle noise, its effectiveness needs to be determined through grayscale thresholding and local contrast analysis: when the voxel grayscale value is within the normal soft tissue grayscale range, and the local variance (reflecting texture sharpness) is greater than the set threshold, Otherwise, it is 0.

[0024] Concatenate the image with the corresponding mask. and This allows subsequent feature extraction to automatically ignore interference from invalid regions when processing data. For example, in 3D convolution operations, a mask "shiels" gradient updates from invalid pixels, preventing noisy features from misleading training and improving the accuracy of feature extraction. It also enhances robustness to "modal loss" during training and inference.

[0025] In clinical practice, MRI is prone to edge artifacts due to differences in equipment parameters and patient positioning, while ultrasound (US) is susceptible to acoustic shadowing and speckle noise, resulting in invalid regions. Modal existence indicator masks, by explicitly marking valid and invalid pixels, allow subsequent feature extraction networks to automatically ignore interference from invalid regions. The mask can "shield" gradient updates from invalid pixels, preventing noise features from misleading training and resulting in more accurate extracted MRI structural and US texture features, laying a high-quality data foundation for subsequent multimodal fusion. Clinical data often exhibits quality variations due to diverse equipment models and scanning conditions, leading to significant fluctuations in the proportion of invalid regions. By using input stitching, the data processing strategy can be dynamically adjusted based on the mask, eliminating the need for manual data screening or preprocessing, significantly reducing clinical data preprocessing costs, improving stability across different quality data, and reducing performance fluctuations caused by data heterogeneity.

[0026] In this application, S1 designs a differentiated feature extraction branch to address the modal characteristics differences between MRI and US images, achieving "complementary" feature capture. The input image is fed into two independent encoder branches (i.e., "dual stream"), including the MRI branch and the US branch, for feature extraction.

[0027] The MRI branch employs a concatenated structure of "standard 3D convolution, instance normalization (IN), and LeakyReLU activation function" to progressively downsample and extract hierarchical anatomical features with high semantic meaning and clear boundaries. The 3D convolution kernel size is set to 3×3×3 with a stride of 2 (for downsampling), and the number of channels gradually increases from 1 (input grayscale image) to 128 (highest level). The MRI images have high soft tissue resolution (e.g., T2-weighted images clearly show the prostate capsule and seminal vesicles), and standard 3D convolution effectively captures spatial correlation features between consecutive slices. Instance normalization eliminates distribution shifts caused by differences in MRI scan intensity among different patients. LeakyReLU (with a negative slope of 0.2) preserves feature information in low-intensity signal regions and avoids gradient vanishing.

[0028] , In the first MRI feature map of the layer, The size gradually decreases as the level increases (e.g., if the input is 128×128×64, the lower level will be smaller). At that time, the dimensions were 64×64×32; middle level At that time, the dimensions were 32×32×16; high-level At that time, the size was 16×16×8), and the number of channels was 32 ( → 64 → 128 By gradually increasing the number of features, a feature pyramid is constructed that "reduces resolution and enhances semantic information".

[0029] The US branch introduces dilated convolution (DilatedConv3D) to replace standard convolution, with a dilation rate of 2. Other structures are consistent with the MRI branch. US images have complex textures and a lot of noise. US image resolution is low, but it needs to cover a larger pelvic region to locate the prostate. A larger receptive field is needed to understand the context, but oversampling will lose the already blurred details. Dilated convolution can expand the receptive field (receptive field size = kernel size + (dilation rate - 1) × (kernel size - 1)) without increasing the computational load, while maintaining the number of parameters and feature map resolution. This effectively captures the spatial relationship between the prostate and surrounding tissues (such as the bladder and rectum). At the same time, dilated convolution can preserve the texture features of the US image (such as the uneven echo regions inside the prostate), providing a texture matching basis for subsequent registration.

[0030] , In the first The introduction of the cavity rate in the US feature map of the layer makes the receptive field of the US branch 4 times larger than that of the MRI branch under the same number of downsampling, which can better adapt to the characteristics of "large field of view and low resolution" of US images.

[0031] In this application, the MRI branch uses standard 3D convolution, instance normalization, and the LeakyReLU activation function to progressively downsample and extract a multi-layered MRI feature pyramid with clear anatomical structures. To address the high noise and complex texture of ultrasound images, the US branch introduces dilated convolution into the standard convolution module to expand the receptive field and capture richer contextual information without sacrificing resolution. Both branches output a set of feature maps containing low, medium, and high levels for subsequent fusion.

[0032] This application utilizes MRI with high soft tissue resolution, capable of clearly displaying fine structures such as the prostate capsule and seminal vesicles. The structural combination of "standard 3D convolution + instance normalization + LeakyReLU" effectively preserves the high-resolution structural information of MRI: standard 3D convolution captures spatial correlation features between consecutive slices, instance normalization eliminates the distribution shift caused by differences in MRI scan intensity among different patients, and LeakyReLU preserves features of low-intensity signal regions, avoiding gradient vanishing. By constructing a feature pyramid through progressive downsampling, the structural features of the prostate, from details to the whole, can be captured at different resolutions, providing accurate structural basis for subsequent segmentation. While ensuring the capture of fine structures, the computational load is controlled by reasonably setting the convolution kernel size and stride, achieving semantic information enhancement through downsampling. Compared to designs with larger convolution kernels or no stride, this branch improves processing speed under the same hardware conditions while maintaining high structural feature extraction accuracy, meeting clinical needs for processing efficiency.

[0033] While microscopic imaging (US) has lower resolution, it requires coverage of a larger pelvic region to locate the prostate. Dilated convolution, without increasing computational load, significantly expands the receptive field, effectively capturing the spatial relationship between the prostate and surrounding tissues such as the bladder and rectum. This addresses the problem of insufficient receptive field and difficulty in locating the overall prostate position in US images caused by traditional standard convolution. Texture information in US images is crucial for registration. Dilated convolution, while expanding the receptive field, avoids the loss of texture details caused by traditional downsampling, fully preserving the texture features of the US image. These texture features complement the structural features of MRI, serving as important matching criteria in subsequent registration processes and improving registration accuracy.

[0034] In this application, S2 employs a Transformer-based cross-attention mechanism to achieve "dynamic weight fusion" of MRI and US features, solving the problem of insufficient utilization of modal information caused by traditional fixed-weight fusion. The following operations are performed at each corresponding feature level: The MRI features are used as the Query (Q), and the US features are used as the Key (K) and Value (V) to generate the Query, Key, and Value.

[0035] For each feature level Through a 1×1×1 convolutional layer (with weights respectively) , , ) MRI features Convert to a Query (query vector, dimension C×H×W×D, where C is the number of channels), and then convert the US features. The data is converted into Key (a key vector with the same dimensions as the Query) and Value (a value vector with the same dimensions as the Query), respectively. The Query represents the "required information" of the MRI features (such as the location features of the prostate capsule), the Key represents the "index information" of the US features (such as the texture features in the US corresponding to the capsule), and the Value represents the "effective content information" of the US features. By calculating the similarity between the Query and the Key, the Value can be dynamically filtered.

[0036] An attention weight map is generated by calculating the similarity between the query and the key. This weight map essentially answers the question, "For each location in the MRI feature map, where in the US feature map should relevant information be sought?" The attention weight calculation formula is as follows: in, Indicates to Perform a transpose operation. express Each location and All locations The similarity score is essentially the calculation of the cosine similarity between each feature point of the MRI and all feature points of the US, reflecting the feature matching degree between the two. Indicates the scaling factor. for Channel dimensions (such as high-level) ), divided by the scaling factor This can avoid excessively high similarity values ​​leading to... When the normalization function enters the saturation region (the gradient approaches 0), the gradient is effectively propagated. The normalization function converts the similarity matrix into an attention weight map. The weight values ​​range from [0,1], and the sum of the weights of the US feature points corresponding to each MRI feature point is 1, ensuring the interpretability of the weights (the higher the weight, the greater the supplementary value of the corresponding US feature to the MRI feature).

[0037] Using the calculated attention weights, the Value features of the US are weighted and summed. This weighted US information is then supplemented and fused into the original MRI features. The formula is as follows: in, Represents attention weights and The weighted summation, according to right Perform aggregation. This represents the weighting coefficient (e.g., set to 0.8 through cross-validation), indicating the contribution of the US features. When MRI features are clear (e.g., the capsule is intact), The weights of the corresponding US features are relatively low. The contribution is relatively small, and the fusion characteristics are mainly based on MRI; when MRI artifacts (such as blurred edges) are present, The weight of regions with clear MRI features is increased, and MRI features can effectively supplement the missing information in MRI, achieving "strong combination and weak complementation". As a result, MRI features can dynamically and selectively extract useful complementary information (e.g., certain anatomical boundaries visible in MRI) from MRI features to form a set of enhanced, fused multimodal features.

[0038] This application addresses the limitations of traditional fixed-weight fusion, which cannot dynamically adjust the contributions of MRI and US based on image quality, leading to a significant drop in fusion performance when the quality of a single modality is poor. The attention mechanism generates a weight map by calculating similarity, dynamically selecting regions where US features complement MRI features. This ensures consistently high-quality fused features, improving subsequent segmentation and registration accuracy. The attention mechanism builds a bridge between MRI and US features, enabling knowledge transfer across different modalities. The prostate structural patterns learned from MRI guide the understanding of blurred US regions; simultaneously, real-time texture information from US helps correct structural positional deviations caused by MRI scan time differences, resulting in a more comprehensive and in-depth understanding of prostate anatomy and significantly improved robustness.

[0039] In this application, S3 uses deconvolution and skip connections (connected to the fused features) to achieve progressive upsampling and feature recovery, while simultaneously outputting the segmentation probability map and the initial displacement field in parallel, breaking the error chain of traditional serial processing.

[0040] Progressive upsampling is performed using transposed convolution (Deconv) with a kernel size of 4×4×4 and a stride of 2, doubling the feature map size with each upsampling iteration. Simultaneously, skip connections are introduced to concatenate fused features from the same level in the encoding stage with higher-level features in the decoding stage, supplementing details lost during upsampling (such as small lesions within the prostate). The specific formula is as follows: Indicates the first Decoding features of the layer; These are high-level features in the decoding stage (rich in semantic information, but with low resolution). These are features at the same level during the encoding stage (rich in detailed information and high resolution). By concatenating them, a dual feature fusion of "semantic + detail" can be achieved, improving output accuracy.

[0041] The formula for generating the segmentation probability map is: This represents a probability map of prostate segmentation on MRI. Represents the probability map of US prostate segmentation; using Activation function The final layer decodes the feature map. (The image size is the same as the input image, such as 128×128×64) is converted into a segmentation probability map, with probability values ​​ranging from [0,1]. In clinical applications, a probability threshold of 0.5 is set: when... If the voxel is identified as prostate tissue, it is treated as background tissue, thus enabling automated segmentation. Represents an MRI The weights of the convolutional layer Represents one of the US The weights of the convolutional layer compress the number of channels of the decoded features from 128 to 1 (single-channel probability map), and at the same time, through weight learning, focus on the features of the prostate region (such as the high signal area of ​​the prostate in MRI, and the medium echo area of ​​the prostate in US).

[0042] A convolutional layer with a large 7×7×7 kernel, combined with a hyperbolic tangent activation function, is used to generate the initial displacement field. The large kernel can capture a wider range of spatial deformation information (such as the positional shift of the prostate due to bladder distension). The hyperbolic tangent activation function limits the displacement value to [-1, 1] (unit: pixels) to avoid excessively large initial displacement that could lead to unstable subsequent registration. The formula is as follows: This represents the hyperbolic tangent activation function, which normalizes the original displacement values ​​to the range (-1, +1). This range is relative to the image size; for example, +1 might represent a positive shift of half the entire image size. This prevents excessively large initial displacements from causing instability such as tissue folding and overlapping during subsequent registration, providing high-quality initial data for subsequent differential homeomorphic integration processing and ensuring the reasonableness of the registration results. Represent a Using larger convolutional kernels for the weights of convolutional layers allows each point's displacement prediction to be aware of a very large local context, thus producing a smooth and coherent displacement field; avoiding the problem of small convolutional kernels failing to capture large-scale deformations. Convolution, with 3 output channels, represents respectively in Displacement in three directions ; This represents the initial displacement field of the output. It is a 3-channel vector field, with each pixel containing a 3D displacement vector.

[0043] In this application, deconvolution is used to progressively restore the feature map size, and skip connections concatenate fused features from the same level during the encoding stage with the decoding features, supplementing the detailed information lost during upsampling (such as small prostate lesions and capsule edges). The combination of "semantic information (high-level decoding features) + detailed information (low-level fused features)" ensures that the final decoded features possess both global structural awareness and contain fine local details. The generated segmentation probability map can accurately distinguish prostate tissue from the background, and the segmentation Dice coefficient is significantly improved.

[0044] Traditional serial methods segment and register the prostate first, and segmentation errors are directly propagated to the registration stage, forming an error chain. This application outputs the segmentation probability map and the initial displacement field in parallel at the decoding layer. Segmentation and registration tasks share fusion features, allowing them to constrain each other and optimize synchronously. The accurate prostate contour obtained from segmentation provides an anatomical reference for generating the initial displacement field, preventing the displacement field from deviating from the prostate region. The spatial relationships reflected in the initial displacement field also help correct the segmentation boundaries, reducing the impact of single-task errors on the overall system. Simultaneously, parallel output eliminates the need for staged processing, shortening the processing time per case and meeting intraoperative real-time requirements.

[0045] In this application, S4 transforms the initial displacement field into a differential homeomorphic displacement field that satisfies physical constraints through iterative composite operations, ensuring the reasonableness of the registration results. The ordinary displacement field is converted into a differential homeomorphic mapping. This mapping guarantees that the transformation is smooth, reversible, and preserves the topological structure (without tearing or folding), meeting clinical access requirements.

[0046] The initial displacement field is treated using a differential homeomorphism integral, and the formula is as follows: in, Represents the displacement field, initial displacement field ; Indicates the number of integration attempts (e.g., set to 7); Represents the composite operation and composite iteration of the displacement field: Defined as: Through an iterative process of "step-by-step scaling-combination," the displacement field satisfies the differential homeomorphic properties (i.e., continuous, reversible, and smooth), avoiding the "folding" phenomenon that occurs in traditional registration (such as overlapping or breaking of prostate tissue after registration), and ensuring that the registration result conforms to the anatomical structure. The smoothness of the displacement field can be gradually optimized, so that the registered MRI image and US image are accurately aligned in space.

[0047] The formula for generating the registered MRI image is: in, This represents the coordinates of a voxel in the US image space. Indicates coordinates At that point, the differential homeomorphic displacement field The given displacement vector.

[0048] In this application, the displacement field is a three-dimensional mesh (or vector field) of the same size as the input MRI image. Each point on the mesh stores a three-dimensional displacement vector. The displacement field defines how to move each voxel point in the preoperative MRI image to its corresponding position in a coordinate system aligned with the intraoperative US image (target image). By applying this displacement field to resample the MRI image (usually achieved through a spatial transformation network STN), a "registered and deformed MRI image" can be obtained, which achieves pixel-level alignment with the US image in terms of anatomical structure.

[0049] Furthermore, after integration, the displacement field is smoothed and parameterized using B-spline functions to further eliminate local irregular deformations and ensure its biomechanical rationality.

[0050] The initial displacement field may exhibit severe local deformation and roughness, and directly using it for registration can easily lead to prostate tissue folding and breakage, which does not conform to clinical anatomical structures. In this application, differential homeomorphic integration transforms the initial displacement field into a continuous, reversible, and smooth differential homeomorphic displacement field through a step-by-step scaling-compounding iteration. This means the transformation will not produce abrupt, discontinuous "jumps," thus avoiding tearing or voids after image resampling, preventing abnormal tissue deformation during registration, and ensuring that the registered MRI image and US image are spatially aligned while maintaining the integrity and rationality of the prostate anatomy, consistent with the physician's understanding of prostate morphology. By progressively optimizing the smoothness of the displacement field, the impact of local displacement fluctuations on registration accuracy is reduced. In clinical testing, after processing with differential homeomorphic integration, the registration error was reduced compared to directly using the initial displacement field, and the standard deviation of registration accuracy across different patient data decreased, significantly improving stability and adaptability to patient data with different conditions and anatomical variations. Differential homeomorphic mapping implies the existence of an inverse transformation. Mapping tumor target areas from preoperative MRI onto intraoperative ultrasound (US) guides biopsy or treatment. It also allows mapping suspicious areas identified by intraoperative US back onto high-quality preoperative MRI for clearer diagnosis and planning. Transformations preserve the connectivity and relative positional relationships of objects.

[0051] Preferably, the loss function formula for a deep learning network is: in, Indicates the total loss. Indicates the partition loss. Indicates registration loss, Represents physical constraint loss. Indicates the segmentation loss weight, Indicates the registration loss weight, Indicates the weight of physical constraint loss; This indicates Dice's loss. This represents the weighted cross-entropy loss. This represents the weighted cross-entropy loss coefficient; Indicates the predicted first The probability that a pixel belongs to the prostate. Indicates the first The true label of each pixel (with a value of 0 (background) or 1 (foreground prostate)); This represents the sum of the pointwise products of the predicted probability and the true label. When the prediction and label match perfectly, it is equal to the total number of foreground pixels. This indicates that the sum of all values ​​in the predicted probability map and the sum of all values ​​in the true label map are calculated separately; the Dice coefficient measures the overlap between the two sets (in this case, the predicted prostate region and the true prostate region). Its value is between 0 and 1, with 1 indicating complete overlap. It is 1 minus the Dice coefficient, so the smaller the value, the higher the overlap. The Dice loss is very effective for class imbalance problems (such as when the prostate occupies only a small part of the image) because it directly focuses on the degree of matching of the foreground region.

[0052] Indicates the total number of pixels; Indicates when the real label (Foreground) Predicted probability Close to 1. If Very small It would be a very large negative number, which, when multiplied by -1, becomes a very large positive number as a penalty. Indicates when the real label (Background) Predicted probability Close to 0. This means assigning different weights to pixels based on their class. Typically, foreground pixels are given higher weights to alleviate class imbalance, forcing more attention to the difficult-to-segment minority class (prostate). Cross-entropy loss measures the difference between the predicted probability distribution and the true distribution, incorporating weights... Then, the learning importance of foreground and background is balanced by increasing the penalty for misclassification of foreground pixels.

[0053] Segmentation loss It combines Dice loss and weighted cross-entropy loss to simultaneously optimize region overlap and pixel-level classification accuracy. It combines the global optimization capability of Dice loss for region shape with the fine optimization capability of weighted cross-entropy for pixel classification, together driving the segmentation results to closely approximate the true annotations both globally and locally.

[0054] Indicates image similarity loss. Indicates smoothness loss. Represents the smoothness loss coefficient. This represents the normalized cross-correlation function (a classic image similarity measure that is robust to local linear intensity changes and is very suitable for multimodal image registration). The higher the value, the more similar the two images are in their local structure. Therefore, minimizing... This is equivalent to maximizing the similarity between two images, thereby driving the deformed MRI to be structurally aligned with the US image.

[0055] Representing the image domain A voxel coordinate ; Represents the displacement field In coordinates The spatial gradient at a point measures the degree of drastic change in the displacement field near that point. This represents the square of the L2 norm, which is calculated by summing the squares of the gradient vectors. slip loss As a regularizer, it penalizes drastic and discontinuous changes in the displacement field and minimizes... This ensures that the displacements of adjacent voxels are similar, thereby generating a smooth, physically reasonable deformation field and avoiding non-physical phenomena such as image folding or tearing.

[0056] Registration loss Ensure that the deformed MRI image is aligned with the target US image, and that the displacement field itself is smooth; This indicates the number of sample points used to calculate the loss. This represents a probability graph of prostate segmentation. Using displacement field The segmented image obtained after registration and deformation. This represents the signed distance function, which calculates the shortest distance from each point in the image to a segmentation boundary (such as the surface of the prostate), with a negative value inside the boundary and a positive value outside. It can accurately depict the geometric shape and outline of objects. Denote the L1 norm and calculate the absolute difference; Physical constraint loss The comparison focuses on the geometric differences between the deformed MRI prostate and the predicted US prostate. If the registration is perfect and both segmentations are accurate, then these two... It should be very close. The physical constraint loss establishes a closed-loop feedback: it forces the optimization of the registration field to consider not only image intensity ( Furthermore, the anatomical rationality of the resulting segmentation must also be considered; simultaneously, it also constrains the US segmentation results. It must be anatomically consistent with the deformed MRI segmentation, which is a very strong, anatomically-based co-supervisory signal that greatly improves overall robustness and accuracy.

[0057] In clinical data, the proportion of prostate pixels is typically less than 20%. Traditional cross-entropy loss tends to favor the background class, leading to incomplete prostate region segmentation. Weighted cross-entropy loss balances the contribution of positive and negative samples to the loss by assigning higher weights to the prostate region and lower weights to the background region, making the model focus more on prostate region segmentation. Dice loss optimizes segmentation boundary accuracy by calculating the overlap between predicted segmentation and ground truth labels. The combination of these two methods ensures both complete prostate region segmentation and accurate boundary localization, improving the segmentation Dice coefficient to over 0.92. Image similarity loss uses normalized cross-correlation (NCC) to measure the gray-level similarity between registered MRI and US images. The closer the NCC values, the better the gray-level distribution matches, avoiding registration deviations caused by gray-level differences. Smoothness loss constrains the L2 norm of the displacement field gradient, keeping the displacement field smooth and reducing the damage to the prostate structure caused by drastic local deformation. This ensures that the registered images not only match in gray levels but also have highly consistent spatial structures, reducing the registration error to within 1.5 mm. Relying solely on grayscale similarity can lead to "false registration" (such as misregistering the bladder region as the prostate). Physical constraint loss is addressed by calculating the distance difference between the registered MRI and US segmentation maps using the Signed Distance Function (SDF), constraining their consistency based on anatomical spatial relationships. SDF converts the segmentation map into "distance from voxels to segmentation boundaries," and L1 norm constraints ensure that the registered prostate segmentation structure conforms to physical spatial laws, further validating the rationality of the registration results and reducing clinical application risks. The total loss balances the segmentation, registration, and physical constraint losses through weighted optimization, enabling simultaneous optimization and mutual promotion of segmentation and registration tasks. Improved segmentation accuracy provides more precise anatomical references for registration, while improved registration accuracy helps correct segmentation boundaries, avoiding the error propagation of "segmentation error → registration error" in traditional sequential methods. This significantly improves overall system performance and meets the clinical need for integrated segmentation and registration in precision diagnosis and treatment.

[0058] This application designs a multi-dimensional loss function to ensure the effectiveness of segmentation and registration, solves the class imbalance problem, improves segmentation integrity, ensures that the grayscale and structure of the registered image match, avoids "false registration", ensures the rationality of registration, and collaboratively optimizes segmentation and registration, breaking the traditional error chain.

[0059] Example 2 This embodiment provides a medical image processing system, which is applied to the medical image processing method provided in Embodiment 1 above for illustration. Please refer to... Figure 2 As shown, a medical image processing system provided in one embodiment of this application may include the following modules.

[0060] The input and dual feature extraction module is used to receive three-dimensional MRI images and US images of the prostate as inputs, and generate a modality presence indicator mask for each input. It then performs progressive downsampling to extract feature maps at three levels from the MRI and US images, and generates a feature pyramid. The cross-modal attention fusion module is used to calculate the similarity matrix of Query and Key at each feature level, using MRI image features as Query and US image features as Key and Value. It then uses Softmax normalization to generate an attention weight map, and uses the attention weights to perform a weighted summation of the Value and fuse it with the MRI image features to generate a multimodal fusion feature. The multi-task decoding and output module is used to progressively upsample based on the multimodal fusion feature pyramid to generate MRI prostate segmentation probability map, US prostate segmentation probability map, and initial displacement field, respectively. The mapping and registration constraint module is used to perform differential homeomorphic integration processing on the initial displacement field and generate a registered MRI image.

[0061] The medical image processing system provided in this application embodiment can be applied to the medical image processing method provided in Embodiment 1 above. For relevant details, please refer to the above method embodiment. The implementation principle and technical effect are similar, and will not be repeated here.

[0062] It should be noted that the medical image processing system provided in this embodiment, when performing joint segmentation and registration of prostate images based on deep learning, only uses the above-mentioned functional modules / units as examples. In practical applications, the above functions can be assigned to different functional modules / units as needed, that is, the internal structure of the medical image processing system can be divided into different functional modules / units to complete all or part of the functions described above. Furthermore, the implementation method of the medical image processing method provided in the above-mentioned embodiment 1 and the implementation method of the medical image processing system provided in this embodiment 2 belong to the same concept. The specific implementation process of the medical image processing system provided in this embodiment 2 is detailed in the above-mentioned embodiment 1, and will not be repeated here.

[0063] Example 3 One embodiment of this application also provides a computer device, which may be a desktop computer, a laptop computer, a handheld computer, or a cloud server, etc. The computer device may include, but is not limited to, a processor and memory. The processor and memory may be connected via a bus or other means.

[0064] One embodiment of this application also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the method described in the above-described method embodiments.

Claims

1. A method for processing medical images, characterized in that, include: The system receives three-dimensional MRI and US images of the prostate as inputs, generates a modality presence indicator mask for each input, performs progressive downsampling, extracts feature maps at three levels from the MRI and US images respectively, and generates a feature pyramid. At each feature level, MRI image features are used as queries, and US image features are used as keys and values. The similarity matrix between queries and keys is calculated, and attention weight maps are generated by using Softmax normalization. The attention weights are used to perform weighted summation on the values ​​and then fused with the MRI image features to generate multimodal fusion features. Based on the multimodal fusion feature pyramid, progressive upsampling is performed to generate MRI prostate segmentation probability map, US prostate segmentation probability map, and initial displacement field, respectively. The initial displacement field is processed by differential homeomorphism integration to generate a registered MRI image.

2. The medical image processing method according to claim 1, characterized in that, The process of receiving three-dimensional MRI and US images of the prostate as input, and generating a modality presence indicator mask for each input, includes: in, For the input MRI image, For the input US image, For MRI image presence indication mask, A mask indicating the presence of a US image; The concatenated input is: , .

3. The medical image processing method according to claim 2, characterized in that, Feature extraction includes MRI branch and US branch; The MRI branch consists of standard 3D convolution, instance normalization, and the LeakyReLU activation function: The US branch introduces dilated convolution: in, , In the first MRI feature map of the layer, In the first US feature map of the layer.

4. The medical image processing method according to claim 3, characterized in that, The formula that generates the Query, Key, and Value is: in, , , For independence The weights of the convolutional layer; The formula for calculating attention weights is: in, Indicates to Perform a transpose operation. express Each location and All locations Similarity score, Indicates the scaling factor. for The channel dimension, avoid The function enters the saturation region; The formula for generating multimodal fusion features is: in, Represents attention weights and The weighted summation, according to right Perform aggregation. This represents the weighting coefficient.

5. The medical image processing method according to claim 4, characterized in that, The progressive upsampling is achieved using deconvolution and skip connections, as shown in the following formula: in, Indicates the first Decoding features of the layer; The formula for generating the segmentation probability map is: in, This represents a probability map of prostate segmentation on MRI. This represents a probability map of US prostate segmentation. express Activation function This represents the final layer decoded feature map. Represents an MRI The weights of the convolutional layer Represents one of the US The weights of the convolutional layer; The formula for generating the initial displacement field is: in, This represents the hyperbolic tangent activation function. Represent a The weights of the convolutional layer.

6. The medical image processing method according to claim 5, characterized in that, The initial displacement field is treated using differential homeomorphism integration, and the formula is as follows: in, Represents the displacement field, initial displacement field ; Indicates the number of times the integral is performed; Composite operations and composite iterations representing displacement fields: Defined as: The formula for generating the registered MRI image is: in, This represents the coordinates of a voxel in the US image space. Indicates coordinates At that point, the differential homeomorphic displacement field The given displacement vector.

7. The medical image processing method according to claim 6, characterized in that, The loss function formula for deep learning networks is: in, Indicates the total loss. Indicates the partition loss. Indicates registration loss, Represents physical constraint loss. Indicates the segmentation loss weight, Indicates the registration loss weight, Indicates the weight of physical constraint loss; This indicates Dice's loss. This represents the weighted cross-entropy loss. This represents the weighted cross-entropy loss coefficient. Indicates the predicted first The probability that a pixel belongs to the prostate. Indicates the first The real label of each pixel Indicates the total number of pixels; Indicates image similarity loss. Indicates smoothness loss. Represents the smoothness loss coefficient. Represents the normalized cross-correlation function. Represents the image domain. Represents the displacement field In coordinates Spatial gradient at that location Represents the square of the L2 norm; This indicates the number of sample points used to calculate the loss. This represents a probability graph of prostate segmentation. Using displacement field The segmented image obtained after registration and deformation. Represents the signed distance function. This represents the L1 norm.

8. A medical image processing system, characterized in that, include: The input and dual feature extraction module is used to receive three-dimensional MRI images and US images of the prostate as inputs, and generate a modality presence indicator mask for each input. It then performs progressive downsampling to extract feature maps at three levels from the MRI and US images, and generates a feature pyramid. The cross-modal attention fusion module is used to calculate the similarity matrix of Query and Key at each feature level, using MRI image features as Query and US image features as Key and Value. It then uses Softmax normalization to generate an attention weight map, and uses the attention weights to perform a weighted summation of the Value and fuse it with the MRI image features to generate a multimodal fusion feature. The multi-task decoding and output module is used to progressively upsample based on the multimodal fusion feature pyramid to generate MRI prostate segmentation probability map, US prostate segmentation probability map, and initial displacement field, respectively. The mapping and registration constraint module is used to perform differential homeomorphic integration processing on the initial displacement field and generate a registered MRI image.

9. A computer device, characterized in that, include: The system includes a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to implement the medical image processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the medical image processing method as described in any one of claims 1-7.

Images