Radiodiagnostic image processing system and method

Abstract

The application provides a radiodiagnosis image processing system and method. The first modality image and the second modality image obtained are coarsely registered based on rigid transformation and affine transformation to obtain a coarsely registered image pair, then fine registration is performed using a non-rigid deformation field to obtain a finely registered image pair, a registration quality measurement value between the finely registered image pair is determined, then adaptive spatial transformation correction is performed on the second modality image, the corrected second modality image and the first modality image are respectively input into a feature extraction network to obtain a first modality feature map and a second modality feature map, channel attention fusion is performed on the first modality feature map and the second modality feature map to obtain a fused feature map, and then a spatial consistency enhanced radiodiagnosis fusion image is reconstructed based on the fused feature map. The scheme of the application can quantize the alignment quality of multi-modality images at the global and local levels to improve the correction accuracy of the fusion image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a radiological diagnostic image processing system and method. Background Technology

[0002] Image processing is a discipline that studies how to use computers to analyze, manipulate, and enhance digital images. Its core objective is to extract useful information from images or to transform them into a form that is more suitable for a specific application. Image processing typically includes: low-level processing of the pixels themselves, mid-level processing for analyzing and recognizing the content of the image, and high-level processing for attempting to understand the overall content of the image.

[0003] Radiological diagnostic image processing is the application of image processing in medical radiology. It specifically refers to methods that use computer algorithms to improve image quality, assist doctors in detecting lesions, and perform accurate measurements and diagnoses for medical images such as CT, MRI, X-ray, and PET. Its core objective is to enhance the value, accuracy, and reliability of images in clinical diagnosis. However, traditional radiological diagnostic image processing requires a large amount of manual intervention, such as parameter adjustment and result verification, which severely restricts the efficiency of large-scale application in clinical settings. Furthermore, traditional radiological diagnostic image processing is based on evaluation methods using single indicators or subjective visual examinations, which cannot comprehensively reflect the alignment quality of multimodal images at both global and local levels. Therefore, how to quantify the alignment quality of multimodal images at both global and local levels to improve the correction accuracy of fused images has become a challenge for the industry. Summary of the Invention

[0004] Based on this, this application provides a radiological diagnostic image processing system and method for quantifying the alignment quality of multimodal images at the global and local levels.

[0005] In a first aspect, this application provides a multimodal radiological diagnostic image fusion method, applied to a radiological diagnostic image processing system, the method comprising the following steps: Acquire a first modal radiographic image and a second modal radiographic image of the same anatomical site of the target patient, wherein the first modal image is a high-resolution structural image and the second modal image is a functional image; Coarse registration is performed on the first modal image and the second modal image based on rigid transformation and affine transformation to obtain coarsely registered image pairs; Based on the coarsely registered image pairs, fine registration is performed using a non-rigid deformation field to obtain finely registered image pairs, and then the registration quality metric between the finely registered image pairs is determined. Based on the registration quality metric, the second modality image is subjected to adaptive spatial transformation correction. The corrected second modality image and the first modality image are then input into the feature extraction network to obtain the first modality feature map and the second modality feature map. Channel attention fusion is performed on the first modality feature map and the second modality feature map to obtain a fused feature map, and then a spatially consistent radiological diagnostic fused image is reconstructed based on the fused feature map.

[0006] In some embodiments, coarse registration of the first modal image and the second modal image based on rigid transformation and affine transformation is performed to obtain a coarsely registered image pair, specifically including: Using the first modal image as a fixed image and the second modal image as a moving image, a rigid transformation based on normalized mutual information optimization is performed; Based on the rigid transformation, an affine transformation based on mutual information metric is performed; Output coarsely registered image pairs that have undergone continuous transformations and spatial alignment.

[0007] In some embodiments, fine registration is performed using a non-rigid deformation field based on the coarsely registered image pair to obtain a finely registered image pair, specifically including: The non-rigid deformation field between the coarsely registered image pairs is generated based on the B-spline free deformation model. The second modal image is spatially transformed based on the non-rigid deformation field to obtain a finely registered image pair.

[0008] In some embodiments, determining the registration quality metric between the finely registered image pairs specifically includes: Calculate the global intensity similarity metric for the finely registered image pairs; Extract the measure of local multi-scale feature similarity of the finely registered image pairs; The registration quality metric is generated by fusing the global intensity similarity metric with the local multi-scale feature similarity metric.

[0009] In some embodiments, adaptive spatial transformation correction of the second modality image based on the registration quality metric specifically includes: A correction strategy is selected based on the comparison result between the registration quality metric and the preset threshold. Generate the corresponding spatial deformation field based on the selected correction strategy; The second modal image is spatially transformed based on the spatial deformation field to achieve adaptive correction.

[0010] In some embodiments, inputting the corrected second modality image and the first modality image into a feature extraction network to obtain a first modality feature map and a second modality feature map specifically includes: The first modality image is input into a high-resolution feature preservation network to extract the first modality feature map; The corrected second modality image is input into a multi-scale context-aware network to extract the second modality feature map; The extracted first and second modal feature maps are then subjected to dimensional unification and spatial alignment.

[0011] In some embodiments, channel attention fusion is performed on the first modality feature map and the second modality feature map to obtain a fused feature map, specifically including: Channel attention weights are calculated for the first modality feature map and the second modality feature map, respectively; The first modality feature map and the second modality feature map are weighted and fused based on the calculated channel attention weights to generate the final fused feature map.

[0012] Secondly, this application provides a radiological diagnostic image processing system, which includes a multimodal radiological diagnostic image fusion unit, the multimodal radiological diagnostic image fusion unit comprising: The acquisition module is used to acquire a first modal radiographic image and a second modal radiographic image of the same anatomical site of the target patient, wherein the first modal image is a high-resolution structural image and the second modal image is a functional image; The processing module is used to perform coarse registration of the first modal image and the second modal image based on rigid transformation and affine transformation to obtain coarsely registered image pairs; The processing module is further configured to perform fine registration using a non-rigid deformation field based on the coarsely registered image pairs to obtain finely registered image pairs, and then determine the registration quality metric between the finely registered image pairs. The processing module is further configured to perform adaptive spatial transformation correction on the second modal image based on the registration quality metric, and input the corrected second modal image and the first modal image into the feature extraction network respectively to obtain the first modal feature map and the second modal feature map; The execution module is used to perform channel attention fusion on the first modality feature map and the second modality feature map to obtain a fused feature map, and then reconstruct a spatially consistent radiological diagnostic fusion image based on the fused feature map.

[0013] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described multimodal radiological diagnostic image fusion method.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described multimodal radiological diagnostic image fusion method.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The radiological diagnostic image processing system and method provided in this application first acquires the patient's CT image as a first-modality high-resolution structural image and PET image as a second-modality functional image. Second, through coarse registration based on rigid transformation and affine transformation, and subsequent fine registration based on a B-spline free deformation model, the two images are spatially aligned. Next, the registration quality metric of the finely registered image pair is calculated. This metric integrates global mutual information and local multi-scale feature similarity. Based on this metric, an adaptive correction strategy is selected: when the metric is below 0.75... The method employs thin-plate splines for global nonlinear correction. When the registration quality metric meets the standard but the similarity of local regions is insufficient, local B-splines are used for fine-tuning. The corrected PET image and the original CT image are input into feature extraction networks with HRNet and U-Net architectures, respectively. The extracted feature maps are then fused through channel attention. Finally, an encoder-decoder reconstruction network generates a CT-PET fused image with significantly enhanced spatial consistency. In summary, the scheme of this application can quantify the alignment quality of multimodal images at both global and local levels, thereby improving the correction accuracy of the fused image. Attached Figure Description

[0016] Figure 1 This is an exemplary flowchart of a multimodal radiological diagnostic image fusion method according to some embodiments of this application; Figure 2 This is a schematic diagram illustrating an application scenario of a multimodal radiological diagnostic image fusion data processing system according to some embodiments of this application; Figure 3 This is a schematic flowchart illustrating the process of determining registration quality metrics according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of a multimodal radiological diagnostic image fusion unit according to some embodiments of this application; Figure 5 This is a schematic diagram of the structure of a computer device for implementing a multimodal radiological diagnostic image fusion method according to some embodiments of this application. Detailed Implementation

[0017] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0018] refer to Figure 1 The figure is an exemplary flowchart of a multimodal radiological diagnostic image fusion method according to some embodiments of this application. The multimodal radiological diagnostic image fusion method mainly includes the following steps: In step 101, a first modal radiographic image and a second modal radiographic image of the same anatomical site of the target patient are acquired, wherein the first modal image is a high-resolution structural image and the second modal image is a functional image.

[0019] In specific implementation, acquiring the first and second modal radiographic images of the same anatomical site of the target patient can be achieved in the following way: First, multimodal image sequences of the same anatomical site can be retrieved and acquired based on the patient ID and examination time; second, the DICOM file header information of the images is parsed, and high-resolution structural images and functional images are selected according to the modality type field. Specifically, the high-resolution structural images are computed tomography images or T1-weighted magnetic resonance images, and the functional images are positron emission tomography images or diffusion-weighted magnetic resonance images; then, the consistency of the anatomical site is verified by comparing the patient's position information and sequence description of the two images, and geometric consistency is further verified by calculating the spatial overlap between the images. When the overlap is lower than a preset threshold, a review process based on feature point matching is triggered; finally, the verified spatially corresponding first and second modal radiographic images are output. Other methods can also be used in other embodiments, and this application does not limit them.

[0020] In some embodiments, reference Figure 2 As shown in the figure, this figure is a schematic diagram of an application scenario of a multimodal radiological diagnostic image fusion data processing system according to some embodiments of this application. The figure includes three main components: an acquisition device, a server, and a data storage device. The acquisition device is responsible for collecting first-modal radiological images and second-modal radiological images of the same anatomical site of the target patient, and sending the acquired first-modal radiological images and second-modal radiological images to the server through a communication network. Multimodal radiological diagnostic image fusion is performed on the server, and the server stores the processing results in the data storage device and visualizes them.

[0021] In step 102, coarse registration is performed on the first modal image and the second modal image based on rigid transformation and affine transformation to obtain coarsely registered image pairs.

[0022] In some embodiments, coarse registration of the first modal image and the second modal image based on rigid transformation and affine transformation to obtain coarsely registered image pairs can be achieved by the following steps: Using the first modal image as a fixed image and the second modal image as a moving image, a rigid transformation based on normalized mutual information optimization is performed; Based on the rigid transformation, an affine transformation based on mutual information metric is performed; Output coarsely registered image pairs that have undergone continuous transformations and spatial alignment.

[0023] In specific implementation, with the first modality image as the fixed image and the second modality image as the moving image, the rigid transformation based on normalized mutual information optimization can be implemented as follows: First, the second modality image is mapped to the coordinate system of the first modality image through an initial transformation matrix. Then, using normalized mutual information as a similarity measure, the gradient descent algorithm is used to iteratively optimize the six parameters of the rigid transformation, including three rotation parameters and three translation parameters, so that the two images achieve preliminary spatial alignment. In specific implementation, based on the rigid transformation, the affine transformation improved by mutual information metric can be implemented as follows. That is, based on the rigid transformation result, the transformation model is extended to a twelve-parameter affine transformation that includes scaling and shearing transformations. Mattes mutual information is used as a similarity measure, and the transformation parameters are further optimized through an optimization algorithm to eliminate scaling differences and shearing distortion between images. Finally, the output is a coarsely registered image pair after continuous transformation and spatial alignment. It should be noted that in this application, the second modal image has completed the spatial mapping from its own coordinate system to the coordinate system of the first modal image, providing accurate initial conditions for subsequent fine registration. Other registration strategies can also be used in other embodiments, and this application does not limit them.

[0024] In step 103, based on the coarsely registered image pairs, fine registration is performed using a non-rigid deformation field to obtain finely registered image pairs, and then the registration quality metric between the finely registered image pairs is determined.

[0025] In some embodiments, fine registration is performed using a non-rigid deformation field based on the coarsely registered image pair to obtain a finely registered image pair. This can be achieved by the following steps: The non-rigid deformation field between the coarsely registered image pairs is generated based on the B-spline free deformation model. The second modal image is spatially transformed based on the non-rigid deformation field to obtain a finely registered image pair.

[0026] In specific implementation, the non-rigid deformation field between the coarsely registered image pairs, based on the B-spline free deformation model, can be generated in the following way: First, a three-dimensional control point grid is constructed in the spatial domain of the first modal image. The physical spacing of the control point grid is adaptively set according to the spatial resolution of the first modal image. For example, for a first modal image with a spatial resolution higher than 1mm×1mm×1mm, the spacing of the control point grid is set to 8 to 12 pixels. For a first modal image with a spatial resolution lower than or equal to 1mm×1mm×1mm, the spacing of the control point grid is set to 8 to 12 pixels. The image is constructed with the spacing of the control point grid set to 12 to 16 pixels. Subsequently, the three-dimensional displacement vector of each control point is iteratively adjusted using the L-BFGS optimization algorithm, with the weighted sum of the normalized cross-correlation of the image pair to be registered and the bending penalty term of the deformation field as the objective function. Finally, the displacements of these discrete control points are smoothly interpolated using third-order B-spline basis functions to calculate the continuous displacement vector of each voxel position in the image, generating a smooth, continuous, and reversible non-rigid deformation field. Other methods can be used in other embodiments, and this application does not limit them.

[0027] In specific implementation, the spatial transformation of the second modal image based on the non-rigid deformation field to obtain a finely registered image pair can be achieved in the following way: the deformation field generated by the B-spline free deformation model can be applied to the original coordinate space of the second modal image, the corresponding source coordinates of each target voxel coordinate can be calculated, and the pixel intensity value at the corresponding position can be obtained by resampling from the second modal image using a third-order B-spline interpolation algorithm. Finally, a finely registered image pair that is aligned with the first modal image in terms of anatomical details can be output. Other methods can also be used in other embodiments, and this application does not limit them.

[0028] In some embodiments, reference Figure 3 As shown in the figure, this is a schematic flowchart of determining the registration quality metric value in some embodiments of this application. In this embodiment, determining the registration quality metric value between the finely registered image pairs can be achieved by the following steps: In step 1031, a metric for the global intensity similarity of the finely registered image pairs is calculated; In step 1032, the measure of local multi-scale feature similarity of the finely registered image pair is extracted; In step 1033, the registration quality metric is generated by fusing the global intensity similarity metric with the local multi-scale feature similarity metric.

[0029] In specific implementation, the global intensity similarity metric of the finely registered image pair can be calculated in the following way: the global intensity similarity can be quantified by calculating the mutual information between the intensity histograms of the first modal image and the second modal image after fine registration, and the global intensity similarity metric can be obtained. The calculation of mutual information involves the joint probability distribution and edge probability distribution of the intensity values ​​of the two images. Other methods can be used in other embodiments, and this application does not limit them.

[0030] It should be noted that the global intensity similarity metric in this application is used to evaluate the consistency of two images after registration from the perspective of the overall pixel intensity distribution. Its role is to capture the large-scale, statistically based similarity between images, serving as a fundamental and global component in the overall registration quality assessment.

[0031] In specific implementation, the measure of local multi-scale feature similarity of the finely registered image pair can be extracted in the following way: the finely registered image pair can be input into a pre-trained VGG16 network, feature maps can be extracted from the conv3_3 layer respectively, and the structural similarity index of the corresponding feature maps can be calculated at three scales: original resolution, 1 / 2 downsampling and 1 / 4 downsampling. Finally, the arithmetic mean of the structural similarity indices at the three scales can be taken as the measure of local multi-scale feature similarity. Other methods can also be used in other embodiments, and this application does not limit them.

[0032] It should be noted that the local multi-scale feature similarity metric in this application is used to evaluate the alignment accuracy of two images in a multi-scale space from the perspective of local anatomical structural features. Its role is to make up for the insensitivity of global intensity metric to local structural changes. By analyzing the abstract features extracted by the deep learning network, it can finely measure the consistency of the images in detail structure, which is another key component of the overall registration quality assessment.

[0033] In specific implementation, the registration quality metric generated by fusing the global intensity similarity metric and the local multi-scale feature similarity metric can be achieved as follows: The global intensity similarity metric and the local multi-scale feature similarity metric are respectively subjected to min-max normalization; then, the two normalized values ​​are linearly weighted and summed with weights of 0.6 and 0.4, where the weights are preferred weights determined by grid search on a multimodal image validation set including brain, chest, and abdomen images, generating a final registration quality metric ranging from 0 to 1. A registration quality metric closer to 1 indicates higher registration quality. In other embodiments, other similarity metrics or fusion weights can also be used, which are not limited in this application.

[0034] It should be noted that the registration quality metric in this application is a scalar index used to comprehensively quantify the spatial alignment degree between the first modal image and the second modal image after fine registration. Its core function is to provide a decision basis for the subsequent adaptive spatial transformation correction step. It uses a normalized value to objectively evaluate the quality of the registration result, thereby driving the system to determine whether to perform global correction, local correction or maintain the current state.

[0035] In step 104, based on the registration quality metric, the second modality image is subjected to adaptive spatial transformation correction. The corrected second modality image and the first modality image are then input into the feature extraction network to obtain the first modality feature map and the second modality feature map.

[0036] In some embodiments, adaptive spatial transformation correction of the second modality image based on the registration quality metric can be achieved by the following steps: A correction strategy is selected based on the comparison result between the registration quality metric and the preset threshold. Generate the corresponding spatial deformation field based on the selected correction strategy; The second modal image is spatially transformed based on the spatial deformation field to achieve adaptive correction.

[0037] In specific implementation, the correction strategy selected based on the comparison result between the registration quality metric and the preset threshold can be implemented in the following way: setting the first preset threshold to 0.75 and the second preset threshold to 0.6; when the registration quality metric is lower than 0.75, a global nonlinear correction strategy based on thin plate splines is selected; when the registration quality metric is higher than or equal to 0.75 but the structural similarity index of continuous image block regions is lower than 0.6, a refined correction strategy based on local B-splines is selected; when the above conditions are not met, an identity transformation strategy is selected to maintain the fine registration result. It should be noted that the first and second preset thresholds in this application can be determined based on the correlation analysis between the subjective scores and objective metrics of the registration results by clinical experts. Other threshold settings or decision logics can also be used in other embodiments, which are not limited in this application.

[0038] In specific implementation, the corresponding spatial deformation field generated based on the selected correction strategy can be achieved in the following ways: For global nonlinear correction, at least 25 pairs of significant anatomical feature points are extracted from the finely registered image pairs and the coordinate residuals are calculated. Based on these feature point pairs, a thin-plate spline algorithm is used to generate a global deformation field; For fine correction based on local B-splines, a B-spline grid with control point spacing densified to 4-6 pixels is established in local regions where the structural similarity index is less than 0.6. The control point displacement is optimized with the goal of maximizing the normalized mutual information of the local region to generate a local deformation field; For identity transformation, a zero-displacement field is generated. Other deformation field generation methods can also be used in other embodiments, which are not limited in this application.

[0039] In specific implementation, the second modal image is spatially transformed based on the spatial deformation field to complete adaptive correction. This can be achieved in the following way: the generated deformation field is applied to the voxel coordinates of the second modal image, the pixel intensity value at each new coordinate position is resampled and calculated using a third-order B-spline interpolation algorithm, and the image boundary region is mirror-filled to maintain intensity continuity. Finally, the corrected second modal image is output. In other embodiments, other interpolation algorithms or boundary processing strategies can also be used, which are not limited in this application.

[0040] In some embodiments, inputting the corrected second modality image and the first modality image into a feature extraction network to obtain the first modality feature map and the second modality feature map can be achieved by the following steps: The first modality image is input into a high-resolution feature preservation network to extract the first modality feature map; The corrected second modality image is input into a multi-scale context-aware network to extract the second modality feature map; The extracted first and second modal feature maps are then subjected to dimensional unification and spatial alignment.

[0041] In specific implementation, the first modality image is input into the high-resolution feature preservation network, and the first modality feature map is extracted. This can be achieved in the following way: the HRNet network architecture can be used as the high-resolution feature preservation network, and its input channel number can be adapted to a single-channel medical image input. The high-resolution representation is maintained throughout the network. The spatial detail preservation capability is enhanced by connecting multiple resolution subnets in parallel and repeatedly exchanging cross-resolution information. Finally, the first modality feature map with a spatial resolution ratio of 1:4 to the input image is output from the fourth stage of the network. Other high-resolution network architectures can also be used in other embodiments, and this application does not limit them.

[0042] In specific implementation, the corrected second modality image is input into a multi-scale context-aware network, and the extraction of the second modality feature map can be achieved in the following way: a U-Net-like network with skip connections can be used as the multi-scale context-aware network. Its encoder part captures multi-scale context information through four downsampling layers, where each downsampling layer contains two 3x3 convolutional layers and one ReLU activation layer. The decoder part recovers spatial details through the corresponding upsampling layer, and finally outputs a second modality feature map with the same spatial size as the first modality feature map from the end of the decoder. Other multi-scale network architectures can also be used in other embodiments, which are not limited in this application.

[0043] In specific implementation, the dimensional unification and spatial alignment of the extracted first modality feature map and second modality feature map can be achieved in the following way: First, the number of channels of the first modality feature map and the second modality feature map can be unified to 256 dimensions through 1×1 convolution. Then, the spatial resolution of the second modality feature map is adjusted to be completely consistent with that of the first modality feature map through bilinear interpolation, so as to ensure that the two sets of feature maps are aligned in both channel dimension and spatial dimension. In other embodiments, other dimensional unification or spatial alignment methods can also be used, which are not limited in this application.

[0044] In step 105, channel attention fusion is performed on the first modality feature map and the second modality feature map to obtain a fused feature map, and then a spatially consistent radiological diagnostic fused image is reconstructed based on the fused feature map.

[0045] In some embodiments, channel attention fusion of the first modality feature map and the second modality feature map to obtain a fused feature map can be achieved by the following steps: Channel attention weights are calculated for the first modality feature map and the second modality feature map, respectively; The first modality feature map and the second modality feature map are weighted and fused based on the calculated channel attention weights to generate the final fused feature map.

[0046] In specific implementation, the channel attention weights for the first modality feature map and the second modality feature map can be calculated in the following way: First, the first modality feature map and the second modality feature map with input dimensions of H×W×256 are subjected to global average pooling along the spatial dimension to obtain two 256-dimensional channel statistical vectors; then, these two channel statistical vectors are respectively input into a shared weight attention module composed of two fully connected layers, wherein the first fully connected layer compresses the number of channels from 256 to 16 dimensions and uses the ReLU activation function, and the second fully connected layer restores the number of channels from 16 dimensions to 256 dimensions and uses the Sigmoid activation function. Finally, the 256-dimensional channel attention weight vectors with numerical values ​​between 0 and 1 corresponding to the first modality feature map and the second modality feature map are output. In other embodiments, other compression ratios or activation functions can also be used, which are not limited in this application.

[0047] In specific implementation, the first modality feature map and the second modality feature map are weighted and fused based on the calculated channel attention weights to generate the final fused feature map. This can be achieved in the following way: the two channel attention weight vectors are multiplied channel by channel with the corresponding first modality feature map and second modality feature map respectively to obtain two sets of weighted feature maps; then the two sets of weighted feature maps are concatenated along the channel dimension to form a concatenated feature map with 512 channels; finally, a 1×1 convolution operation is applied to the concatenated feature map to fuse the number of channels from 512 and reduce it to 256 dimensions, while using the ReLU activation function to introduce a nonlinear transformation and output the final fused feature map. In other embodiments, other fusion strategies or dimensionality reduction methods can also be used, which are not limited in this application.

[0048] In some embodiments, reconstructing a spatially consistent radiological diagnostic fusion image based on the fusion feature map can be achieved by the following steps: Deep feature extraction and spatial resolution restoration are performed on the fused feature map using an encoder-decoder network; The final radiological diagnostic fusion image is generated through the output layer.

[0049] In specific implementation, the deep feature extraction and spatial resolution restoration of the fused feature map through the encoder-decoder network can be achieved in the following way: the encoder part contains four cascaded residual blocks, each residual block consists of two 3×3 convolutional layers and a skip connection. Downsampling is performed through convolution with a stride of 2, gradually compressing the 256×H×W fused feature map to a deep feature representation of 512×H / 16×W / 16; the decoder part contains four corresponding upsampling modules, each module sequentially performs 2x nearest neighbor upsampling, 3×3 convolution operation, and skip connection concatenation with the corresponding level feature map of the encoder, gradually restoring the spatial resolution of the feature map to the size H×W of the original input image. In other embodiments, upsampling methods such as transposed convolution or bilinear interpolation can also be used, which are not limited in this application.

[0050] In specific implementation, the final radiological diagnostic fusion image can be generated through the output layer in the following way: a 1×1 convolutional layer is used at the end of the decoder to map the number of channels from 256 to 1, corresponding to the generation of a single-channel grayscale fusion image, and the Tanh activation function is used to constrain the pixel values ​​within the range of [-1,1]. Finally, a linear transformation is used to map the output values ​​from [-1,1] back to the standard medical image pixel value range of 0-4095, generating a radiological diagnostic fusion image that retains the spatial details of the high-resolution structural image and fuses the physiological information of the functional image. In other embodiments, the Sigmoid activation function or other pixel value mapping strategies can also be used, which are not limited in this application.

[0051] It should be noted that the radiological diagnostic fusion image in this application integrates the clear anatomical information provided by the first modality high-resolution structural image with the physiological functional information provided by the second modality functional image, while ensuring high spatial consistency, to generate a single image that combines the advantages of both.

[0052] It should also be noted that this application first acquires the patient's CT image as the first modality high-resolution structural image and the PET image as the second modality functional image; secondly, through coarse registration based on rigid transformation and affine transformation, and subsequent fine registration based on the B-spline free deformation model, the two images are spatially aligned; next, the registration quality metric of the finely registered image pair is calculated. The registration quality metric integrates global mutual information and local multi-scale feature similarity. Based on this registration quality metric, a correction strategy is adaptively selected: when the metric is below 0.75, thin plate splines are used for global nonlinear correction; when the registration quality metric meets the standard but local region similarity is insufficient, local B-splines are used for fine adjustment; the corrected PET image and the original CT image are respectively input into the feature extraction networks of HRNet and U-Net architectures, and the extracted feature maps are fused by channel attention; finally, an encoder-decoder reconstruction network generates a CT-PET fused image with significantly enhanced spatial consistency.

[0053] Furthermore, in another aspect of this application, in some embodiments, this application provides a radiological diagnostic image processing system, which includes a multimodal radiological diagnostic image fusion unit, referencing... Figure 4 The figure is a schematic diagram of the structure of a multimodal radiological diagnostic image fusion unit according to some embodiments of this application. The multimodal radiological diagnostic image fusion unit includes: an acquisition module 401, a processing module 402, and an execution module 403, which are described below: The acquisition module 401 in this application is mainly used to acquire a first modal radiographic image and a second modal radiographic image of the same anatomical site of the target patient, wherein the first modal image is a high-resolution structural image and the second modal image is a functional image. Processing module 402, in this application, is mainly used to perform coarse registration of the first modal image and the second modal image based on rigid transformation and affine transformation to obtain coarsely registered image pairs; The processing module 402 described in this application is further configured to perform fine registration using a non-rigid deformation field based on the coarsely registered image pair, to obtain finely registered image pairs, and then determine the registration quality metric between the finely registered image pairs; The processing module 402 described in this application is further configured to perform adaptive spatial transformation correction on the second modality image based on the registration quality metric value, and input the corrected second modality image and the first modality image into the feature extraction network respectively to obtain the first modality feature map and the second modality feature map; The execution module 403 in this application is mainly used to perform channel attention fusion on the first modality feature map and the second modality feature map to obtain a fused feature map, and then reconstruct a spatially consistent radiological diagnostic fused image based on the fused feature map.

[0054] Each module in the aforementioned multimodal radiological diagnostic image fusion unit can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0055] In another embodiment, this application provides a computer device, which may be a server, and its internal structure diagram may be as follows. Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores multimodal radiological diagnostic image fusion data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal radiological diagnostic image fusion method.

[0056] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0057] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described embodiments of the multimodal radiological diagnostic image fusion method.

[0058] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps described in the multimodal radiological diagnostic image fusion method embodiment.

[0059] In one embodiment, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps described in the multimodal radiological diagnostic image fusion method embodiment.

[0060] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0061] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0062] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-modal radiodiagnostic image fusion method applied to a radiodiagnostic image processing system, characterized in that, The method includes the following steps: Acquire a first modal radiographic image and a second modal radiographic image of the same anatomical site of the target patient, wherein the first modal image is a high-resolution structural image and the second modal image is a functional image; Coarse registration is performed on the first modal image and the second modal image based on rigid transformation and affine transformation to obtain coarsely registered image pairs; Based on the coarsely registered image pairs, fine registration is performed using a non-rigid deformation field to obtain finely registered image pairs, and then the registration quality metric between the finely registered image pairs is determined. Based on the registration quality metric, the second modality image is subjected to adaptive spatial transformation correction. The corrected second modality image and the first modality image are then input into the feature extraction network to obtain the first modality feature map and the second modality feature map. Channel attention fusion is performed on the first modality feature map and the second modality feature map to obtain a fused feature map, and then a spatially consistent radiological diagnostic fused image is reconstructed based on the fused feature map.

2. The method as described in claim 1, characterized in that, Coarse registration is performed on the first modal image and the second modal image based on rigid transformation and affine transformation to obtain the coarsely registered image pair, specifically including: Using the first modal image as a fixed image and the second modal image as a moving image, a rigid transformation based on normalized mutual information optimization is performed; Based on the rigid transformation, an affine transformation based on mutual information metric is performed; Output coarsely registered image pairs that have undergone continuous transformations and spatial alignment.

3. The method as described in claim 1, characterized in that, Based on the coarsely registered image pair, fine registration is performed using a non-rigid deformation field to obtain a finely registered image pair, specifically including: The non-rigid deformation field between the coarsely registered image pairs is generated based on the B-spline free deformation model. The second modal image is spatially transformed based on the non-rigid deformation field to obtain a finely registered image pair.

4. The method as described in claim 1, characterized in that, Determining the registration quality metric between the finely registered image pairs specifically includes: Calculate the global intensity similarity metric for the finely registered image pairs; Extract the measure of local multi-scale feature similarity of the finely registered image pairs; The registration quality metric is generated by fusing the global intensity similarity metric with the local multi-scale feature similarity metric.

5. The method as described in claim 1, characterized in that, Based on the registration quality metric, the adaptive spatial transformation correction of the second modality image specifically includes: A correction strategy is selected based on the comparison result between the registration quality metric and the preset threshold. Generate the corresponding spatial deformation field based on the selected correction strategy; The second modal image is spatially transformed based on the spatial deformation field to achieve adaptive correction.

6. The method as described in claim 1, characterized in that, The corrected second modality image and the first modality image are respectively input into the feature extraction network to obtain the first modality feature map and the second modality feature map, specifically including: The first modality image is input into a high-resolution feature preservation network to extract the first modality feature map; The corrected second modality image is input into a multi-scale context-aware network to extract the second modality feature map; The extracted first and second modal feature maps are then subjected to dimensional unification and spatial alignment.

7. The method as described in claim 1, characterized in that, Channel attention fusion is performed on the first modality feature map and the second modality feature map to obtain the fused feature map, specifically including: Channel attention weights are calculated for the first modality feature map and the second modality feature map, respectively; The first modality feature map and the second modality feature map are weighted and fused based on the calculated channel attention weights to generate the final fused feature map.

8. A radiological diagnostic image processing system, comprising a multimodal radiological diagnostic image fusion unit, characterized in that, The multimodal radiological diagnostic image fusion unit includes: The acquisition module is used to acquire a first modal radiographic image and a second modal radiographic image of the same anatomical site of the target patient, wherein the first modal image is a high-resolution structural image and the second modal image is a functional image; The processing module is used to perform coarse registration of the first modal image and the second modal image based on rigid transformation and affine transformation to obtain coarsely registered image pairs; The processing module is further configured to perform fine registration using a non-rigid deformation field based on the coarsely registered image pairs to obtain finely registered image pairs, and then determine the registration quality metric between the finely registered image pairs. The processing module is further configured to perform adaptive spatial transformation correction on the second modal image based on the registration quality metric, and input the corrected second modal image and the first modal image into the feature extraction network respectively to obtain the first modal feature map and the second modal feature map; The execution module is used to perform channel attention fusion on the first modality feature map and the second modality feature map to obtain a fused feature map, and then reconstruct a spatially consistent radiological diagnostic fused image based on the fused feature map.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the multimodal radiological diagnostic image fusion method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the multimodal radiological diagnostic image fusion method as described in any one of claims 1 to 7.

Images