A deep learning-based multi-modal medical image fusion method and system

Abstract

The application provides a kind of multi-modal medical image fusion method and system based on deep learning, it is related to medical image processing technical field, the method includes: obtaining the CT image and PET image to be processed, and extracting target lesion area, to obtain initial image data set;To initial image data set, carry out grey value normalization processing, to obtain standardization image data set;For standardization image data set, the unit pixel grey change of adjacent pixel grey difference of standardization CT lesion area image in target lesion area edge is measured and calculated point by point, to obtain edge change feature set.The application improves the sharpness of lesion edge in multi-modal fusion image and the authenticity of functional expression.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a multimodal medical image fusion method and system based on deep learning. Background Technology

[0002] In the process of multimodal medical image fusion processing, CT images mainly provide density information of human anatomical structures and clearly show the morphological boundaries of organs and lesions, while PET images reflect the intensity and distribution of tissue metabolic functions and are highly sensitive to the physiological activity of lesions. Clinical diagnosis often requires the fusion of the precise anatomical localization capabilities provided by CT and the functional metabolic information provided by PET to improve the accuracy of lesion identification and boundary delineation.

[0003] In practical fusion operations, existing methods typically involve directly overlaying the entire image at the pixel level or performing rigid registration based on fixed transformation parameters before fusion. Such approaches struggle to adequately consider the abrupt changes in anatomical structures at the lesion margins in CT images and the true distribution of metabolic signals in the same region in PET images. This is particularly true in structural transition areas where adjacent pixels at the lesion margin exhibit dramatic changes in grayscale values. Traditional fusion methods are prone to spatial mismatch, causing metabolic signals to either diffuse outwards or excessively contract inwards from the anatomical boundary. This results in blurred lesion margins and distorted metabolic distribution in the fused image, ultimately affecting clinicians' accurate assessment of the lesion's infiltration extent and activity level. Summary of the Invention

[0004] This invention provides a deep learning-based multimodal medical image fusion method and system, which improves the sharpness of lesion edges and the realism of functional expression in multimodal fused images.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] Firstly, a deep learning-based multimodal medical image fusion method is provided, the method comprising:

[0007] Step 1: Acquire the CT and PET images to be processed, and extract the target lesion area to obtain the initial image dataset; perform grayscale value normalization processing on the initial image dataset to obtain the standardized image dataset.

[0008] Step 2: For the standardized image dataset, calculate the unit pixel gray level change of the gray level difference between adjacent pixels at the edge of the target lesion area in the standardized CT lesion area image point by point to obtain the edge change feature set.

[0009] Step 3: Based on the edge change feature set, divide the target lesion area into multiple local sub-regions;

[0010] Step 4: For each local sub-region, accumulate and calculate the global total metabolic signal intensity integral value distribution of the corresponding region in the PET image to obtain the regional metabolic signal intensity integral value feature set;

[0011] Step 5: Extract the spatial coordinate information from the edge change feature set and the regional metabolic signal intensity integral value feature set, and perform spatial alignment operation to obtain the joint feature set;

[0012] Step 6: Based on the joint feature set, decompose and strip the deep structural features of the multimodal image to obtain the fused feature matrix;

[0013] Step 7: Based on the fused feature matrix, perform pixel-level weight allocation to obtain the weighted feature matrix;

[0014] Step 8: Based on the weighted feature matrix and combined with the original spatial coordinate system, reconstruct and generate multimodal medical fusion images.

[0015] Secondly, a deep learning-based multimodal medical image fusion system includes:

[0016] The image acquisition and preprocessing module is used to acquire CT and PET images to be processed, extract the target lesion area, and obtain an initial image dataset; grayscale value normalization processing is performed on the initial image dataset to obtain a standardized image dataset;

[0017] The edge change feature extraction module is used to measure the unit pixel gray level change of the gray level difference between adjacent pixels at the edge of the target lesion area in a standardized CT lesion area image point by point for a standardized image dataset, and obtain the edge change feature set.

[0018] The local sub-region segmentation module is used to segment the target lesion area into multiple local sub-regions based on the edge change feature set;

[0019] The regional metabolic signal intensity integral value feature calculation module is used to accumulate and calculate the global total regional metabolic signal intensity integral value distribution of the corresponding region in the PET image for each local sub-region, and obtain the regional metabolic signal intensity integral value feature set.

[0020] The joint feature alignment module is used to extract the spatial coordinate information of the edge change feature set and the regional metabolic signal intensity integral value feature set, and perform spatial alignment operation to obtain the joint feature set;

[0021] The deep feature decomposition and fusion module is used to decompose and peel off the deep structural features of multimodal images based on the joint feature set to obtain a fused feature matrix;

[0022] The pixel-level weight allocation module is used to perform pixel-level weight allocation operations based on the fused feature matrix to obtain the weighted feature matrix;

[0023] The fusion image reconstruction module is used to reconstruct and generate multimodal medical fusion images based on the weighted feature matrix and the original spatial coordinate system.

[0024] The above-described solution of the present invention has at least the following beneficial effects:

[0025] By extracting the unit pixel grayscale change of adjacent pixels at the lesion edge, a refined edge change feature set is constructed. Simultaneously, the distribution of the total metabolic signal intensity integral value across the entire region of the PET image is calculated. After spatial coordinate alignment, depth feature decomposition and pixel-level weight allocation are performed. This process ensures the complete preservation of high-frequency details at the edges of anatomical structures in the fused image, while precisely constraining metabolic function signals within the true anatomical boundaries. This overcomes the boundary blurring and metabolic distortion problems caused by existing methods in edge transition areas, improving the sharpness of lesion edges and the realism of functional expression in multimodal fused images. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating a deep learning-based multimodal medical image fusion method provided by an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of a deep learning-based multimodal medical image fusion system provided by an embodiment of the present invention. Detailed Implementation

[0028] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art.

[0029] like Figure 1 As shown, an embodiment of the present invention proposes a multimodal medical image fusion method based on deep learning, the method comprising the following steps:

[0030] Step 1: Acquire the CT and PET images to be processed, and extract the target lesion area to obtain the initial image dataset; perform grayscale value normalization processing on the initial image dataset to obtain the standardized image dataset.

[0031] Step 2: For the standardized image dataset, calculate the unit pixel gray level change of the gray level difference between adjacent pixels at the edge of the target lesion area in the standardized CT lesion area image point by point to obtain the edge change feature set.

[0032] Step 3: Based on the edge change feature set, divide the target lesion area into multiple local sub-regions;

[0033] Step 4: For each local sub-region, accumulate and calculate the global total metabolic signal intensity integral value distribution of the corresponding region in the PET image to obtain the regional metabolic signal intensity integral value feature set;

[0034] Step 5: Extract the spatial coordinate information from the edge change feature set and the regional metabolic signal intensity integral value feature set, and perform spatial alignment operation to obtain the joint feature set;

[0035] Step 6: Based on the joint feature set, decompose and strip the deep structural features of the multimodal image to obtain the fused feature matrix;

[0036] Step 7: Based on the fused feature matrix, perform pixel-level weight allocation to obtain the weighted feature matrix;

[0037] Step 8: Based on the weighted feature matrix and combined with the original spatial coordinate system, reconstruct and generate multimodal medical fusion images.

[0038] In this embodiment of the invention, lesion extraction and grayscale normalization are performed on CT and PET images to eliminate intermodal scale differences. Then, the unit pixel grayscale change of adjacent pixels at the lesion edge is calculated, and sub-regions are divided based on edge change characteristics to achieve adaptive partitioning based on structural transition characteristics. On this basis, the regional distribution of the integral value of metabolic signal intensity in the PET metabolic region is calculated and associated with spatial coordinates to form a joint feature set aligned with structure and function. A deep feature decomposition network is used to extract and fuse the deep features of anatomical structure and functional metabolism. Then, a pixel-level weight allocation network generates adaptive modal fusion weights for each pixel. Finally, the fused image is reconstructed in the original spatial coordinate system and boundary smoothing is performed. This overall method deeply fuses the anatomical structural mutation information and regional metabolic function information at the pixel level, solving problems such as blurred lesion edges and distorted metabolic signals crossing anatomical boundaries in traditional fusion. It simultaneously and clearly presents the precise anatomical boundaries and true metabolic activity of the lesion on the same image, improving the fusion quality and clinical diagnostic value of multimodal medical images.

[0039] In a preferred embodiment of the present invention, step 1 above may include:

[0040] Step 1.1: Perform spatial registration on the acquired CT and PET images of the same subject to obtain a spatially aligned bimodal image pair. Specifically, this includes: performing spatial registration on the acquired CT and PET images of the same subject to obtain a spatially aligned bimodal image pair. Using CT images as reference images and PET images as images to be registered, rigid body transformations are performed on both images. The initial transformation parameters include translation along the three coordinate axes and rotation angles around the three coordinate axes. Normalized mutual information (MFI) is used as a similarity measure to assess the spatial alignment of the two images. MFI is determined by the gray-level entropy of each image and their joint gray-level entropy. It is calculated by dividing the sum of the gray-level entropies of the two images by their joint gray-level entropy. The closer this ratio is to the preset upper limit, the higher the spatial alignment of the two images. The rigid body transformation parameters of the PET image are iteratively adjusted, and the MFI is recalculated after each adjustment. When the MFI reaches its maximum value, the corresponding rigid body transformation parameters are determined as the registration transformation parameters. Based on the registration transformation parameters, the PET image is spatially resampled to align it pixel-by-pixel with the CT image in spatial position, resulting in a spatially aligned bimodal image pair.

[0041] Step 1.2: Based on the spatially aligned bimodal image pairs, a pre-trained lesion segmentation deep learning model is invoked to identify and extract lesion regions from CT and PET images respectively, resulting in CT lesion region images and PET lesion region images, which together constitute the initial image dataset. Specifically, this includes: based on the obtained spatially aligned bimodal image pairs, a pre-trained lesion segmentation deep learning model is invoked to identify and extract lesion regions from CT and PET images respectively, resulting in CT lesion region images and PET lesion region images, which together constitute the initial image dataset.

[0042] The pre-trained deep learning model for lesion segmentation employs a residual U-Net network, which incorporates a residual connection module based on the U-Net architecture. The encoder portion of this network contains five downsampling stages, each consisting of two convolutional layers and a max-pooling layer. The kernel size is preset, and the activation function is a modified linear unit function. A residual connection path connects adjacent downsampling stages, element-wise adding the output of the previous stage to the output processed by the two convolutional layers in the current stage to mitigate gradient decay during training of deep networks. The decoder portion contains four upsampling stages, each consisting of an upsampling layer, a feature concatenation layer, and two convolutional layers. The feature concatenation layer concatenates the feature map of the corresponding resolution stage in the encoder with the feature map of the current stage in the decoder along the channel dimension, enabling the decoder to utilize the shallow detail information extracted by the encoder during spatial resolution restoration.

[0043] During the model training phase, CT and PET images with lesion boundaries annotated pixel-by-pixel by radiologists were collected from a historical case database as training datasets. During training, a weighted combination of the Dice coefficient and binary cross-entropy was used as the loss function. The Dice coefficient was calculated as twice the number of elements in the intersection of the predicted result and the ground truth annotation, divided by the sum of the number of elements in both. The binary cross-entropy was used to measure the classification error between the pixel-by-pixel predicted probability and the ground truth annotation. A balance coefficient in the loss function was used to adjust the contribution ratio between the two loss terms. After training, spatially aligned bimodal image pairs were input into the residual U-Net network, which output binary segmentation masks for CT and PET lesions. A pixel-by-pixel logical AND operation was performed between the CT image and its corresponding binary segmentation mask to extract the CT lesion region image; a pixel-by-pixel logical AND operation was performed between the PET image and its corresponding binary segmentation mask to extract the PET lesion region image. The CT and PET lesion region images together constituted the initial image dataset.

[0044] Step 1.3: For the CT lesion region images and PET lesion region images in the initial image dataset, calculate the mean and standard deviation of their respective pixel grayscale values. Based on the mean and standard deviation, map the pixel grayscale values ​​of each image to a preset numerical range to obtain the standardized image dataset. Specifically, this includes: calculating the mean and standard deviation of the pixel grayscale values ​​of the CT lesion region images and PET lesion region images in the obtained initial image dataset, and mapping the pixel grayscale values ​​of each image to a preset numerical range based on the mean and standard deviation to obtain the standardized image dataset. For the CT lesion region image, iterate through the grayscale values ​​of all pixels within the lesion region, sum the grayscale values ​​of all pixels, and divide by the total number of pixels to obtain the CT grayscale mean. The calculation method is as follows: ;

[0045] in, This represents the total number of pixels within the lesion area. Indicates the first The original grayscale value of each pixel. This represents the arithmetic mean of the grayscale values ​​of all pixels in the lesion area.

[0046] Based on the mean grayscale value of CT, the square of the difference between the grayscale value of each pixel and the mean grayscale value of CT is calculated. The sum of these squared values ​​for all pixels is then divided by the total number of pixels, and the square root is taken to obtain the standard deviation of the CT grayscale value. The calculation method is as follows: ;

[0047] in, and The meaning is the same as the previous formula. This represents the standard deviation of the grayscale values ​​of all pixels in the lesion area. This represents the total number of pixels within the lesion area.

[0048] Based on the CT gray mean and CT gray standard deviation, Z-score standardization is performed on each pixel in the CT lesion area image: the difference between the original gray value of each pixel and the CT gray mean is divided by the CT gray standard deviation, so that the standardized pixel gray value follows a distribution with a mean of zero and a standard deviation of one. Using the same method as the CT lesion area image, the PET gray mean and PET gray standard deviation are calculated for the PET lesion area image, and Z-score standardization is performed on each pixel in the PET lesion area image.

[0049] The standardized CT lesion region images and the standardized PET lesion region images are merged to obtain a standardized image dataset. After the above standardization process, the pixel gray values ​​of the dual-modal images are converted into dimensionless standard scores, and the differences in gray value magnitude and distribution range between CT images and PET images caused by differences in imaging equipment type and scanning protocol parameters are eliminated.

[0050] In this embodiment of the invention, spatial registration of CT and PET images is performed in advance to eliminate spatial positional deviations between the two modal images. A deep learning model is then invoked to automatically perform lesion region identification and extraction, avoiding the subjectivity and inefficiency of manually delineating lesion boundaries. Based on this, normalization processing is performed according to the mean and standard deviation of each image to eliminate the overall shift and dynamic range differences in grayscale values ​​caused by different imaging devices and different scanning parameters, so that the lesion region has a consistent numerical distribution scale in the two modal images.

[0051] In a preferred embodiment of the present invention, step 2 above may include:

[0052] Step 2.1: Based on the CT lesion region images in the standardized image dataset, an edge detection operator is applied to extract the boundary contour lines of the target lesion region, obtaining a set of lesion edge pixels. Specifically, this includes: processing with the Sobel edge detection operator, performing two-dimensional convolution operations on the standardized CT lesion region images using horizontal and vertical Sobel convolution kernels respectively. The horizontal Sobel convolution kernel is a 3x3 weight matrix, with negative weights in the left column, positive weights in the right column, and zero weights in the middle column, responding to changes in grayscale in the horizontal direction; the vertical Sobel convolution kernel is a 3x3 weight matrix, with negative weights in the top row, positive weights in the bottom row, and zero weights in the middle row, responding to changes in grayscale in the vertical direction.

[0053] After the convolution operation, the first-order grayscale value along the horizontal direction at each pixel is obtained. and the first derivative value of gray level along the vertical direction The gradient magnitude of a pixel is obtained by squaring the horizontal and vertical derivatives, adding them together, and then taking the square root. gradient magnitude The calculation method is as follows: ;

[0054] in, This represents the first derivative of the grayscale value at the pixel along the horizontal direction. This represents the first derivative of the grayscale value at a pixel along the vertical direction. This represents the gradient magnitude at that pixel. The larger the gradient magnitude, the more drastic the spatial change in grayscale values ​​within the pixel's neighborhood, and the higher the probability that the pixel is located at the boundary of a lesion.

[0055] Set gradient magnitude threshold The gradient magnitude exceeds Pixels with gradient magnitudes not exceeding a certain threshold are marked as candidate edge pixels. Pixels that are not edge pixels are marked as such. Non-maximum suppression is then performed on the binarization result: centering on each candidate edge pixel, along the gradient direction of that pixel, ... and Within a pixel width determined by the arctangent of the ratio, the gradient magnitude of the pixel is compared with that of its preceding and following pixels. If the gradient magnitude of the pixel is not a local maximum in that direction, it is suppressed as a non-edge pixel. Candidate edge pixels with local maximum gradient magnitudes are retained, so that the detected edge lines are thinned to a single pixel width.

[0056] After performing nonmaximum suppression, a double-threshold hysteresis connection is further applied: a high threshold is set. and low threshold The gradient magnitude exceeds The pixels that are identified as strong edge pixels are set to have gradient magnitudes between [values ​​missing]. and Pixels between these points are identified as weak edge pixels, and those with gradient magnitudes lower than these are considered weak edge pixels. Pixels with strong edges are discarded. All strong edge pixels are traversed, and weak edge pixels connected in eight neighborhood directions are traced pixel by pixel. The traced weak edge pixels are added to the edge pixel set, thus obtaining the set of lesion edge pixels that constitute the entire boundary contour of the target lesion region, denoted as . .

[0057] Step 2.2: Based on the set of pixels at the lesion edge, for each edge pixel, calculate the gray-level difference between adjacent pixels along the boundary normal direction to obtain the gray-level jump variable corresponding to each edge pixel. Specifically, this includes: for the set of pixels at the lesion edge... any edge pixel in Determine the boundary normal direction at the edge pixel. Boundary normal direction angle. The calculated horizontal derivative value and vertical derivative value The arctangent of the ratio is determined as follows: ;

[0058] in, Represents edge pixels The direction angle of the boundary normal at that location. and The values ​​are determined by the Sobel convolution results at the pixel in step 2.1.

[0059] After determining the boundary normal direction, use the edge pixels. Taking the origin as the reference point, and following the positive normal direction from the inside of the lesion area outwards, select a pixel at a step size, denoted as . Along the negative normal direction, from the outside of the lesion area inward, take a pixel at a step size, and denote it as... Pixels were extracted from standardized CT images of lesion areas. grayscale value and pixels grayscale value Calculate the absolute value of the difference between the two to obtain the edge pixel. The corresponding grayscale jump variable is denoted as Gray-scale jump variable The calculation method is as follows: ;

[0060] in, Represents pixels in a standardized CT lesion region image. grayscale value, Represents pixels in a standardized CT lesion region image. grayscale value, Represents edge pixels The grayscale jump variable. This grayscale jump variable reflects the magnitude of the jump in CT grayscale value when crossing the boundary of the lesion: A larger value indicates a greater difference in tissue density between the lesion tissue and the surrounding normal tissue at this boundary location, meaning that the anatomical boundary is relatively clear. A smaller value indicates that the difference in tissue density between the lesion and the surrounding normal tissue at this boundary location is relatively weak, that is, the anatomical boundary is relatively blurred or the lesion exhibits an infiltrative growth pattern.

[0061] Traverse the set of pixels at the edge of the lesion Perform the above operation on each edge pixel to obtain the grayscale jump variable corresponding to each edge pixel.

[0062] Step 2.3: Based on the grayscale jump variables corresponding to each edge pixel and the spatial distance between adjacent pixels, calculate the instantaneous rate of change of the grayscale difference at each edge pixel as a function of its spatial location. Associate and record the spatial coordinates of each edge pixel with its instantaneous rate of change to obtain the edge change feature set. Specifically, this includes: based on the obtained grayscale jump variables corresponding to each edge pixel... By combining the spatial distance between adjacent pixels, the instantaneous rate of change of grayscale difference at each edge pixel with spatial position is calculated. The spatial coordinates of each edge pixel are associated with the instantaneous rate of change of that edge pixel to obtain the edge change feature set.

[0063] Adjacent pixels and Spatial distance between Defined as the Euclidean distance between the two points in the image space. The calculation method is as follows: ;

[0064] in, Represents pixels spatial coordinates, Represents pixels spatial coordinates, It represents the spatial distance between two adjacent pixels along the boundary normal direction, with the dimension being pixels.

[0065] After obtaining the grayscale jump variable Spatial distance between adjacent pixels Next, calculate the edge pixels. unit pixel grayscale change , The calculation method is as follows: ;

[0066] in, For the calculated edge pixels grayscale jump variable, The spatial distance between adjacent pixels. Represents edge pixels The change in grayscale value per unit pixel at a given location. This change in grayscale value per unit pixel reflects the degree of change in CT grayscale value per unit spatial distance when crossing the boundary of a lesion. A larger value indicates that the tissue density transition at the boundary of the lesion is steeper, that is, the boundary between the lesion and normal tissue is sharper; A smaller value indicates that the transition of tissue density at the boundary of the lesion is relatively gradual, meaning that the lesion may be invasive or the difference in tissue density at that location may be small.

[0067] Set of pixels at the edge of the lesion Each edge pixel in The above calculations are performed on all edge pixels to obtain the unit pixel grayscale change for each edge pixel. The spatial coordinates of each edge pixel. The change in grayscale per unit pixel of the edge pixel Establish connections to form triples Let be a set of elements, which is the edge change feature set, denoted as . .

[0068] In this embodiment of the invention, an edge detection operator is applied to a standardized CT lesion region image to accurately locate the lesion boundary contour. For each edge pixel, the gray-level difference between adjacent pixels along its normal direction and the instantaneous rate of change of this difference with spatial distance are calculated. This numerically and precisely characterizes the degree of gray-level transition from the inside to the outside of the lesion edge structure. This approach overcomes the shortcomings of traditional gradient operators, which only provide the magnitude of directional changes, enabling high-resolution quantitative expression of the sharpness of edge transition zones and the abrupt changes in local anatomical structures.

[0069] In a preferred embodiment of the present invention, step 3 above may include:

[0070] Step 3.1: Based on the instantaneous rate of change value carried by each edge pixel in the edge change feature set, preset at least two rate of change threshold intervals, and group edge pixels whose instantaneous rate of change values ​​fall within the same threshold interval into the same edge group, resulting in multiple edge groups. Specifically, this includes: based on the obtained edge change feature set... The value of the change in grayscale per unit pixel carried by each edge pixel. At least two rate of change threshold ranges are preset. Edge pixels whose unit pixel grayscale change value falls into the same threshold range are grouped into the same edge group to obtain multiple edge groups.

[0071] Preset The rate of change thresholds, arranged in ascending order, are denoted as... ,in It is an integer not less than 2. Each threshold is used to divide There are three non-overlapping rate-of-change threshold intervals, namely the first interval (from negative infinity to...). ), second interval (from to ), and so on up to the 1st interval (from) (To positive infinity). The boundary values ​​of each threshold interval are preset according to the lesion type and imaging parameters.

[0072] Traversing the edge change feature set All elements in the middle, for each edge pixel. and the corresponding change in grayscale per unit pixel ,Will The value is compared with the upper and lower limits of each threshold interval to determine... The threshold range that the value falls into. The value of the change in grayscale per unit pixel falling into the threshold range. All edge pixels in the threshold interval are classified into the first threshold interval. An edge group, denoted as After the above traversal and grouping operations, we obtain... Edge grouping .

[0073] Edge pixels belonging to the same edge group have unit pixel grayscale change values ​​falling within the same threshold range, indicating that the grayscale transition steepness at the boundary locations of these edge pixels is within the same range, meaning that the clarity of the CT anatomical structure boundaries is similar. Edge pixels belonging to different edge groups have unit pixel grayscale change values ​​falling within different threshold ranges, indicating that there are interval differences in the tissue density change characteristics at different boundary locations.

[0074] Step 3.2: Based on the spatial contour range enclosed by the edge pixels contained in each of the multiple edge groups, determine the closed region defined by each edge group, and divide the target lesion region into multiple local sub-regions defined by each edge group. Specifically, this includes: based on the obtained multiple edge groups... The spatial contour range enclosed by the edge pixels contained in each edge group is used to determine the closed region defined by each edge group, thus dividing the target lesion region into multiple local sub-regions defined by each edge group.

[0075] With the first Edge grouping For example, the edge pixels in this group are spatially scattered across different arc segments of the lesion boundary. Spatial connectivity analysis is performed on all edge pixels in this group: using eight-neighbor connectivity as the criterion, spatially adjacent edge pixels are connected into continuous contour arc segments. Each contour arc segment consists of a set of edge pixels whose spatial coordinates are connected end-to-end. Therefore, the... Edge grouping It is decomposed into several contour arc segments.

[0076] Will belong to the Edge grouping All contour arcs and those belonging to adjacent edge groups and The closed spatial region formed by the contour arc segments is defined as the first... A local subregion, denoted as For the first edge group Its outer side is adjacent to the region outside the lesion boundary, and its outer boundary is the set of pixels at the edge of the lesion. The outline is directly determined; for the first... Edge grouping Its inner side is adjacent to the internal region of the lesion, and its inner boundary is determined by the innermost contour line of the lesion. The enclosure method of the closed region is: the first... The contour arcs of each edge group extend along the lesion boundary until they intersect with the contour arcs of adjacent edge groups or connect with a preset spatial boundary line, forming a spatially connected closed loop or band-shaped region. The boundary line between adjacent local sub-regions is determined by a corresponding rate of change threshold. The spatial location of the associated edge pixels determines the enclosure operation, which is implemented using morphological closing operations: first, each edge is grouped. The contour arc segments are expanded to connect with the contour arc segments of adjacent groups or preset spatial boundaries, and then erosion is performed to finally obtain a connected and closed region mask.

[0077] After the above segmentation process, the target lesion area is decomposed into: Local sub-regions ,in The value is determined by a preset number of change rate thresholds. Together with the actual distribution of edge groups, each local sub-region is determined. At its boundary arc segment, by the first The edge pixels of the edge group represent its boundary features, that is, the local sub-region has the same characteristics as the first edge pixel at the corresponding boundary position. The consistent CT grayscale transition characteristics of pixels within each edge group, from the perspective of the geometric hierarchy of the lesion, indicate that local sub-regions located inside the lesion usually correspond to a gentler grayscale transition, while local sub-regions located at the edge of the lesion usually correspond to a steeper grayscale transition, thus forming a progressive layered structure from the inside out.

[0078] In this embodiment of the invention, multiple threshold intervals are set based on the instantaneous rate of change, and edge pixels are grouped to define multiple local sub-regions. The beneficial effect is that the lesion edge is automatically divided into different levels of functional-structural response zones according to the severity of structural changes. This division method is not a simple spatial grid segmentation, but rather a differentiated partitioning based on the actual anatomical boundary transition characteristics. This allows subsequent analysis of metabolic signal intensity integral values ​​in metabolic regions to be performed independently for regions with different edge sharpness, avoiding the confusion between rapidly changing hard boundary regions and slowly transitioning blurred boundary regions, thus improving the physiological rationality and targeted processing of sub-region division.

[0079] In a preferred embodiment of the present invention, step 4 above may include:

[0080] Step 4.1: Based on the spatial range defined by each of the multiple local sub-regions, extract the metabolic signal intensity values ​​of all pixels within the corresponding spatial location in the PET lesion area image of the standardized image dataset, and obtain the metabolic signal intensity distribution data of each local sub-region. Specifically, this includes: taking the first... Local sub-regions For example, based on the determined spatial extent of the local sub-region in the lesion image, the same spatial coordinate region is located in the standardized PET lesion region image. All pixels within this spatial coordinate region are traversed, and their grayscale values ​​in the standardized PET lesion region image are extracted pixel by pixel. This grayscale value is the metabolic signal intensity value at the corresponding pixel, denoted as... ,in These are the spatial coordinates of the pixel.

[0081] Local sub-regions Spatial coordinates of all pixels Its corresponding metabolic signal intensity value The metabolic signal intensity distribution data constituting this local sub-region are saved one by one and recorded as follows: , Each record contains the spatial coordinates of a pixel and the metabolic signal intensity value at that coordinate. For all... Perform the above region-by-region extraction operation on each local sub-region one by one to obtain the metabolic signal intensity distribution data of each local sub-region. .

[0082] Step 4.2: Based on the metabolic signal intensity distribution data of each local sub-region, perform a region-by-region cumulative calculation on the metabolic signal intensity values ​​of all pixels in each local sub-region to obtain the total regional metabolic signal intensity integral value corresponding to each local sub-region. Specifically, this includes: for the... Local sub-regions The integral value of its total regional metabolic signal intensity Defined as the algebraic sum of the metabolic signal intensity values ​​of all pixels within this local sub-region. The calculation method is as follows: ;

[0083] in, Indicates falling into a local subregion Pixel coordinates within a spatial range This represents the pixels extracted in step 4.1. The metabolic signal intensity value at that location, Indicates the first The total regional metabolic signal intensity integral value of each local sub-region. Reflecting local sub-regions The cumulative level of internally standardized PET signal intensity can serve as a statistical characteristic describing the spatial distribution pattern of PET signals.

[0084] Due to local sub-regions The spatial extent is determined by step 3.2, and its area depends on the distribution range of the edge groups corresponding to the local sub-region on the lesion boundary. The areas of different local sub-regions may be different, therefore the total regional metabolic signal intensity integral value This comprehensively reflects two factors: the size of a local sub-region and the average level of metabolic signal intensity within that region, for the entire... The above region-by-region cumulative calculation operation is performed on each local sub-region to obtain the total regional metabolic signal intensity integral value corresponding to each local sub-region.

[0085] Step 4.3 involves associating the total regional metabolic signal intensity integral value of all local sub-regions with the spatial identifier of each local sub-region to obtain a feature set of regional metabolic signal intensity integral values. Specifically, this includes: for each local sub-region... Assign a space identifier, denoted as The spatial identifier is determined by the sequence number of the edge group to which the local sub-region belongs, as described in step 3.2. It can be determined by the spatial hierarchical position of the local sub-region within the target lesion region.

[0086] Spatial identifiers Spatial range of local sub-regions And the total regional metabolic signal intensity integral value of this local sub-region. The three are linked together to form a triplet. All The ternary combinations of each local sub-region are merged into a set to obtain the feature set of regional metabolic signal intensity integral values, denoted as . ,Right now: ;

[0087] The feature set of integral values ​​of metabolic signal intensity in this region fully records the following three types of information for each local sub-region within the target lesion area: spatial identifier, used to distinguish different local sub-regions and provide an index for region reference in subsequent processing steps; spatial range, that is, the set of pixel coordinates covered by the local sub-region in the image space; and total regional metabolic signal intensity integral value, that is, the sum of PET metabolic signal intensity within the local sub-region calculated.

[0088] In this embodiment of the invention, local sub-regions are used as basic units, and the global total metabolic signal intensity integral value distribution of metabolic signal intensity values ​​in PET images is calculated by accumulating the values ​​from region to region, directly linking the structural zoning results with functional metabolic intensity. The beneficial effect is that the total regional metabolic signal intensity integral value of each local sub-region comprehensively reflects the overall level of metabolic activity of all tissues within that region, rather than relying solely on the isolated signal values ​​of individual pixels. Therefore, it can more stably and completely express the metabolic activity level corresponding to regions with different edge characteristics, and suppress the interference of local noise in PET images on the value of a single pixel.

[0089] In a preferred embodiment of the present invention, step 5 above may include:

[0090] Step 5.1: Based on the edge change feature set, extract the spatial coordinate information of each edge pixel in the edge change feature set, and construct an edge feature spatial coordinate mapping table indexed by spatial coordinates. Specifically, this includes: the edge change feature set. Each record in the array is a triple. ,in edge pixels spatial coordinates, The value representing the change in grayscale per unit pixel corresponding to the edge pixel is traversed. All records in the database will contain the spatial coordinates of each edge pixel. As an index key, the unit pixel grayscale change corresponding to this pixel is used. As mapping values, a one-to-one key-value mapping relationship is established. This key-value mapping relationship is then arranged and organized according to the lexicographical order of spatial coordinates to form an edge feature spatial coordinate mapping table, denoted as... For mapping tables any spatial coordinate in The change in grayscale value per unit pixel at a given coordinate location can be directly obtained through coordinate lookup. If a certain spatial coordinate is in If there is no corresponding record in the mapping table, meaning there are no edge pixels at that coordinate, then that coordinate is not in the mapping table. The corresponding mapping value in the table is empty.

[0091] Step 5.2: Based on the regional metabolic signal intensity integral value feature set, extract the spatial identifier and corresponding spatial range information of each local sub-region in the regional metabolic signal intensity integral value feature set, and construct a regional metabolic signal intensity integral value feature spatial coordinate mapping table indexed by the spatial identifier. Specifically, this includes: the regional metabolic signal intensity integral value feature set. Each record in the array is a triple. ,in For the first Spatial identifiers for local sub-regions This refers to the spatial extent of the local sub-region, that is, the set of pixel coordinates covered by this region. This is the integral value of the total regional metabolic signal intensity for this local sub-region. For each record in the local sub-region The coordinates of all covered pixels are integrated with the total metabolic signal intensity of the local sub-region. Establish a connection.

[0092] A spatial coordinate mapping table of regional metabolic signal intensity integral values ​​is constructed using the following method, denoted as: For each local sub-region Iterate through all the pixel coordinates it covers. , pixel coordinates As the index key, the total regional metabolic signal intensity integral value of this local sub-region is used. As a mapping value, the coordinates of all pixels within the same local sub-region are thus mapped in the mapping table. The integral value of metabolic signal intensity will be mapped to the same total region. That is, the integral value of the regional metabolic signal intensity feature at each pixel position within the same local sub-region is the same, and all of them are considered to be the same. After processing each local sub-region in the above manner, a complete regional metabolic signal intensity integral value feature spatial coordinate mapping table is obtained. This mapping table stores the feature information of the integral value of the metabolic signal intensity in the region at the pixel coordinate level, so that each pixel position can query the total integral value of the metabolic signal intensity in its local sub-region.

[0093] Step 5.3: Based on the spatial coordinate mapping table of edge features and the spatial coordinate mapping table of regional metabolic signal intensity integral value features, and using spatial coordinate consistency as the alignment criterion, the edge change feature data and regional metabolic signal intensity integral value feature data at the same spatial location are paired and merged one by one to obtain the joint feature set. Each record in the joint feature set includes spatial coordinates, the instantaneous rate of change value corresponding to the coordinates, and the total regional metabolic signal intensity integral value of the local sub-region to which the coordinates belong. Specifically, this includes traversing all pixel coordinates within the target lesion region. For each pixel coordinate, simultaneously look up the mapping table. and mapping table :from Get the unit pixel grayscale change at this coordinate position. If there are no edge pixels at that coordinate, then The value of is zero; from The integral value of the total regional metabolic signal intensity of the local sub-region to which the coordinate belongs is obtained. If the coordinates are not within any local sub-region, i.e., outside the lesion area, then The value of is zero if a certain pixel coordinate It belongs to multiple local sub-regions. Then, the attribution is determined according to the priority of the edge group to which it belongs: it is given priority to be assigned to the edge group with the smaller sequence number, and the pixels on the boundary of the region are assigned to its adjacent local sub-region with the smaller sequence number.

[0094] Same pixel coordinates The values ​​retrieved from the two mapping tables are paired and merged to form a joint feature record. Each joint feature record is a quadruple. It contains the following four types of information: spatial coordinates Used to uniquely identify the location of the record in the image space; unit pixel grayscale change The value calculated in step 2.3 and provided by the mapping table in step 5.1 represents the steepness of the gray-level transition of the CT anatomical structure boundary at the pixel location; the total regional metabolic signal intensity integral value. The value is calculated in step 4.2 and provided by the mapping table in step 5.2, representing the overall PET functional metabolic activity level of the local sub-region to which the pixel location belongs.

[0095] Merge the joint feature records corresponding to all pixel coordinates into a single set, resulting in the joint feature set, denoted as . Joint feature set It fully carries the dual-modal joint information of the target lesion area at the pixel-level spatial granularity. Each pixel location simultaneously carries the CT edge transition features and the PET region metabolic signal intensity integral value features.

[0096] In this embodiment of the invention, an edge feature spatial coordinate mapping table and a regional metabolic signal intensity integral value feature spatial coordinate mapping table are constructed separately. These two tables are then paired and merged one by one using coordinate consistency as the alignment criterion. This ensures that pixels with completely overlapping spatial locations simultaneously carry the edge instantaneous change rate information of that point and the total regional metabolic signal intensity integral value information of its sub-region. This alignment method eliminates coordinate misalignment or index deviation that may occur due to separate feature extraction, ensuring that each record in the joint feature set is a strictly corresponding combination of structural and functional features at the same spatial location.

[0097] In a preferred embodiment of the present invention, step 6 above may include:

[0098] Step 6.1: Based on the joint feature set, convert the joint feature set into a multidimensional feature tensor, which serves as the input data for the pre-trained deep feature decomposition network. The deep feature decomposition network employs an encoder-decoder architecture, specifically including: the joint feature set... Each joint feature record is a quadruple. .extract All spatial coordinates recorded Determine the maximum horizontal span of the target lesion region in the imaging space. and the maximum span in the vertical direction To establish the size as OK The spatial grid of columns. For each grid cell in the spatial grid. ,in Indicates row index, Represents the column index, representing the change in grayscale value per unit pixel in the joint feature set U. and the integral value of total regional metabolic signal intensity Independent Min-Max normalization was performed on each model to map their values ​​uniformly to the [0, 1] interval, thus eliminating the impact of dimensional differences on subsequent deep learning model training. Find the corresponding joint feature record in the middle:

[0099] If the spatial coordinates of the grid cell are in If a record exists, then the change in grayscale value per unit pixel in that record will be used. and the integral value of total regional metabolic signal intensity Assign values ​​to the two feature channels of the grid cell respectively; if the spatial coordinates of the grid cell are in If no record exists in the grid cell, meaning the grid cell corresponds to an area outside the lesion region or a missing location, then both feature channel values ​​will be assigned to zero.

[0100] Stack the two feature channel values ​​of all grid cells along the channel dimension to form a structure of size [size missing]. The multidimensional feature tensor, denoted as ,in The input is the number of feature channels, with a value of 2. and These represent the pixel span of the lesion area in the vertical and horizontal directions, respectively.

[0101] The deep feature decomposition network adopts an encoder-decoder architecture, specifically a dual-branch decoder network that introduces a channel separation mechanism on the basis of the U-Net architecture.

[0102] The encoder section consists of four downsampling stages:

[0103] The first downsampling stage consists of two convolutional layers. Each convolutional layer uses a preset number of convolutional kernels of a preset size, with a preset stride. The activation function is a modified linear unit function.

[0104] The second downsampling stage consists of two convolutional layers, with the number of convolutional kernels doubled compared to the first downsampling stage. It is followed by a max pooling layer with a preset size, which compresses the spatial resolution to half of the original.

[0105] The third downsampling stage consists of two convolutional layers, with the number of convolutional kernels increasing to twice that of the second downsampling stage, followed by a max pooling layer, which compresses the spatial resolution to half of its original value.

[0106] The fourth downsampling stage consists of two convolutional layers, with the number of convolutional kernels doubled compared to the third downsampling stage, followed by a max pooling layer, which compresses the spatial resolution for the third time.

[0107] Batch normalization layers are inserted between each downsampling stage to stabilize the distribution of activation values ​​during training.

[0108] The decoder is divided into two parallel decoding branches at the bottom of the encoder: the first decoding branch (CT anatomical texture channel group) and the second decoding branch (PET metabolic intensity channel group). The two decoding branches are symmetrical in structure and each contains three upsampling stages. Each upsampling stage first restores the spatial resolution to twice its original value through an upsampling layer. Then, a feature stitching layer stitches the feature map of the corresponding downsampling stage in the encoder with the feature map of the current upsampling stage along the channel dimension. Finally, two convolutional layers are used for feature fusion. The two decoding branches gradually differentiate during training: the first decoding branch learns to extract feature representations related to the edges and textures of anatomical structures in CT images, and the second decoding branch learns to extract feature representations related to the functional metabolic intensity distribution in PET images.

[0109] During the model training phase, paired CT and PET images of lesion regions are collected from a historical case database as training datasets. For each pair of training images, a corresponding joint feature set is generated following the procedures described in steps 1.1 to 5.3. and its multidimensional feature tensor As network input, the training labels consist of manually labeled CT-dominant and PET-dominant region masks. Within the lesion, radiologists, based on their clinical diagnostic experience, labeled regions primarily interpreted based on CT anatomical information and regions primarily interpreted based on PET metabolic information. During training, the Dice coefficient loss between the outputs of the two decoding branches and their corresponding labeled masks is used as a supervision signal; the total loss is the direct sum of the losses of the two branches. The optimizer employs an adaptive moment estimation optimizer with an initial learning rate set to a preset value. During training, a cosine annealing strategy is used to adjust the learning rate.

[0110] Step 6.2 involves performing layer-by-layer convolution and pooling operations on the multidimensional feature tensor using an encoder to progressively compress the spatial resolution and expand the number of feature channels, thereby extracting multi-scale abstract semantic features. Specifically, this includes: processing the multidimensional feature tensor... The first downsampling stage of the input encoder, after convolution operations and activation mapping of two convolutional layers in the first downsampling stage, yields the first-level feature map. . Having input tensor It maintains a consistent spatial resolution, but its feature channel count has been expanded to the preset number of channels in the first downsampling stage of the encoder. The activation map of each channel reflects the response strength of the convolutional kernel to a specific local combination pattern in the input bimodal joint information.

[0111] The first-level feature map In the second downsampling stage, the spatial resolution is first reduced to half of its original value by a max pooling layer, and then subjected to convolution operations and activation mapping by two convolutional layers in the second downsampling stage to obtain the second-level feature map. , The spatial resolution is Half of the number of feature channels, twice the number of channels in the first downsampling stage, and the receptive field of each feature channel compared to It has been doubled in size, enabling it to capture feature combinations and patterns over a larger spatial range.

[0112] The second-level feature map The input is processed through pooling and convolution in the third downsampling stage to obtain the third-level feature map. , The spatial resolution is one-quarter of the original input resolution, and the number of feature channels is twice the number of channels in the second downsampling stage.

[0113] The third-level feature map The fourth downsampling stage inputs the fourth-level feature map, which is then processed by pooling and convolution. That is, the bottleneck layer feature map. The spatial resolution is one-eighth of the original input resolution, and the number of feature channels is twice the number of channels in the third downsampling stage. This bottleneck layer feature map is the multi-scale abstract semantic feature output by the encoder, denoted as... , The input information of the target lesion area is compressed and encoded in the most compact spatial form. The low-level feature channel retains the boundary transition information in the edge change feature set, while the high-level feature channel abstracts the differences in the distribution of metabolic signal intensity integral values ​​and spatial topological relationships between local sub-regions.

[0114] Step 6.3 involves performing layer-by-layer upsampling and deconvolution operations on the multi-scale abstract semantic features using a decoder to progressively restore the spatial resolution to match the target lesion region. During the restoration process, the feature channels are separated into a first feature channel group reflecting the anatomical texture of CT images and a second feature channel group reflecting the functional metabolic intensity of PET images. Specifically, this includes: The first and second decoding branches are input separately. The two decoding branches have symmetrical structures but independent parameters, and each differentiates into different feature extraction preferences during training.

[0115] Taking the first decoding branch as an example: The first upsampling stage of the first decoding branch is used to restore the spatial resolution to twice its original value, i.e., from one-eighth to one-quarter of the original resolution, so that the feature map spatial size is similar to the output of the encoder's third downsampling stage. Consistency. The upsampled feature map is then combined with... The feature maps are concatenated along the channel dimension, allowing the decoder to simultaneously acquire high-level semantic information and mid-level detail information retained by the encoder at the corresponding stage. The concatenated feature maps are then fused through two convolutional layers to output the first-level upsampled feature map of the first decoding branch. During the convolutional fusion process, the convolutional kernel automatically learns to selectively weight and combine information from different sources in the stitching channels, gradually filtering out information components related to the edges and textures of CT anatomical structures.

[0116] Will The second upsampling stage of the input first decoding branch, after upsampling and... After feature concatenation and fusion of two convolutional layers, the second-level upsampled feature map of the first decoding branch is obtained. Its spatial resolution is restored to half of the original resolution.

[0117] Will The third upsampling stage of the first decoding branch inputs, after upsampling and... After feature concatenation and fusion of two convolutional layers, the output feature map of the first decoding branch is obtained. Its spatial resolution has been restored to be completely consistent with the input target lesion area. Size. This output feature map Include The first feature channel group consists of 10 feature channels. Each channel in the first feature channel group corresponds to the activation response of different anatomical texture patterns in the lesion area. The activation value of the channel reflects the strength of the anatomical structural features of the CT image at that spatial location.

[0118] Meanwhile, the second decoding branch performs a completely symmetrical processing procedure: After the three upsampling stages of the second decoding branch, each stage is respectively related to , , Feature stitching is performed to gradually restore spatial resolution, ultimately yielding the second feature channel group. . Include Each feature channel has an activation value that reflects the strength of the functional metabolic intensity characteristics of the PET image at that spatial location.

[0119] Step 6.4: Concatenate and merge the first feature channel group and the second feature channel group along the feature dimension to obtain the fused feature matrix, specifically including: the first feature channel group The size is ,in This indicates the number of channels for CT anatomical texture features. Indicates the number of pixels in the vertical direction. Indicates the number of pixels in the horizontal direction; second feature channel group The size is ,in The number of characteristic channels representing the metabolic intensity of PET. and and The corresponding dimensions are consistent.

[0120] Will and The feature matrix is ​​concatenated along the channel dimension, i.e., the first dimension, and the resulting feature matrix has a size of [size missing]. This concatenation operation merges the feature maps from the two decoding branches into a unified feature matrix while maintaining spatial coordinate alignment. It does not involve any numerical transformation or weighting operation, and the original activation values ​​in the two channel groups are completely preserved.

[0121] The concatenated feature matrix is ​​the fused feature matrix, denoted as . , In each spatial location The location corresponds to a length of The feature vector, the first half of which encodes the deep feature representation of the anatomical texture of the CT image at the pixel location, and the second half which encodes the deep feature representation of the functional metabolic intensity of the PET image at the pixel location, together constitute a multimodal joint deep representation of the pixel location. Fusion feature matrix This will be used as input to the pixel-level weight generation network in step 7.1, providing a deep semantic basis for determining the relative importance of CT modal information and PET modal information at each pixel location.

[0122] In this embodiment of the invention, a deep feature decomposition network with an encoder-decoder architecture is used to process the joint feature set. It automatically learns and separates the first feature channel group reflecting anatomical structure texture and the second feature channel group reflecting functional metabolic intensity in multimodal images. Its beneficial effect is that it does not simply superimpose CT and PET images at the original pixel level, but separates and then fused anatomical structure information and functional metabolic information in the deep semantic space. This preserves the unique and complementary high-level abstract features of each modality, so that the fused feature matrix has both rich structural details and accurate functional expression, avoiding mutual contamination or inhibition of information between modalities.

[0123] In a preferred embodiment of the present invention, step 7 above may include:

[0124] Step 7.1: Based on the fused feature matrix, input the pre-trained pixel-level weight generation network. The pixel-level weight generation network performs a non-linear mapping operation on the feature vector at each spatial location in the fused feature matrix, outputting the CT modality weight coefficient and PET modality weight coefficient corresponding to each pixel location. The sum of the two weight coefficients at the same pixel location is a normalization constant. Specifically, this includes: based on the obtained fused feature matrix... The input is a pre-trained pixel-level weight generation network. The pixel-level weight generation network performs a non-linear mapping operation on the feature vector at each spatial location in the fused feature matrix, and outputs the CT modality weight coefficient and PET modality weight coefficient corresponding to each pixel location.

[0125] The pixel-level weight generation network is structured as a fully convolutional subnetwork operating pixel-wise. This subnetwork contains two convolutional layers and one non-linear activation layer. The first convolutional layer receives the fused feature matrix. As input, its number of input channels is The number of output channels is preset to half the number of input channels, the kernel size is a preset value, and the stride is a preset value. The output of the first convolutional layer undergoes a non-linear mapping using a modified linear unit activation function to obtain an intermediate feature map. The second convolutional layer receives the intermediate feature map as input, with the number of input channels equal to the number of output channels of the first convolutional layer, and two output channels. The kernel size and stride are the same as those of the first convolutional layer. The output of the second convolutional layer is a feature map containing two channels, with a size of [missing value]. .

[0126] The 2-channel feature map output from the second convolutional layer is normalized using a sigmoid function, so that the two channel values ​​at each pixel location are mapped to a preset numerical range. The sigmoid function normalization is calculated as follows: ; ;

[0127] in, Represents pixel space coordinates, Indicates the second convolutional layer at pixels This corresponds to the original output value of the CT weight channel. Indicates the second convolutional layer at pixels This corresponds to the original output value of the PET weight channel. Represents the natural constant. Represents pixels CT modal weighting coefficients at the location, Represents pixels The PET modal weighting coefficient at the location.

[0128] After the above S-shaped function normalization operation, sum-normalization is performed on the two weight coefficients at each pixel location, ensuring that the sum of the two weight coefficients at the same pixel location is always equal to the normalization constant. The sum-normalization is calculated as follows: ; ;

[0129] in, and These are the CT modal weighting coefficients and PET modal weighting coefficients after S-mode function normalization, respectively. and The normalized CT modal weighting coefficients and PET modal weighting coefficients are respectively, satisfying... .

[0130] The pixel-level weight generation network was trained in conjunction with a deep feature decomposition network. Training labels were created by radiologists using heatmaps: for pixel locations within the lesion region where CT anatomical information should be dominant, such as high-density shadows with sharp boundaries, the CT modality weight label was set to a higher value, and the PET modality weight label to a lower value; for pixel locations where PET metabolic information should be dominant, such as metabolically active but infiltrative areas with blurred CT boundaries, the PET modality weight label was set to a higher value, and the CT modality weight label to a lower value. The training loss was the mean squared error between the weight coefficients output by the network at each pixel location and the manually labeled weights.

[0131] Step 7.2: Based on the CT modality weighting coefficient and PET modality weighting coefficient corresponding to each pixel location, extract the gray values ​​of the CT lesion region image and the PET lesion region image at the corresponding pixel location from the standardized image dataset. Multiply the CT gray value by the CT modality weighting coefficient to obtain the CT weighted component, and multiply the PET gray value by the PET modality weighting coefficient to obtain the PET weighted component. Add the CT weighted component and the PET weighted component to obtain the weighted fused gray value of that pixel location. After traversing all pixel locations, the weighted feature matrix is ​​obtained. Specifically, this includes: for any pixel location within the target lesion region... Extract the standardized gray value at this location from the standardized CT lesion area image. Extract the standardized gray value at this location from the standardized PET lesion area image. The weighted fused grayscale value at this pixel location is calculated as follows: ;

[0132] in, and The results are obtained from the normalization process in step 7.1, respectively, and satisfy the following conditions: ; Represents pixels in a standardized CT lesion region image Standardized grayscale value (dimensionless standard score) at the location; Represents pixels in standardized PET lesion area images Standardized grayscale value (dimensionless standard score) at the location; Represents pixels The weighted fused grayscale value (dimensionless standard score) at the location.

[0133] The weighted blended grayscale value The physical meaning is: in pixels At this location, CT modal information contributes to the fused grayscale value of that pixel. Proportional components, PET modal information contribute Proportional component. When When this occurs, it indicates that the CT anatomical structure information dominates at that pixel location, and the fused image tends to retain the grayscale levels and edge sharpness of the CT image at that location; when This indicates that PET metabolic function information dominates at that pixel location, and the fused image tends to highlight the metabolic signal distribution characteristics of the PET image at that location. Since the weighting coefficients are generated independently at each pixel location, the weighting distribution pattern is different in different regions within the lesion. In areas with sharp boundaries, the CT weighting coefficient is higher, and the anatomical structure boundaries of the fused image are clear; in metabolically active but infiltrated areas with blurred boundaries, the PET weighting coefficient is higher, and the metabolic activity information of the fused image is prominent.

[0134] Traverse the entire target lesion area Pixel position Perform the weighted fusion grayscale value calculation one by one, and combine all the calculated values. According to spatial coordinates Arranged in order of size A two-dimensional matrix, which is the weighted characteristic matrix, denoted as . Weighted eigenmatrix Each element in the algorithm is a weighted fused grayscale value at the corresponding pixel location. This value is synthesized by weighting the CT and PET dual-modal weighting coefficients at that pixel location, thus achieving pixel-level adaptive fusion.

[0135] In this embodiment of the invention, a pixel-level weighting generation network adaptively generates CT modality weighting coefficients and PET modality weighting coefficients for each pixel location, and performs grayscale value weighted fusion accordingly, achieving spatial fine-tuning of the fusion strategy. Especially in the transition zone at the edge of the lesion, the network can automatically adjust the contribution ratio of the two modalities according to the edge change rate and metabolic distribution characteristics at that location. Higher weights are assigned to CT in areas with clear anatomical structure boundaries to maintain edge sharpness, and higher weights are assigned to PET in metabolically active areas to highlight functional information, thereby precisely balancing structural display and functional expression at the pixel level.

[0136] In a preferred embodiment of the present invention, step 8 above may include:

[0137] Step 8.1: Based on the spatial coordinate indices of each pixel in the weighted feature matrix, map the weighted feature matrix pixel-by-pixel back to the corresponding spatial position of the target lesion region in the global spatial coordinate system of the original CT and PET images. This specifically includes: the weighted feature matrix... The size is row index of each pixel and column indexes The local spatial coordinate indices that make up this matrix, where The range of values ​​is to , The range of values ​​is to The origin of this local coordinate system is located at the top-left vertex of the smallest bounding rectangle of the target lesion region. Since the global spatial coordinate range of the target lesion region in its original CT and PET images was recorded during lesion region extraction, including the x-coordinate of the top-left vertex of the smallest bounding rectangle of the lesion region in the global coordinate system... and ordinate and the horizontal span of the lesion area and vertical span Therefore, the weighted characteristic matrix Local coordinates in with global spatial coordinates The mapping relationship between them is as follows: ; ;

[0138] in, and Determined from the results of lesion area extraction, Weighted eigenmatrix Row indexes and column indexes in the data. These are the x and y coordinates of the pixel in the global spatial coordinate system of the original CT and PET images.

[0139] Step 8.2: Based on the mapped spatial location, replace the original pixel values ​​at the same spatial location in the original image with the weighted fused grayscale values ​​of the corresponding pixels in the weighted feature matrix. Then, perform neighborhood pixel weighted average smoothing on the boundary transition zone of the target lesion area to eliminate pixel stitching traces between the lesion area and surrounding normal tissue, obtaining image data after pixel replacement and smoothing. Specifically, this includes: establishing a blank fused image canvas with the same size as the original global image. In the original CT image, for each pixel location outside the spatial range of the target lesion area... This process preserves the grayscale value of the location in the original CT image to the same location in the fused image, ensuring that the normal tissue surrounding the lesion area maintains the anatomical structure appearance of the original CT image in the fused image. This applies to each pixel location within the spatial range of the target lesion area. The mapping relationship in step 8.1 is used to find its representation in the weighted feature matrix. Local coordinates corresponding to the middle Extract the weighted fused grayscale value at this location. Assign this value to the merged image canvas. The pixels at that location.

[0140] After the above pixel replacement operation is completed, the pixel values ​​of the fused image inside the lesion area come from the weighted feature matrix, and the pixel values ​​outside the lesion area come from the original CT image. The two may produce a discontinuity in gray values ​​at the lesion boundary. In order to eliminate the discontinuity of pixel stitching traces, neighborhood pixel weighted average smoothing is performed on the transition zone of the lesion boundary.

[0141] The boundary transition zone is determined by using the obtained set of pixels at the edge of the lesion. Taking each edge pixel as the center, extend outwards and inwards by a predetermined number of pixels along the boundary normal of that pixel, denoted as . A strip region with a width of one pixel, and all pixels in this strip region constitute the boundary transition zone. The value of is determined by the size of the lesion and the imaging resolution.

[0142] For each pixel within the boundary transition zone ,by Take a neighborhood window of a preset size centered on the pixel, calculate the weighted average of the grayscale values ​​of all pixels within the window, and assign this weighted average to the specified value. The result is the smoothed grayscale value. The calculation method for neighborhood-weighted average smoothing is as follows: ;

[0143] in, Indicates The set of all pixels covered by the neighborhood window centered on the center. This represents any pixel within the neighborhood window. Represents pixels and The Euclidean distance between them The weighting function is Gaussian, and its value varies with distance. The weight of the neighboring pixels increases monotonically with the increase of the center pixel, meaning that the closer the neighboring pixels are to the center pixel, the greater their contribution to the smoothing result, and the farther the neighboring pixels are, the smaller their contribution. Represents pixels The original grayscale value before smoothing. Represents pixels after smoothing. The grayscale value at that location.

[0144] The weighted average smoothing process described above only applies to pixels within the boundary transition zone, without touching the core area inside the lesion far from the boundary or the normal tissue area outside the lesion. While eliminating boundary splicing traces, it preserves the integrity of the fusion features inside the lesion and the external anatomical structure information, resulting in image data after pixel replacement and smoothing.

[0145] Step 8.3 integrates and outputs the image data after pixel replacement and smoothing into a complete multimodal medical fusion image. This multimodal medical fusion image simultaneously carries the anatomical information from CT images and the functional metabolic information from PET images within the same image frame. Specifically, the image data after pixel replacement and smoothing is a complete single-frame two-dimensional grayscale image, with a coverage area consistent with the global spatial range of the original CT image. In this fusion image, the pixel grayscale values ​​within the lesion region are obtained through spatial mapping using a weighted feature matrix, simultaneously carrying CT anatomical edge information and PET functional metabolic intensity information. At pixel locations with higher CT weighting coefficients, the grayscale levels and edge transitions primarily present the original morphological features of the CT image, facilitating physician observation of the lesion's boundary location, shape, and proximity to surrounding tissues. At pixel locations with higher PET weighting coefficients, the grayscale levels primarily present the metabolic signal distribution of the PET image, facilitating physician assessment of the functionally active areas and metabolic heterogeneity distribution of the lesion. The fused image is encoded and stored according to the medical digital imaging and communication standard format or portable network graphics image format. Its metadata retains the spatial resolution, pixel physical size and body position information of the original CT image, and is output as a multimodal medical fused image.

[0146] This multimodal medical fusion imaging simultaneously carries the anatomical information of CT images and the functional metabolic information of PET images on the same image: the morphological outline of the lesion is provided by the spatial positioning benchmark of CT anatomical information, and the metabolic activity level of each region inside the lesion is provided by PET functional metabolic information for colorimetric basis, so that radiologists can comprehensively interpret the morphological and functional characteristics of the lesion without switching and comparing two independent images.

[0147] In this embodiment of the invention, the weighted feature matrix is ​​mapped back to the original global spatial coordinate system based on spatial coordinate indices for pixel value replacement. Furthermore, a neighborhood pixel-weighted average smoothing process is specifically performed on the transition zone of the lesion boundary. The beneficial effect is that it ensures the fusion result is completely consistent with the original image in spatial location, while simultaneously making the fused lesion area visually seamlessly connected with the surrounding normal tissue, eliminating abrupt stitching marks caused by pixel replacement. The final output multimodal fused image retains the high-resolution details of CT anatomical structures while incorporating sensitive information from PET functional metabolism. It achieves the dual goals of clearly discernible structural boundaries and realistically readable metabolic distribution on the same image, enhancing the clinical ability to comprehensively identify the extent of lesion infiltration and metabolic activity.

[0148] like Figure 2 As shown, embodiments of the present invention also provide a deep learning-based multimodal medical image fusion system, comprising:

[0149] The image acquisition and preprocessing module is used to acquire CT and PET images to be processed, extract the target lesion area, and obtain an initial image dataset; grayscale value normalization processing is performed on the initial image dataset to obtain a standardized image dataset;

[0150] The edge change feature extraction module is used to measure the unit pixel gray level change of the gray level difference between adjacent pixels at the edge of the target lesion area in a standardized CT lesion area image point by point for a standardized image dataset, and obtain the edge change feature set.

[0151] The local sub-region segmentation module is used to segment the target lesion area into multiple local sub-regions based on the edge change feature set;

[0152] The regional metabolic signal intensity integral value feature calculation module is used to accumulate and calculate the global total regional metabolic signal intensity integral value distribution of the corresponding region in the PET image for each local sub-region, and obtain the regional metabolic signal intensity integral value feature set.

[0153] The joint feature alignment module is used to extract the spatial coordinate information of the edge change feature set and the regional metabolic signal intensity integral value feature set, and perform spatial alignment operation to obtain the joint feature set;

[0154] The deep feature decomposition and fusion module is used to decompose and peel off the deep structural features of multimodal images based on the joint feature set to obtain a fused feature matrix;

[0155] The pixel-level weight allocation module is used to perform pixel-level weight allocation operations based on the fused feature matrix to obtain the weighted feature matrix;

[0156] The fusion image reconstruction module is used to reconstruct and generate multimodal medical fusion images based on the weighted feature matrix and the original spatial coordinate system.

[0157] It should be noted that this system is a system corresponding to the above method. All implementation methods in the above method embodiments are applicable to this embodiment and can achieve the same technical effect.

[0158] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based multimodal medical image fusion method, characterized in that, The method includes: Step 1: Acquire the CT and PET images to be processed, and extract the target lesion area to obtain the initial image dataset; perform grayscale value normalization processing on the initial image dataset to obtain the standardized image dataset. Step 2: For the standardized image dataset, calculate the unit pixel gray level change of the gray level difference between adjacent pixels at the edge of the target lesion area in the standardized CT lesion area image point by point to obtain the edge change feature set. Step 3: Based on the edge change feature set, divide the target lesion area into multiple local sub-regions; Step 4: For each local sub-region, accumulate and calculate the global total metabolic signal intensity integral value distribution of the corresponding region in the PET image to obtain the regional metabolic signal intensity integral value feature set; Step 5: Extract the spatial coordinate information from the edge change feature set and the regional metabolic signal intensity integral value feature set, and perform spatial alignment operation to obtain the joint feature set; Step 6: Based on the joint feature set, decompose and strip the deep structural features of the multimodal image to obtain the fused feature matrix; Step 7: Based on the fused feature matrix, perform pixel-level weight allocation to obtain the weighted feature matrix; Step 8: Based on the weighted feature matrix and combined with the original spatial coordinate system, reconstruct and generate multimodal medical fusion images.

2. The deep learning-based multimodal medical image fusion method according to claim 1, characterized in that, Step 1: Acquire the CT and PET images to be processed, and extract the target lesion region to obtain the initial image dataset; perform grayscale normalization processing on the initial image dataset to obtain the standardized image dataset, including: Spatial registration was performed on CT and PET images of the same subject to obtain spatially aligned bimodal image pairs. Based on the spatially aligned bimodal image pairs, a pre-trained lesion segmentation deep learning model is invoked to identify and extract lesion regions from CT and PET images respectively, resulting in CT lesion region images and PET lesion region images, which together constitute the initial image dataset. For the CT lesion area images and PET lesion area images in the initial image dataset, calculate the mean and standard deviation of their respective pixel gray values. Based on the mean and standard deviation, map the pixel gray values ​​of each image to a preset numerical range to obtain the standardized image dataset.

3. The deep learning-based multimodal medical image fusion method according to claim 2, characterized in that, Step 2: For the standardized image dataset, calculate the unit pixel gray-level change of adjacent pixels at the edge of the target lesion region in the standardized CT lesion region image point by point, and obtain the edge change feature set, including: Based on the CT lesion region images in the standardized image dataset, the edge detection operator is applied to extract the boundary contour of the target lesion region, resulting in a set of lesion edge pixels; Based on the set of pixels at the edge of the lesion, for each edge pixel, the gray level difference between adjacent pixels along the boundary normal direction is calculated to obtain the gray level jump variable corresponding to each edge pixel. Based on the grayscale jump variable corresponding to each edge pixel and the spatial distance between adjacent pixels, calculate the instantaneous rate of change of the grayscale difference at each edge pixel with the change of spatial position. Record the spatial coordinates of each edge pixel with the instantaneous rate of change value of that edge pixel to obtain the edge change feature set.

4. The deep learning-based multimodal medical image fusion method according to claim 3, characterized in that, Step 3: Based on the edge change feature set, divide the target lesion area into multiple local sub-regions, including: Based on the instantaneous rate of change value carried by each edge pixel in the edge change feature set, at least two rate of change threshold intervals are preset. Edge pixels whose instantaneous rate of change values ​​fall into the same threshold interval are grouped into the same edge group to obtain multiple edge groups. Based on the spatial contour range enclosed by the edge pixels contained in each edge group, the closed region defined by each edge group is determined, and the target lesion region is divided into multiple local sub-regions defined by each edge group.

5. The deep learning-based multimodal medical image fusion method according to claim 4, characterized in that, Step 4: For each local sub-region, accumulate and calculate the global total metabolic signal intensity integral value distribution of the corresponding region in the PET image to obtain the regional metabolic signal intensity integral value feature set, including: Based on the spatial range defined by each of the multiple local sub-regions, the metabolic signal intensity values ​​of all pixels within the corresponding spatial location in the PET lesion area image of the standardized image dataset are extracted for each region to obtain the metabolic signal intensity distribution data of each local sub-region. Based on the metabolic signal intensity distribution data of each local sub-region, the metabolic signal intensity values ​​of all pixels in each local sub-region are accumulated and calculated region by region to obtain the total metabolic signal intensity integral value of each local sub-region. The total regional metabolic signal intensity integral value of all local sub-regions is associated and combined with the spatial identifier of each local sub-region to obtain the feature set of regional metabolic signal intensity integral values.

6. The deep learning-based multimodal medical image fusion method according to claim 5, characterized in that, Step 5: Extract spatial coordinate information from the edge change feature set and the regional metabolic signal intensity integral value feature set, perform spatial alignment operation, and obtain a joint feature set, including: Based on the edge change feature set, extract the spatial coordinate information of each edge pixel in the edge change feature set, and construct an edge feature spatial coordinate mapping table indexed by spatial coordinates; Based on the feature set of regional metabolic signal intensity integral values, the spatial identifier and corresponding spatial range information of each local sub-region in the feature set of regional metabolic signal intensity integral values ​​are extracted, and a spatial coordinate mapping table of regional metabolic signal intensity integral value features indexed by the spatial identifier is constructed. Based on the spatial coordinate mapping table of edge features and the spatial coordinate mapping table of regional metabolic signal intensity integral value features, and taking spatial coordinate consistency as the alignment criterion, the edge change feature data and regional metabolic signal intensity integral value feature data at the same spatial location are paired and merged one by one to obtain the joint feature set. Each record in the joint feature set includes spatial coordinates, the instantaneous rate of change value corresponding to the coordinates, and the total regional metabolic signal intensity integral value of the local sub-region to which the coordinates belong.

7. The deep learning-based multimodal medical image fusion method according to claim 6, characterized in that, Step 6: Based on the joint feature set, decompose and strip the deep structural features of the multimodal image to obtain the fused feature matrix, including: Based on the joint feature set, the joint feature set is transformed into a multidimensional feature tensor, which serves as the input data for a pre-trained deep feature decomposition network, where the deep feature decomposition network adopts an encoder-decoder architecture. By performing layer-by-layer convolution and pooling operations on the multidimensional feature tensor through the encoder, the spatial resolution is compressed and the number of feature channels is expanded step by step to extract multi-scale abstract semantic features. The decoder performs layer-by-layer upsampling and deconvolution operations on the multi-scale abstract semantic features to restore the spatial resolution consistent with the target lesion area step by step. During the restoration process, the feature channels are separated into a first feature channel group that reflects the anatomical structure texture of CT images and a second feature channel group that reflects the functional metabolic intensity of PET images. The first feature channel group and the second feature channel group are spliced ​​and merged along the feature dimension to obtain the fused feature matrix.

8. The deep learning-based multimodal medical image fusion method according to claim 7, characterized in that, Step 7: Based on the fused feature matrix, perform pixel-level weight allocation to obtain a weighted feature matrix, including: Based on the fused feature matrix, the pre-trained pixel-level weight generation network is input. The pixel-level weight generation network performs a non-linear mapping operation on the feature vector at each spatial location in the fused feature matrix, and outputs the CT modality weight coefficient and PET modality weight coefficient corresponding to each pixel location. The sum of the two weight coefficients at the same pixel location is a normalization constant. Based on the CT modal weighting coefficient and PET modal weighting coefficient corresponding to each pixel position, the gray values ​​of the CT lesion region image and the PET lesion region image at the corresponding pixel position are extracted from the standardized image dataset. The CT gray value is multiplied by the CT modal weighting coefficient to obtain the CT weighted component, and the PET gray value is multiplied by the PET modal weighting coefficient to obtain the PET weighted component. The CT weighted component and the PET weighted component are added to obtain the weighted fused gray value of the pixel position. After traversing all pixel positions, the weighted feature matrix is ​​obtained.

9. The deep learning-based multimodal medical image fusion method according to claim 8, characterized in that, Step 8: Based on the weighted feature matrix and combined with the original spatial coordinate system, reconstruct and generate a multimodal medical fusion image, including: Based on the spatial coordinate index of each pixel in the weighted feature matrix, the weighted feature matrix is ​​mapped pixel by pixel back to the corresponding spatial position of the target lesion area in the global spatial coordinate system of the original CT image and PET image. Based on the mapped spatial location, the original pixel value of the original image at the same spatial location is replaced by the weighted fused gray value of the corresponding pixel in the weighted feature matrix. Neighborhood pixel weighted average smoothing is then performed on the boundary transition zone of the target lesion area to eliminate pixel stitching traces between the lesion area and the surrounding normal tissue, thus obtaining the image data after pixel replacement and smoothing. The image data after pixel replacement and smoothing are integrated and output as a complete multimodal medical fusion image. The multimodal medical fusion image simultaneously carries the anatomical structure information of CT images and the functional metabolic information of PET images on the same image screen.

10. A deep learning-based multimodal medical image fusion system, wherein the system implements the method as described in any one of claims 1 to 9, characterized in that, include: The image acquisition and preprocessing module is used to acquire CT and PET images to be processed, extract the target lesion area, and obtain an initial image dataset; grayscale value normalization processing is performed on the initial image dataset to obtain a standardized image dataset; The edge change feature extraction module is used to measure the unit pixel gray level change of the gray level difference between adjacent pixels at the edge of the target lesion area in a standardized CT lesion area image point by point for a standardized image dataset, and obtain the edge change feature set. The local sub-region segmentation module is used to segment the target lesion area into multiple local sub-regions based on the edge change feature set; The regional metabolic signal intensity integral value feature calculation module is used to accumulate and calculate the global total regional metabolic signal intensity integral value distribution of the corresponding region in the PET image for each local sub-region, and obtain the regional metabolic signal intensity integral value feature set. The joint feature alignment module is used to extract the spatial coordinate information of the edge change feature set and the regional metabolic signal intensity integral value feature set, and perform spatial alignment operation to obtain the joint feature set; The deep feature decomposition and fusion module is used to decompose and peel off the deep structural features of multimodal images based on the joint feature set to obtain a fused feature matrix; The pixel-level weight allocation module is used to perform pixel-level weight allocation operations based on the fused feature matrix to obtain the weighted feature matrix; The fusion image reconstruction module is used to reconstruct and generate multimodal medical fusion images based on the weighted feature matrix and the original spatial coordinate system.

Images