Pseudo ct image target region prediction method and system with cross-level cross-space feature aggregation

Abstract

The application provides a pseudo-CT image target region prediction method and system based on cross-level and cross-space feature aggregation, comprising: generating pseudo-CT image data based on optical body surface images; performing feature extraction on the pseudo-CT image data to obtain multi-level original features; processing the pseudo-CT image data by using a three-dimensional convolutional neural network-long short-term memory network to obtain space-time comprehensive features, and fusing the space-time comprehensive features into the multi-level original features; determining target region prediction features of the pseudo-CT image data based on the fused multi-level original features by using a cross-level and cross-space feature aggregation pyramid network; generating a single-phase static target region mask based on the target region prediction features; and determining a target region prediction position in a future prediction phase based on the static target region mask. The application can improve the target region prediction accuracy based on optical body surface images.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method and system for predicting target regions in pseudo-CT images by cross-level and cross-spatial feature aggregation. Background Technology

[0002] In radiotherapy for thoracic and abdominal tumors, real-time monitoring of the tumor's location and morphological changes is crucial to ensure that high-dose radiation accurately covers the tumor target area and protects surrounding organs at risk.

[0003] To reduce the additional radiation dose from frequent X-ray imaging during treatment, surface-guided radiation therapy (SGRT) can be used. This technique uses optical imaging equipment to collect motion signals from the patient's body surface in real time and attempts to infer the real-time location of the tumor by comparing the surface motion with the internal tumor motion. This serves as the primary means of target area monitoring during radiotherapy.

[0004] However, the accuracy of target prediction schemes in related technologies is often low, which cannot provide effective technical support for target delineation and conformal radiotherapy in radiotherapy. Summary of the Invention

[0005] The embodiments of this application provide a pseudo-CT image target region prediction method and system with cross-level and cross-spatial feature aggregation, which aims to improve the target region prediction accuracy based on optical body surface images.

[0006] In a first aspect, embodiments of this application provide a method for predicting target regions in pseudo-CT images by aggregating cross-level and cross-spatial features. This method includes:

[0007] Based on optical surface images, pseudo-CT image data is generated.

[0008] Feature extraction is performed on the pseudo-CT image data to obtain multi-level original features;

[0009] The pseudo-CT image data is processed using a three-dimensional convolutional neural network-long short-term memory network to obtain spatial-temporal integrated features, and the spatial-temporal integrated features are then fused into the multi-level original features.

[0010] By utilizing a cross-level and cross-spatial feature aggregation pyramid network, and based on the fused multi-level original features, the target area prediction features of the pseudo-CT image data are determined.

[0011] Based on the target region prediction features, a static target region mask for a single time phase is generated;

[0012] Based on the static target mask, the target prediction location at the future prediction time phase is determined.

[0013] In the above embodiments, by integrating optical surface reconstruction and spatiotemporal feature extraction technologies, using 3D CNN-LSTM to capture dynamic evolution patterns, and combining feature aggregation pyramids to enhance the representation of small targets, not only is radiation-free pseudo-CT high-precision segmentation achieved, but also future dynamic target area prediction is achieved based on deformation field inference, effectively improving the real-time performance and accuracy of tumor tracking under respiratory motion.

[0014] In one embodiment, the step of utilizing a cross-level, cross-spatial feature aggregation pyramid network to determine the target region prediction features of the pseudo-CT image data based on the fused multi-level original features includes:

[0015] In the fused multi-level original features, the first-level features, second-level features, and third-level features with successively decreasing levels are identified;

[0016] Using the first cross-level and cross-space feature aggregation module, a first intermediate feature is determined based on the first-level feature and the second-level feature;

[0017] Using the second cross-level and cross-space feature aggregation module, the second intermediate feature is determined based on the first intermediate feature and the third-level feature;

[0018] Using the third cross-level and cross-space feature aggregation module, the third intermediate feature is determined based on the first intermediate feature, the second intermediate feature, and the first level feature;

[0019] Based on the third intermediate feature, the first-level output feature corresponding to the first-level feature is determined; based on the first intermediate feature and the second intermediate feature, the second-level output feature corresponding to the second-level feature is determined; and based on the second intermediate feature, the third-level output feature corresponding to the third-level feature is determined.

[0020] The target area prediction features of the pseudo-CT image data are determined based on the first-level output features, the second-level output features, and the third-level output features.

[0021] In the above embodiments, a specific pyramid network architecture is constructed, utilizing three cascaded cross-level and cross-spatial feature aggregation modules to establish a high-speed information pathway from deep to shallow, and backtracking connections are designed to enhance semantic expression. This scheme effectively integrates the high abstraction of deep semantics with the high resolution of shallow details, solving the challenges of complex structures and varying scales in tumor segmentation, and improving the robustness and accuracy of the model in predicting target regions of different sizes.

[0022] In one embodiment, the step of using the first cross-level cross-spatial feature aggregation module to determine the first intermediate feature based on the first-level feature and the second-level feature includes:

[0023] Based on the first-level features, determine the global semantic association weights;

[0024] The first-level features are weighted and adjusted using the global semantic association weights to obtain the semantically enhanced first-level features.

[0025] Based on the second-level features, determine the multi-scale spatial dependency weights;

[0026] The second-level features are weighted and adjusted using the multi-scale spatial dependency weights to obtain the second-level features after selective spatial enhancement.

[0027] The first intermediate feature is determined based on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement.

[0028] In the above embodiments, by designing a unique dual attention enhancement path, feature recalibration is introduced in the channel dimension to enhance semantics, and multi-receptive field aggregation is introduced in the spatial dimension to adapt to multi-scale targets. This effectively solves the feature blurring problem caused by low soft tissue contrast in pseudo-CT images, enabling the generated intermediate features to take into account both accurate global semantic classification and fine local spatial positioning.

[0029] In one embodiment, determining the global semantic association weight based on the first-level features includes:

[0030] The first-level features are subjected to global standard deviation pooling and global average pooling respectively to obtain two types of global feature vectors;

[0031] The two types of global feature vectors are stacked along the channel dimension to generate a global information summarization vector.

[0032] The global semantic association weights are generated based on the global information induction vector.

[0033] In the above embodiments, by introducing global standard deviation pooling and global average pooling in parallel, a global information induction vector containing second-order statistical information (discretion) is constructed, so that the generated global semantic association weights can not only identify significant targets, but also keenly capture small lesion features with high variability, thereby improving the comprehensiveness and accuracy of feature selection.

[0034] In one embodiment, determining the multi-scale spatial dependency weights based on the second-level features includes:

[0035] The second-level features are input in parallel into three deep-dilated convolutional modules to obtain three spatial feature responses with different receptive fields. Among them, the kernel size or dilation rate of any two deep-dilated convolutional modules is different.

[0036] By using a grouping channel aggregation strategy, the spatial feature responses of the three different receptive fields are grouped and recombined along the channel dimension to generate multi-scale spatial information aggregation features.

[0037] Based on the multi-scale spatial information aggregation features, the multi-scale spatial dependency weights are generated.

[0038] In the above embodiments, by deploying deep dilated convolutional modules with different kernel sizes and different dilation rates in parallel, and combining them with a grouping channel aggregation strategy, the effective receptive field of the network is not only greatly expanded to meet the detection needs of tumors of different volumes, but also the computational complexity is effectively reduced. This allows the generated multi-scale spatially dependent weights to accurately remove irrelevant background interference and focus on the real target area that is significant at multiple scales.

[0039] In one embodiment, determining the first intermediate feature based on the semantically explicitly enhanced first-level features and the selectively spatially enhanced second-level features includes:

[0040] The first-level semantic matching process is performed on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement, respectively, to obtain the first semantic matching features and the second semantic matching features.

[0041] Based on the first semantic matching feature and the second semantic matching feature, determine the semantic alignment feature;

[0042] The first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement are subjected to second-level spatial matching processing to obtain the first spatial matching features and the second spatial matching features.

[0043] Based on the first spatial matching feature and the second spatial matching feature, spatial alignment features are determined;

[0044] Based on the semantic alignment features and the spatial alignment features, they are concatenated along the channel dimension to generate cross-level and cross-space aggregated features, which are then used as the first intermediate features.

[0045] In the above embodiments, by designing a bidirectional cross-alignment mechanism, two independent matching and fusion processes are performed in the low-resolution space belonging to high-level semantics and the high-resolution space belonging to low-level details, and then the two are unified. This solves the problem of misalignment and loss of semantic and spatial information in pseudo-CT images during transmission, and maximizes the preservation of the category and geometric features of the target area.

[0046] In one embodiment, the first-level semantic matching processing is performed on the semantically explicitly enhanced first-level features and the selectively spatially enhanced second-level features to obtain first semantic matching features and second semantic matching features, including:

[0047] The first semantic matching feature is obtained by using the first convolutional layer to perform semantic feature transformation on the first-level features after explicit semantic enhancement, while keeping the semantic level of the first-level features after explicit semantic enhancement unchanged.

[0048] The second convolutional layer is used to perform semantic feature transformation on the second-level features after selective space enhancement, and semantic level enhancement is performed on the second-level features after selective space enhancement to obtain the second semantic matching feature;

[0049] The second-level spatial matching processing is performed on the semantically explicitly enhanced first-level features and the selectively spatially enhanced second-level features to obtain first spatial matching features and second spatial matching features, including:

[0050] The first spatial matching feature is obtained by using the third convolutional layer to perform spatial feature transformation on the semantically explicitly enhanced first-level features and improving the spatial resolution of the semantically explicitly enhanced first-level features.

[0051] The second spatial matching feature is obtained by using the fourth convolutional layer to perform spatial feature transformation on the second-level features after selective spatial enhancement, while keeping the spatial resolution of the second-level features after selective spatial enhancement unchanged.

[0052] In the above embodiments, before feature fusion, dual interoperability of heterogeneous features at the semantic level (low resolution) and spatial level (high resolution) is achieved, ensuring that each addition fusion is performed on a strictly aligned grid, eliminating feature ambiguity caused by scale differences, and significantly improving the accuracy of target area prediction in boundary localization.

[0053] In one embodiment, the step of using a third cross-level, cross-spatial feature aggregation module to determine a third intermediate feature based on the first intermediate feature, the second intermediate feature, and the first level feature includes:

[0054] Based on the second intermediate feature, spatial downsampling is performed to obtain the fourth intermediate feature;

[0055] Based on the fourth intermediate feature and the first intermediate feature, the fifth intermediate feature is obtained by stacking them in the channel dimension;

[0056] Based on the fifth intermediate feature, spatial downsampling is performed to obtain the sixth intermediate feature;

[0057] The third intermediate feature is determined using the third cross-level and cross-space feature aggregation module, based on the sixth intermediate feature and the first level feature.

[0058] In the above embodiments, by constructing a cascaded downsampling and stacking path, a feature transmission chain from shallow to deep is formed to gradually fuse the intermediate results generated by the preceding modules with the original encoded features and transmit them to the deepest layer. This ensures that the third cross-level and cross-space feature aggregation module can work in a highly rich context environment. As a result, the generated third intermediate features have both the robustness of deep semantics and the coherence of shallow details, effectively improving the overall accuracy of pseudo-CT image prediction.

[0059] In one embodiment, determining the target region prediction location at the future prediction phase based on the static target region mask includes:

[0060] Using the regression prediction layer in the three-dimensional convolutional neural network-long short-term memory network, a deformation registration field pointing to the future prediction phase is generated;

[0061] Based on the deformation registration field pointing to the future prediction phase, the static target mask is deformed to obtain the target prediction position under the future prediction phase.

[0062] In the above embodiments, by introducing an indirect prediction strategy based on deformation fields, the complex problem of tumor shape prediction is transformed into a relatively simple voxel motion estimation problem, so as to generate a high-precision deformation registration field to drive the static mask to deform, thereby achieving accurate tracking of the position of the moving target area in the future while ensuring the topological integrity of the anatomical structure.

[0063] Secondly, embodiments of this application provide a pseudo-CT image target region prediction system with cross-level and cross-spatial feature aggregation, the pseudo-CT image target region prediction system with cross-level and cross-spatial feature aggregation being used to perform the pseudo-CT image target region prediction method with cross-level and cross-spatial feature aggregation as described in any of the preceding claims. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a schematic flowchart of an embodiment of the pseudo-CT image target region prediction method with cross-level and cross-spatial feature aggregation provided in this application.

[0066] Figure 2 This is a schematic diagram of the overall network structure of the pseudo-CT image target region prediction method with cross-level and cross-spatial feature aggregation provided in the embodiments of this application;

[0067] Figure 3 This is a schematic diagram of a cross-level, cross-space feature aggregation module provided in an embodiment of this application;

[0068] Figure 4 This is a schematic diagram of a cross-level, cross-spatial feature aggregation pyramid network provided in an embodiment of this application.

[0069] in, Figure 3 and Figure 4 This is the image after rotating 90 degrees counterclockwise. Detailed Implementation

[0070] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. In addition, in the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0071] In a first aspect, embodiments of this application provide a pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation, wherein the execution subject is a pseudo-CT image target region prediction system based on cross-level and cross-spatial feature aggregation.

[0072] Specifically, a pseudo-CT image target region prediction method that aggregates features across levels and spaces may include: S101. Generate pseudo-CT image data based on optical surface images.

[0073] In the embodiments of this application, optical surface images refer to image data reflecting the geometric shape and motion state of a patient's body surface, acquired by an optical three-dimensional imaging system (such as a structured light camera, a stereo vision system, or a time-of-flight camera). This data typically does not contain ionizing radiation and only records depth information, texture information, and dynamic displacement signals of the human body surface.

[0074] Synthetic CT (Pseudo-CT) image data refers to three-dimensional volumetric data synthesized through computational methods based on non-CT imaging modalities (in this case, optical surface images) that possess characteristics similar to CT (Computer Tomography) images. This data aims to simulate the Hounsfield Unit (HU) value of a real CT scan to reflect the density distribution of internal anatomical structures in the human body, thereby enabling the calculation of radiotherapy doses or target delineation without performing actual X-ray scans on the patient, thus reducing radiation dose.

[0075] In some embodiments of this application, the specific implementation path for generating pseudo-CT image data includes using a traditional image generation model, specifically an edge-structure-guided CycleGAN (Cycle-Consistent Generative Adversarial Networks). For example, as... Figure 1 In the flowchart shown, "body surface" ph1 "To "pseudo-CT" ph1 The process can be guided by CycleGAN based on the edge structure of wavelet transform frequency domain. This model uses optical surface images as conditional input and generates high-fidelity pseudo-CT image data by learning the nonlinear mapping relationship between surface morphology and internal anatomical structures.

[0076] In some embodiments of this application, a CycleGAN guided by edge structure based on wavelet transform frequency domain is described. Specifically, wavelet transform frequency domain refers to projecting the input optical surface image from the spatial domain to the frequency domain using Discrete Wavelet Transform (DWT). Wavelet transform decomposes the image into low-frequency and high-frequency components, where the low-frequency components characterize the overall structure and contour information of the image, while the high-frequency components contain the texture and edge details of the image. Compared to generating directly in pixel space, the feature distribution in the frequency domain is sparser and more independent, which can reduce the computational complexity of model generation and improve the sharpness of the generated image.

[0077] The core architecture of the edge-structure-guided CycleGAN includes dual generators (G1, G2) and dual discriminators (D1, D2), and introduces an edge detection module to construct edge structure constraints. The dual generators aim to establish a bidirectional nonlinear mapping relationship between two domains (i.e., the real optical surface image domain and the real CT image domain), and ensure the accuracy of the anatomical structure in the generated images through edge structure guidance.

[0078] In the model training phase of the edge structure-guided CycleGAN, generator G1 is responsible for mapping the input real optical surface image to the corresponding pseudo CT image, and generator G2 is responsible for mapping the corresponding pseudo CT image generated by generator G1 back to the image in the style of optical surface image. Discriminator D1 is used to distinguish between real CT image and corresponding pseudo CT image generated by generator G1, and discriminator D2 is used to distinguish between real optical surface image and image in the style of optical surface image generated back by generator G2.

[0079] To enhance the edge and structural integrity of the pseudo-CT images generated by generator G1, an edge detection module (e.g., an edge detection operator) is embedded in the edge structure-guided CycleGAN. This module extracts edges from the input real optical surface image, obtaining an edge feature map which is then used as a conditional input to the dual generators. During model training, in addition to the adversarial loss and cycle consistency loss of traditional CycleGAN, an edge-aware loss is introduced to constrain the edge structure of the pseudo-CT images generated by generator G1 to maintain consistency with the edge structure of real CT images. Compared to the structural distortion problem that traditional CycleGAN is prone to, the edge structure-guided mechanism can accurately capture the anatomical structural features of medical images.

[0080] The process of generating pseudo-CT image data using edge structure-guided CycleGAN involves: combining the frequency domain features of the optical body surface image after discrete wavelet transform with the corresponding edge features. Figure 1 The same input is fed into generator G1; generator G1 outputs the frequency domain wavelet coefficients of the pseudo-CT image data through multi-scale feature fusion and transformation; finally, through inverse discrete wavelet transform (IDWT), the wavelet coefficients of all frequency bands generated are recombined and mapped back to the spatial domain to synthesize the final pseudo-CT image data. This scheme effectively solves the mode collapse problem of traditional generative adversarial networks (GANs), and through dual optimization of frequency domain acceleration and edge guidance, ensures that the generated pseudo-CT image data maintains the accuracy of anatomical structures while possessing realistic texture details and clear boundaries.

[0081] S102. Extract features from the pseudo-CT image data to obtain multi-level original features.

[0082] In the embodiments of this application, feature extraction refers to the process of extracting the implicit high-dimensional representations in an image using a deep neural network. Multi-level primal features refer to the set of feature maps obtained at different depth levels of a Convolutional Neural Network (CNN). These feature maps have different spatial resolutions and semantic abstractions: shallow features typically retain high spatial resolution and contain rich geometric details such as edges and textures; deep features have lower spatial resolution but contain highly abstract semantic information (such as organ category, tumor location discrimination, etc.).

[0083] In some embodiments of this application, a U-Net (U-shaped Network, U-shaped convolutional neural network) architecture is used as the backbone network to perform this step. For example... Figure 2 As shown in the backbone network, the input pseudo-CT image undergoes multiple convolution and pooling operations to generate a series of feature maps with gradually decreasing size and increasing channel number. These multi-level raw features constitute the basic data stream for subsequent accurate segmentation and motion prediction. Through multi-level extraction, the model ensures that it neither loses the edge details of small tumors nor fails to grasp the contextual information of macroscopic anatomical structures.

[0084] S103. Using a three-dimensional convolutional neural network-long short-term memory network, the pseudo-CT image data is processed to obtain spatial-temporal integrated features, and the spatial-temporal integrated features are fused into multi-level original features.

[0085] In the embodiments of this application, the 3D Convolutional Neural Network-Long Short-Term Memory Network (3D CNN-LSTM) is a hybrid deep learning model that combines spatial feature extraction capabilities with temporal dependency modeling capabilities. Specifically, the 3D CNN (Three-Dimensional Convolutional Neural Network) is used to extract the 3D spatial structural features of a single frame of pseudo-CT image, while the LSTM (Long Short-Term Memory) network is used to capture the evolutionary patterns between consecutive time-series image frames. The spatial-temporal integrated features refer to the high-dimensional feature vectors or feature maps output by the above model, which simultaneously encode the 3D anatomical structural information at the current moment and the motion trend information at historical moments.

[0086] In some embodiments of this application, such as Figure 2As shown on the left, the model receives a series of consecutive temporal image inputs (t=1, t=2, ..., t=10). The 3D CNN extracts the spatial features of each frame, and the LSTM unit processes these feature sequences and passes the hidden state. This design enables the generated spatial-temporal integrated features to not only know "where the tumor is now," but also, through the memory unit, know "where the tumor is moving."

[0087] The fusion operation refers to injecting the extracted spatial-temporal integrated features into the multi-level original features generated in step S102 through methods such as channel concatenation, element-wise addition, or attention-based weighting. This process endows the static original features with a dynamic temporal context, enabling subsequent segmentation networks to utilize motion prior information to assist in determining ambiguous tumor boundaries.

[0088] S104. Utilize cross-level and cross-spatial feature aggregation pyramid network, and determine the target area prediction features of pseudo-CT image data based on the fused multi-level original features.

[0089] In the embodiments of this application, the cross-level, cross-spatial feature aggregation pyramid network is a high-level feature processing architecture built on top of a backbone network, designed to solve the semantic gap and resolution misalignment problems in the transmission of features at different levels. Figure 2 Within the dashed box and Figure 4 As shown, this network achieves full interaction of features at different scales through top-down and bottom-up paths, as well as specific aggregation modules. Target region prediction features refer to highly discriminative feature maps refined by this network, specifically used to characterize the category and location of tumor target regions.

[0090] In some embodiments of this application, this step is implemented through a specific cross-level dual dependency aggregation module, which not only aligns deep semantics with shallow details but also enhances the sensitivity of features to small targets. For example... Figure 4 As shown, features at different levels (low-level, intermediate-level, and high-level) undergo complex routing and fusion within the pyramid structure. The resulting feature map exhibits strong robustness, effectively addressing noise or artifact interference that may exist in pseudo-CT images, providing a solid basis for high-precision pixel-level classification. Figure 4 The CBS (Convolution-BatchNorm-SiLU) module is a commonly used combination of "convolution + normalization + activation function". Its specific structure and function will not be elaborated here.

[0091] S105. Based on the target area prediction features, generate a static target area mask for a single time phase.

[0092] In the embodiments of this application, a single temporal phase refers to a specific discrete time point. A static target mask is a binary image or probability map obtained by decoding and thresholding the predicted features of the target region. In the mask, pixels belonging to the tumor target region are marked with a specific value (e.g., 1), and background regions are marked with another value (e.g., 0). This step is equivalent to completing the automatic segmentation of the pseudo-CT image at the current time.

[0093] In some embodiments of this application, such as Figure 2 The "Segmentation Prediction" module shown on the right uses convolutional layers (usually 1x1 convolutions) to map high-dimensional target region prediction features into a class probability map, and applies a sigmoid or softmax activation function to output the final segmentation result. This provides an extremely accurate geometric benchmark for subsequent dynamic prediction.

[0094] S106. Based on the static target area mask, determine the target area prediction location in the future prediction phase.

[0095] In the embodiments of this application, the future predicted phase refers to one or more time points (T+n) after the current time T, which is crucial for compensating for the mechanical delay of the radiotherapy system. The predicted target location refers to the expected three-dimensional spatial distribution of the tumor at future times.

[0096] As can be seen, the embodiments of this application, by integrating optical surface reconstruction and spatiotemporal feature extraction technologies, using 3DCNN-LSTM to capture dynamic evolution patterns, and combining feature aggregation pyramids to enhance the representation of small targets, not only achieve radiation-free pseudo-CT high-precision segmentation, but also achieve future dynamic target area prediction based on deformation field inference, effectively improving the real-time performance and accuracy of tumor tracking under respiratory motion.

[0097] In some embodiments of this application, a cross-level, cross-spatial feature aggregation pyramid network is used to determine the target region prediction features of pseudo-CT image data based on the fused multi-level original features, including: S201. In the fused multi-level original features, determine the first-level features, second-level features, and third-level features with successively decreasing levels.

[0098] In the embodiments of this application, "level" refers to the depth position of the feature map in a deep convolutional neural network. The deeper the network, the higher the level, the richer the semantic information, but the lower the spatial resolution. The first-level features, second-level features, and third-level features refer to the three most representative feature sets selected from high to low levels (i.e., from deep to shallow networks). For example, the first-level features can be the deepest output of the backbone network, possessing highly abstract semantics; the second-level features are located in the middle layers, balancing semantic and spatial information; and the third-level features are located in shallower layers, containing rich detail information such as edge textures. "Determination" refers to selecting these specific three layers from the numerous feature layers extracted in the preceding steps as the input nodes of the pyramid network.

[0099] In some embodiments of this application, such as Figure 4 As shown on the left, "high-level features," "intermediate-level features," and "low-level features" are selected as the objects determined in step S201, respectively. This selection method constructs a multi-scale feature system, laying a multi-resolution data foundation for subsequently addressing the common problems of varying sizes and shapes in tumor detection.

[0100] S202. Using the first cross-level and cross-space feature aggregation module, determine the first intermediate feature based on the first-level feature and the second-level feature.

[0101] In the embodiments of this application, the Cross-Level Cross-Spatial Feature Aggregation Module is used to receive feature inputs from two different levels or different processing stages and perform deep fusion on them. This cross-level cross-spatial feature aggregation can simultaneously handle semantic differences in channel dimension and size differences in spatial resolution. The first intermediate feature refers to the temporary feature map generated after the first aggregation operation, which initially fuses the deepest global semantic information and the local geometric information of the intermediate layer.

[0102] In some embodiments of this application, such as Figure 4 As shown in the lower left section, the first cross-level, cross-spatial feature aggregation module receives "high-level features" and "intermediate-level features" as input. During this process, the high-level features undergo upsampling to match the resolution of the intermediate-level features. Then, both are non-linearly fused in the cross-level, cross-spatial feature aggregation module, outputting the first intermediate feature. This allows deep abstract knowledge to be transferred to shallower layers, enhancing the network's ability to recognize ambiguous boundaries.

[0103] S203. Using the second cross-level and cross-space feature aggregation module, determine the second intermediate feature based on the first intermediate feature and the third level feature.

[0104] In the embodiments of this application, the second cross-level, cross-spatial feature aggregation module has a similar or identical structure to the aforementioned module, but processes different data objects. It is responsible for further transmitting the fused information to shallower layers of the network. The second intermediate feature refers to the feature map generated after the second aggregation, which not only contains the mixed information in the first intermediate feature, but also incorporates high-resolution texture details from the third-level features.

[0105] In some embodiments of this application, such as Figure 4 As shown in the upper left section, this cross-level and cross-spatial feature aggregation module receives the "first intermediate feature" from the lower level and the "low-level feature" from the original input. This is a continuation of the bottom-up path and the end point of the top-down information flow. Through this cascading method, the final generated second intermediate feature simultaneously possesses information elements from all three levels, which is particularly important for the localization of small-volume tumors.

[0106] S204. Using the third cross-level and cross-space feature aggregation module, determine the third intermediate feature based on the first intermediate feature, the second intermediate feature, and the first level feature.

[0107] In embodiments of this application, a third cross-level, cross-spatial feature aggregation module is used to perform a retrospective feature enhancement or refinement operation. The third intermediate feature is a feature map located deep within the network output. Notably, this step introduces skip connections across multiple stages, integrating the initial first-level features, the intermediate first intermediate features, and the later second intermediate features.

[0108] In some embodiments of this application, such as Figure 4 As shown in the lower right section, this cross-level and cross-spatial feature aggregation module is located at the end of the feature flow loop. This design forms a closed loop or a complex directed acyclic graph structure, which allows deep features to be collected again and downsampled after multiple upsampling and shallow fusion processes. This corrects and enhances the accuracy of high-level semantic representation and reduces information attenuation during long-path transmission.

[0109] S205. Based on the third intermediate feature, determine the first-level output feature corresponding to the first-level feature; based on the first intermediate feature and the second intermediate feature, determine the second-level output feature corresponding to the second-level feature; and based on the second intermediate feature, determine the third-level output feature corresponding to the third-level feature.

[0110] In the embodiments of this application, the first-level output features, the second-level output features, and the third-level output features correspond to the feature maps at three scales of the final output of the pyramid network, which will be directly used in the subsequent prediction head. This step involves organizing, convolutionally transforming, and adjusting the dimensions of the intermediate features flowing within the network to meet the output requirements.

[0111] In some embodiments of this application, such as Figure 4 As shown on the right, the network ultimately outputs "high-level output features," "intermediate-level output features," and "low-level output features." The specific path includes: the third intermediate feature undergoes convolution processing to directly form the high-level output feature; the first intermediate feature is combined with the transformed second intermediate feature to form the intermediate-level output feature; and the second intermediate feature undergoes convolution processing to directly form the low-level output feature. This multi-output design ensures the network has multi-scale predictive capabilities.

[0112] S206. Based on the first-level output features, the second-level output features, and the third-level output features, determine the target area prediction features of the pseudo-CT image data.

[0113] In the embodiments of this application, this step refers to summing the output features from the three scales mentioned above as the final target region prediction features. These features can be predicted independently (e.g., multi-scale prediction) or concatenated and then predicted uniformly. Figure 2 The right side corresponds to the arrows pointing to the three "segmentation prediction" boxes. This set comprehensively describes the target region characteristics of the pseudo-CT image at different resolutions.

[0114] As can be seen, the embodiments of this application construct a specific pyramid network architecture, utilize three cascaded cross-level and cross-spatial feature aggregation modules to establish a high-speed information pathway from deep to shallow, and design backtracking connections to enhance semantic expression. This scheme effectively integrates the high abstraction of deep semantics with the high resolution of shallow details, solving the problems of complex structure and variable scale in tumor segmentation, and improving the robustness and accuracy of the model in predicting target regions of different sizes.

[0115] In some embodiments of this application, a first cross-level, cross-spatial feature aggregation module is used to determine a first intermediate feature based on first-level features and second-level features, including: S301. Determine the global semantic association weights based on the first-level features.

[0116] In the embodiments of this application, the global semantic association weight refers to a weight vector generated based on the channel attention mechanism. The length of this weight vector is consistent with the number of channels in the first-level feature, and each value represents the importance of the corresponding channel to the target region recognition task. The global semantic association weight can be determined by extracting it from high-dimensional features through global statistics and nonlinear transformation.

[0117] In some embodiments of this application, combined with Figure 3 As shown in the upper part, the input is "high-level feature F". h (This corresponds to the first-level feature in this step, with dimensions C×H×W, where C represents the number of channels, H represents the height, and W represents the width) to generate the global semantic association weight W. g .

[0118] S302. The first-level features are weighted and adjusted using global semantic association weights to obtain the first-level features after explicit semantic enhancement.

[0119] In the embodiments of this application, weighted adjustment refers to performing a channel-by-channel multiplication operation (scaling) on ​​the feature map using a weight vector. The first-level features after explicit semantic enhancement (corresponding to...) Figure 3 F' in h () refers to the features after channel filtering.

[0120] Here, the term "explicit" does not simply refer to ordinary feature transformation, but specifically to the fact that the weight vector generated based on the channel attention mechanism has explicitly quantified the correlation between each feature channel and the target category. Therefore, this application explicitly represents which channels are important by calculating the weights directly generated from global statistics (i.e., global semantic association weights), and uses the explicit global semantic association weights to directly perform multiplicative relabeling on the first-level features.

[0121] In the first-level features after explicit semantic enhancement, semantic channels related to tumors and key organs (such as shape and texture) are amplified based on explicit global semantic association weights, while channel responses related to imaging noise and irrelevant background are suppressed, thereby achieving targeted enhancement of feature semantic expression.

[0122] In some embodiments of this application, such as Figure 3 The multiplication symbol (⊗) in the upper right corner represents the global semantic association weight W with dimension C×1×1. gBroadcast multiplication is performed with the first-level features of dimension C×H×W. This "explicit enhancement" means that the model no longer passively accepts all input information, but actively recalibrates the importance of features based on the global context, thereby enhancing the accuracy of high-level abstract semantic expression.

[0123] S303. Based on the second-level features, determine the multi-scale spatial dependency weights.

[0124] In the embodiments of this application, multi-scale spatially dependent weights refer to a weight map generated based on a spatial attention mechanism. This weight map encodes the importance of pixel positions in the spatial dimension (H×W). The key is "multi-scale," meaning that the weight generation process considers receptive fields of different sizes, enabling it to adapt to various size variations of tumors, from small nodules to large masses.

[0125] In some embodiments of this application, combined with Figure 3 The lower half of the diagram shows that the input is "low-level feature F". l (This corresponds to the second-level feature in this step, with dimensions C×H×W), to generate multi-scale spatially dependent weights W. s .

[0126] S304. The second-level features are weighted and adjusted using multi-scale spatial dependence weights to obtain the second-level features after selective spatial enhancement.

[0127] In the embodiments of this application, selective spatial enhancement refers to a focusing operation performed on the model at the spatial pixel level. The second-level features after selective spatial enhancement (corresponding to...) Figure 3 F' in l () refers to the feature map after spatial filtering, in which tumor boundaries and texture details are preserved and enhanced, while flat background areas are weakened.

[0128] In some embodiments of this application, such as Figure 3 The multiplication symbol (⊗) in the lower right corner indicates the multi-scale spatial dependency weights W generated. s Perform element-wise matrix multiplication with the original second-level features. Due to the weight W... s By incorporating multi-scale information, this weighted adjustment enables the second-level features to maintain high resolution while possessing adaptive capture capabilities for targets of different sizes. This allows subsequent processing to focus on the spatial coordinates that actually contain the lesions, rather than invalid body surface or air regions.

[0129] S305. Based on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement, determine the first intermediate feature.

[0130] In the embodiments of this application, the first intermediate feature (corresponding to) Figure 3 The rightmost F A The final output of this cross-level and cross-space feature aggregation module is a hybrid feature body that integrates strong semantics at high levels and strong details at low levels.

[0131] As can be seen, the embodiments of this application, by designing a unique dual attention enhancement path, introduce feature recalibration in the channel dimension to enhance semantics and introduce multi-receptive field aggregation in the spatial dimension to adapt to multi-scale targets, effectively solve the feature blurring problem caused by low soft tissue contrast in pseudo-CT images, so that the generated intermediate features can take into account both accurate classification of global semantics and fine local spatial positioning.

[0132] In some embodiments of this application, global semantic association weights are determined based on first-level features, including: S401. Perform global standard deviation pooling and global average pooling on the first-level features to obtain two types of global feature vectors.

[0133] In the embodiments of this application, Global Average Pooling (GAP) is a feature dimensionality reduction operation used to calculate the average distribution of feature values ​​within each channel, reflecting the overall strength of the semantic information represented by that channel. Global Standard Deviation Pooling (GSP), on the other hand, calculates the dispersion of feature values ​​within each channel, effectively capturing high-frequency information in the feature map, such as edges, areas with drastic texture changes, and abnormal lesions with significant contrast differences. The two types of global feature vectors refer to one-dimensional vectors of size C×1×1 (where C is the number of channels) generated after the above two pooling operations.

[0134] In some embodiments of this application, such as Figure 3 The upper branch structure shown represents the first-level input feature (F in the diagram). h The data (with dimensions C×H×W) is split into two paths. One path performs global average pooling to generate a descriptor representing the average intensity of the background and the subject; the other path performs global standard deviation pooling to generate a descriptor representing texture complexity and edge sharpness. By introducing standard deviation pooling, the insensitivity of single average pooling to detailed information is compensated for.

[0135] S402. Stack the two types of global feature vectors along the channel dimension to generate a global information induction vector.

[0136] In the embodiments of this application, stacking refers to the operation of concatenating two or more feature vectors together along the channel axis, without changing the spatial dimension of the features (all are 1x1 here), but multiplying the number of channels. Global information induction vector (corresponding to...) Figure 3 V in g () refers to the concatenated feature vector, whose channel dimension usually becomes 2C. The global information induction vector is a highly compressed information carrier that fully summarizes the "average state" and "distribution state" of the input features.

[0137] In some embodiments of this application, such as Figure 3 The arrows of two C×1×1 cuboids converge to point to a longer 2C×1×1 cuboid (labeled V). g The process of ( ) is the intuitive manifestation of this step. This joint representation method allows subsequent network layers to make decisions based on both the strength and variability of features. For example, some channels may have low average response but large standard deviation (meaning there are small lesions that are locally highlighted against the background). These channels can still be recognized by the network and assigned high weights after stacking.

[0138] S403. Generate global semantic association weights based on global information induction vectors.

[0139] In the embodiments of this application, the generation process of global semantic association weights involves dimensionality reduction and nonlinear activation, aiming to convert high-dimensional joint descriptors into standardized weight coefficients. Global semantic association weights (corresponding to...) Figure 3 W in g The final result is a vector whose size is restored to C×1×1, which is used to filter the original channels.

[0140] In some embodiments of this application, such as Figure 3 As shown, a 1×1 convolutional layer (fully connected layer) is used to process the 2C-dimensional global information induction vector, not only compressing the number of channels from 2C back to C, but also using the convolutional kernel to learn the dependencies between channels. Subsequently, a sigmoid activation function is connected, restricting the output value of each channel to the (0, 1) interval, which serves as the final weight coefficient. Values ​​close to 1 indicate that the channel contains key tumor or organ semantics and should be enhanced; values ​​close to 0 indicate that the channel is mostly noise and should be suppressed.

[0141] As can be seen, the embodiments of this application introduce global standard deviation pooling and global average pooling in parallel to construct a global information induction vector containing second-order statistical information (discretion). This enables the generated global semantic association weights to not only identify significant targets, but also to keenly capture small lesion features with high variability, thereby improving the comprehensiveness and accuracy of feature selection.

[0142] In some embodiments of this application, multi-scale spatial dependency weights are determined based on second-level features, including: S501. The second-level features are input in parallel into three deep-dilated convolutional modules to obtain three spatial feature responses with different receptive fields. Among them, the kernel size or dilation rate of any two deep-dilated convolutional modules is different.

[0143] In the embodiments of this application, the deep-dilated convolution module refers to a convolutional unit that combines the efficiency of depthwise separable convolution with the large receptive field of dilated convolution. While performing convolution on each input channel individually, it expands the coverage of the convolutional kernel by introducing a dilation rate parameter, thereby capturing a wider range of contextual information without increasing the number of parameters. Different receptive fields mean that the size of the input image region that the convolutional kernel can "see" is different; a small receptive field focuses on local details (such as tumor edges), while a large receptive field focuses on global structure (such as organ location). Spatial feature response refers to the feature map output after the above convolutional processing, which reflects the degree of activation of spatial information in the image at a specific scale.

[0144] In some embodiments of this application, combined with Figure 3 As shown on the left side of the lower half, the three parallel deep dilated convolutional modules are configured as follows: the first module uses a 3×3 kernel with a dilation rate of 1 (i.e., standard convolution) to extract high-resolution local texture features; the second module uses a 3×3 kernel with a dilation rate of 3 to moderately expand the receptive field to capture medium-scale structures; and the third module uses a 5×5 kernel with a dilation rate of 3 to significantly expand the receptive field to cover the macroscopic background context. This differentiated configuration ensures that the network can simultaneously process target region features of different sizes, and the three spatial feature responses generated correspond to spatial correlation information at near, medium, and far distances, respectively.

[0145] S502. Using a grouping channel aggregation strategy, the spatial feature responses of three different receptive fields are grouped and recombined along the channel dimension to generate multi-scale spatial information aggregation features.

[0146] In the embodiments of this application, the grouped channel aggregation strategy refers to a non-linear feature fusion method. It does not simply concatenate features from different sources, but rather treats them as different feature groups and arranges them in an orderly manner. Multi-scale spatial information aggregation features (corresponding to...) Figure 3 F in sThis refers to a comprehensive feature body that integrates information from the three scales mentioned above. It typically has three times the number of channels (if each branch outputs C channels, then the aggregated feature body has 3C channels, such as...). Figure 3 The example shown is a set of 3C×H×W.

[0147] In some embodiments of this application, such as Figure 3 Middle finger F s As shown by the concatenation symbol (C), spatial feature responses at three different scales are concatenated along the channel dimension. This grouping and recombination method preserves the independence of multiple scales, while also allowing subsequent network layers to learn complementary relationships between features at different scales through cross-channel interaction, such as using large-scale features to help eliminate false positive noise in small-scale features.

[0148] S503. Generate multi-scale spatial dependency weights based on multi-scale spatial information aggregation features.

[0149] In the embodiments of this application, the generation process of multi-scale spatial dependency weights aims to compress wide-channel multi-scale features into a compact spatial weight map. Multi-scale spatial dependency weights (corresponding to...) Figure 3 W in s ) is a weight matrix with spatial resolution, whose numerical value indicates the importance of the corresponding spatial location pixel.

[0150] In some embodiments of this application, such as Figure 3 China F s As shown in the flowchart on the right, this step first uses a 3×3 grouped convolution (number of groups = C) to process multi-scale spatial information aggregation features. Here, the number of groups is set equal to the number of channels C (which may actually correspond to the number of channels in the original input, or compress the 3C features back to C channels). The aim is to reduce computational cost while promoting local fusion of features at different scales within the group. Subsequently, the Sigmoid activation function is used to map the feature values ​​to probabilistic weights W. s (Dimensions are C×H×W). This weight map exhibits a high response in the tumor region and a low response in the background region, thereby achieving spatial weighting of the feature map.

[0151] As can be seen, the embodiments of this application, by deploying deep dilated convolutional modules with different kernel sizes and different dilation rates in parallel and combining them with a grouping channel aggregation strategy, not only significantly expand the effective receptive field of the network, enabling it to adapt to the detection needs of tumors of different volumes, but also effectively reduce computational complexity. This allows the generated multi-scale spatially dependent weights to accurately eliminate irrelevant background interference and focus on the real target region that is significant at multiple scales.

[0152] In some embodiments of this application, a first intermediate feature is determined based on the semantically explicitly enhanced first-level features and the selectively spatially enhanced second-level features, including: S601. Perform first-level semantic matching processing on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement to obtain the first semantic matching feature and the second semantic matching feature.

[0153] In the embodiments of this application, the first-level semantic matching processing refers to the transformation operation performed to eliminate the differences in semantic distribution of features at different levels along the channel dimension. Since the first-level features (high-level) and the second-level features (low-level) may differ in their level of feature abstraction and number of channels, direct fusion would lead to semantic misalignment. This processing aims to map both to a unified high-dimensional semantic space. The first semantic matching feature and the second semantic matching feature refer to intermediate features that are semantically compatible and obtained by transforming the high-level features and low-level features, respectively.

[0154] S602. Based on the first semantic matching feature and the second semantic matching feature, determine the semantic alignment feature.

[0155] In the embodiments of this application, semantic alignment features refer to fused features generated at a high-level resolution scale. It focuses on utilizing low-level texture details to supplement high-level abstract semantics, correcting potential boundary ambiguity issues in high-level features.

[0156] In some embodiments of this application, such as Figure 3 As shown by the addition symbol (⊕) in the upper right corner, the first semantic matching feature and the second semantic matching feature generated above are added element-wise. The result of the addition is then smoothed by a 3×3 convolutional layer to generate a semantic alignment feature. This feature retains the overall structure of the high-level features while injecting the key spatial index information carried by the low-level features.

[0157] S603. Perform second-level spatial matching processing on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement to obtain the first spatial matching features and the second spatial matching features.

[0158] In the embodiments of this application, the second-level spatial matching process is the process opposite to S601, aiming to unify the two to a high-resolution spatial scale of low-level features. The first spatial matching feature and the second spatial matching feature are intermediate features that are mutually compatible in spatial resolution, obtained by transforming high-level features and low-level features, respectively.

[0159] S604. Based on the first spatial matching feature and the second spatial matching feature, determine the spatial alignment feature.

[0160] In the embodiments of this application, spatial alignment features refer to fused features generated at a low-level resolution scale. It focuses on utilizing the class judgment capabilities of higher levels to guide the detailed segmentation of lower levels and suppress background noise in the lower-level features.

[0161] In some embodiments of this application, such as Figure 3 As shown by the addition symbol (⊕) in the lower right corner, the first spatial matching feature and the second spatial matching feature are added element-wise. The result of the addition is also processed by a 3×3 convolutional layer to generate a spatial alignment feature. This feature has fine edge information, and the category of each pixel is confirmed by high-level semantics.

[0162] S605. Based on semantic alignment features and spatial alignment features, concatenate them in the channel dimension to generate cross-level and cross-space aggregated features, and use them as the first intermediate features.

[0163] In the embodiments of this application, cross-level and cross-space aggregation features (corresponding to) Figure 3 The rightmost F A The final output is ). The concatenation operation merges the two sets of aligned features generated based on different emphases (semantics or space) into a feature body with extremely rich information.

[0164] In some embodiments of this application, such as Figure 3 As shown by the rightmost concatenation symbol (C), because the semantic alignment features and spatial alignment features have different resolutions, the semantic alignment features need to be upsampled again before concatenation to make their resolution consistent with that of the spatial alignment features. The final generated feature F... A It simultaneously incorporates strong semantic guidance from higher levels and detailed spatial descriptions from lower levels, serving as the first intermediate feature passed to subsequent modules.

[0165] As can be seen, the embodiments of this application design a bidirectional cross-alignment mechanism to perform two independent matching and fusion operations in the low-resolution space belonging to high-level semantics and the high-resolution space belonging to low-level details, and finally unify the two, which solves the problem of misalignment and loss of semantic and spatial information in pseudo-CT images during transmission, and maximizes the preservation of the category and geometric features of the target area.

[0166] In some embodiments of this application, the first-level semantically enhanced features and the second-level semantically enhanced features are subjected to first-level semantic matching processing to obtain first semantic matching features and second semantic matching features, including: S701. The first convolutional layer is used to perform semantic feature transformation on the first-level features after explicit semantic enhancement, while keeping the semantic level of the first-level features after explicit semantic enhancement unchanged, to obtain the first semantic matching feature.

[0167] In the embodiments of this application, the first convolutional layer refers to the convolutional operation unit configured on the high-level branch of the semantic alignment path, typically using a 1×1 convolutional kernel. Semantic feature transformation refers to the linear combination of feature channels to adapt to the number of channels required for subsequent fusion, and may include normalization (such as batch normalization (BN)) and activation functions (such as rectified linear unit (ReLU)). Maintaining the semantic level unchanged means that this operation does not change the spatial resolution (size H×W) of the feature map, but only adjusts it in the channel dimension, making it the baseline feature at this level. The first semantic matching feature is the processed output, which retains the original deep abstract semantic attributes of the high-level features.

[0168] In some embodiments of this application, such as Figure 3 As shown in the upper right part, for the input F' h The first convolutional layer performs convolution operations using a 1×1 kernel. Assume the input channels are C. h The output channel is set to the number of fusion channels C. embed This step will change the feature dimension from C h Mapping to C embed .

[0169] S702. The second convolutional layer is used to perform semantic feature transformation on the second-level features after selective space enhancement, and the semantic level of the second-level features after selective space enhancement is improved to obtain the second semantic matching features.

[0170] In the embodiments of this application, the second convolutional layer refers to the convolutional operation unit configured on the low-level branch of the semantic alignment path, typically using a convolutional kernel with a stride greater than 1. Semantic level enhancement refers to reducing the spatial resolution of the feature map through downsampling operations, aligning it with the smaller size of the high-level features, thereby spatially simulating the receptive field of the high-level semantics, and giving the low-level features a "semantic upgrade" in form. The second semantic matching feature is the processed output; although it originates from the low level, its spatial scale is now consistent with the high-level features.

[0171] In some embodiments of this application, such as Figure 3 As indicated by the arrow pointing upwards to the right in the crossover line on the right, for the input F' l(Size 2H×2W), the second convolutional layer performs 3×3 convolutions with a stride of 2. This not only adjusts the resolution to H×W, but also encodes shallow detail information into the deep semantic space through convolution, enabling it to directly interact with the first semantic matching features.

[0172] Similarly, in some embodiments of this application, second-level spatial matching processing is performed on the semantically explicitly enhanced first-level features and the selectively spatially enhanced second-level features to obtain first spatial matching features and second spatial matching features, including: S703. The third convolutional layer is used to perform spatial feature transformation on the first-level features after explicit semantic enhancement, and the spatial resolution of the first-level features after explicit semantic enhancement is improved to obtain the first spatial matching features.

[0173] In the embodiments of this application, the third convolutional layer refers to the operational unit configured on the high-level branch of the spatial alignment path, and its core typically incorporates an upsampling algorithm (such as bilinear interpolation or deconvolution). Spatial resolution enhancement refers to enlarging the smaller size of the high-level features so that they can cover more pixels, thereby aligning with the high-resolution grid of the low-level features. The first spatial matching feature is the processed output, which diffuses deep semantic information to a finer spatial coordinate system.

[0174] In some embodiments of this application, such as Figure 3 As shown by the arrow pointing downwards to the right in the crossover line on the right (the dashed arrow marked "Up-sampling"), for the input F' h (With dimensions of H×W), a 1×1 convolution is first performed to adjust the channels, followed by a 2x bilinear interpolation upsampling to achieve a resolution of 2H×2W. This process realizes the projection of high-level information into the low-level space, providing pixel-level class priors for low-level features.

[0175] S704. The fourth convolutional layer is used to perform spatial feature transformation on the second-level features after selective spatial enhancement, while keeping the spatial resolution of the second-level features after selective spatial enhancement unchanged, to obtain the second spatial matching features.

[0176] In the embodiments of this application, the fourth convolutional layer refers to the convolutional operation unit configured on the low-level branch of the spatial alignment path, typically using a 1×1 convolutional kernel. Maintaining the spatial resolution means that this operation does not change the high resolution of the input feature map (size 2H×2W), but only performs channel dimension adaptation. The second spatial matching feature is the processed output, which retains the rich edge and texture details originally present in the low-level features.

[0177] In some embodiments of this application, such as Figure 3 As shown in the lower right part, for the input F'l The fourth convolutional layer performs convolution operations using a 1×1 kernel, mapping its channel count to C. embed Thus, the second spatial matching features become the baseline features at this resolution scale, waiting to be fused with the upsampled first spatial matching features.

[0178] As can be seen, before feature fusion, the embodiments of this application achieve dual interoperability of heterogeneous features at the semantic level (low resolution) and spatial level (high resolution), ensuring that each addition fusion is performed on a strictly aligned grid, eliminating feature ambiguity caused by scale differences, and significantly improving the accuracy of target area prediction in boundary localization.

[0179] In some embodiments of this application, a third intermediate feature is determined using a third cross-level cross-space feature aggregation module based on a first intermediate feature, a second intermediate feature, and a first-level feature, including: S801. Based on the second intermediate feature, perform spatial downsampling to obtain the fourth intermediate feature.

[0180] In the embodiments of this application, the second intermediate feature refers to the feature map output at a higher resolution level during the network encoder or feature extraction stage. Spatial downsampling refers to the operation of reducing the spatial size (height and width) of the feature map and increasing the receptive field, using methods such as convolution with a stride of 2, max pooling, or average pooling. The fourth intermediate feature refers to the transitional feature whose resolution matches that of the first intermediate feature after one downsampling step. This step aims to address the problem of inconsistent spatial dimensions between features at different levels, preparing for subsequent fusion.

[0181] In some embodiments of this application, combined with Figure 4 Assuming the resolution of the second intermediate feature is H / 4×W / 4 and the number of channels is 128, this step downsamples its resolution to H / 8×W / 8 through a 3×3 convolution with a stride of 2. This facilitates the transmission of information to deeper network layers, and the generated fourth intermediate feature provides contextual reference from shallow layers for subsequent feature fusion.

[0182] S802. Based on the fourth intermediate feature and the first intermediate feature, stack them in the channel dimension to obtain the fifth intermediate feature.

[0183] In the embodiments of this application, stacking along the channel dimension refers to splicing feature maps from different sources in the depth direction. This not only preserves their original information but also increases the richness of the features. The fifth intermediate feature refers to this spliced ​​composite feature, which carries a mixed information flow from a shallower layer (through the fourth intermediate feature) and the current layer (the first intermediate feature).

[0184] In some embodiments of this application, such as Figure 4 As shown, the first intermediate feature (from the previous aggregation module) and the fourth intermediate feature (from shallow downsampling) merge before entering the next level of processing. This design constitutes a dense feature reuse mechanism, which not only strengthens feature propagation but also facilitates the backpropagation of gradients. The generated fifth intermediate feature acts as an information hub, carrying richer multi-scale semantics.

[0185] S803. Based on the fifth intermediate feature, perform spatial downsampling to obtain the sixth intermediate feature.

[0186] In embodiments of this application, the sixth intermediate feature refers to a feature generated to further pass information to deeper network layers. This step performs spatial downsampling again to reduce the resolution of the feature map to match that of the first-level features (typically the deepest features).

[0187] In some embodiments of this application, the fifth intermediate feature undergoes a convolution operation (e.g., with a stride of 2), and its resolution is further reduced (e.g., from H / 8×W / 8 to H / 16×W / 16). The resulting sixth intermediate feature has a further enhanced level of semantic abstraction and is ready to interact with the deepest global features in the network.

[0188] S804. Using the third cross-level cross-space feature aggregation module, the third intermediate feature is determined based on the sixth intermediate feature and the first level feature.

[0189] In the embodiments of this application, the internal structure of the third cross-level cross-space feature aggregation module is consistent with that of the aforementioned module (such as the first cross-level cross-space feature aggregation module), and it also includes a dual enhancement mechanism of channel attention and spatial attention. The third intermediate feature is the output of the third cross-level cross-space feature aggregation module, and it is also the advanced product of the entire feature aggregation stage.

[0190] In some embodiments of this application, such as Figure 4 As shown, the sixth intermediate feature (as a relatively high-resolution input, although it has been downsampled twice) and the first-level feature (as the lowest-resolution input) are fed into the cross-level, cross-space feature aggregation module. Within this module, the semantic explicit enhancement, selective spatial enhancement, and bidirectional cross-alignment strategies described in the preceding embodiments are used to deeply fuse the deepest global semantics of the encoder with the shallow and mid-level details passed through layers, generating the third intermediate feature.

[0191] As can be seen, the embodiments of this application form a feature transmission chain from shallow to deep by constructing a cascaded downsampling and stacking path, so as to gradually fuse the intermediate results generated by the preceding modules with the original encoded features and pass them to the deepest layer. This ensures that the third cross-level and cross-space feature aggregation module can work in a highly rich context environment. As a result, the generated third intermediate features have both the robustness of deep semantics and the coherence of shallow details, effectively improving the overall accuracy of pseudo-CT image prediction.

[0192] In some embodiments of this application, determining the target prediction location at a future prediction time phase based on a static target mask includes: S901. Using the regression prediction layer in a three-dimensional convolutional neural network-long short-term memory network, a deformation registration field pointing to the future prediction phase is generated.

[0193] In the embodiments of this application, the regression prediction layer refers to the output layer located at the end of the three-dimensional convolutional neural network-long short-term memory network, typically composed of convolutional layers. Its task is not to output classification probabilities, but rather to output continuous numerical values, specifically voxel displacement vectors. The deformation registration field pointing to the future prediction phase is a three-dimensional tensor with the same spatial dimensions (D×H×W) as the input image, with 3 channels (corresponding to the x, y, and z directions). Each vector in this field represents the relative displacement required for a pixel at the corresponding position to move from the current static time to a future time.

[0194] In some embodiments of this application, the deformation registration field pointing to the future predicted phase can be learned through a two-stage registration strategy that combines a free-form deformation (FFD) model and a viscous fluid flow model.

[0195] Specifically, the registration process first applies a free deformation registration stage based on the sparse distribution of control points. The free deformation model uses a B-spline function as the interpolation kernel, defining a regularly arranged grid of control points within the image domain. Deformation of surrounding pixels is driven by moving the positions of these control points. This stage primarily captures large-scale, global anatomical displacements between images, such as the overall expansion and contraction of the thoracic cavity caused by respiratory movements. Due to the relatively sparse distribution of control points, this model has strong smoothing constraints, preventing unnatural topological folds in the image and providing a coarse but robust initial deformation estimate for subsequent steps.

[0196] The model then enters the viscous fluid registration stage to obtain a detailed deformation field estimate. Based on the Navier-Stokes equations in fluid mechanics, this model treats the source image as a viscous fluid that undergoes flow deformation under the influence of external forces (i.e., the driving force propelling the source image towards the target image for matching). Compared to the parameterized FFD model, the viscous fluid model has extremely high degrees of freedom, capable of simulating the independent, minute displacements of each pixel, thus accurately capturing the non-rigid deformation details within and around the tumor.

[0197] In solving the viscous fluid model, the fluid velocity calculation is not performed using the time-consuming successive over-relaxation method, but is simplified to a convolution operation between the force field and a specific filter. The new displacement field is determined by the increment caused by the current displacement in the previous iteration and the currently calculated fluid velocity. Furthermore, to ensure the physical validity of the displacement field, Gaussian filtering is applied to smooth the calculated displacement field, ensuring the continuity of the deformation field. This optimization process continues iterating until the preset maximum number of optimization iterations is reached or the cost function converges.

[0198] By fusing the global constraint capabilities of a free-form deformation model with the high-degree-of-freedom local deformation capabilities of a viscous fluid model, a high-quality deformation registration field that combines topology preservation and local accuracy can be generated. This coarse-to-fine strategy can not only effectively capture large-scale respiratory motions but also accurately characterize minute changes in tumor morphology, improving the spatial accuracy of dynamic target region prediction.

[0199] S902. Based on the deformation registration field pointing to the future prediction phase, the static target area mask is deformed to obtain the target area prediction position under the future prediction phase.

[0200] In the embodiments of this application, deformation processing refers to spatial transformation operations, which are typically implemented using the resampling mechanism in a Spatial Transformer Network (STN). The predicted target location in the future prediction phase refers to the geometric shape and spatial coordinates of the target area at a future time (e.g., 200 milliseconds later) after displacement transformation.

[0201] In some embodiments of this application, the trilinear interpolation algorithm can be used as the resampling kernel in the resampling mechanism. For each coordinate point P in the future predicted temporal image... t+1 Based on the displacement vector V in the deformation registration field, its corresponding coordinate P in the static target area mask is calculated in reverse. t =P t+1 -V(P t+1The interpolation process takes the mask value at that coordinate as the prediction result. Since the mask is a binary image, the interpolated result may contain decimals. A threshold (e.g., 0.5) is typically set to rebinarize it, thus obtaining an accurate target area boundary. This method avoids the shape fragmentation problem that may result from directly predicting the future mask, ensuring the consistency of the topology.

[0202] As can be seen, the embodiments of this application introduce an indirect prediction strategy based on deformation fields, which transforms the complex problem of tumor shape prediction into a relatively simple problem of voxel motion estimation, in order to generate a high-precision deformation registration field to drive the static mask to deform, thereby achieving accurate tracking of the position of the moving target area in the future while ensuring the topological integrity of the anatomical structure.

[0203] In some embodiments of this application, the overall network of the pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation can be named a cross-level and cross-spatial feature aggregation prediction network. This network includes at least one model used in at least one of the above embodiments. The model training process for the cross-level and cross-spatial feature aggregation prediction network may specifically include: S1001. Acquire multi-phase computed tomography data and establish the temporal correspondence between the reference phase and the other phases.

[0204] In embodiments of this application, multi-temporal computed tomography (CT) data, such as four-dimensional computed tomography (4D-CT), refers to image data containing information about the complete human respiratory cycle. Figure 1 As shown, ph1, ph2, ph3 to ph10 represent different phases in the respiratory cycle, for example, dividing a complete respiratory cycle into 10 equal phases. The reference phase is usually selected as the phase where respiratory movements are relatively stable, for example... Figure 1 In the text, pH 1 (which can correspond to the end of inspiration or the end of expiration); the other phases correspond to... Figure 1 Ph2 through Ph10. This step aims to build the foundational spatiotemporal data pool required for model training.

[0205] S1002. Based on the CT image of the reference time phase and the CT images of other time phases, generate the corresponding deformation registration field using the image registration algorithm.

[0206] In the embodiments of this application, the image registration algorithm refers to the calculation process (such as optical flow or B-spline registration) used to calculate the geometric transformation relationship between two images. Deformation registration field ( Figure 1 The deformation fields 1 to 10 shown are three-dimensional vector fields describing voxel displacement.

[0207] In some embodiments of this application, such as Figure 1 Chinese CTph1 With CT ph2 The connection relationship between them is shown, referring to the CT images of the time phase. ph1 CT images of the target time phase ph2 The input registration module calculates deformation field 1, which describes the tissue movement from ph1 to ph2 (the number here corresponds to the target time phase). Similarly, as... Figure 1 Chinese CT ph1 With CT ph3 CT scan ph1 With CT ph10 As shown in the connection relationship, deformation registration fields pointing to each future time phase are generated respectively. These deformation fields accurately record the non-rigid deformation information of human anatomical structures during respiratory movements, serving as the basis for the supervisory network to learn motion laws during training.

[0208] S1003. Determine the static target area mask at the reference time phase as the initial anatomical structure label.

[0209] In the embodiments of this application, such as Figure 1 As shown, surface pH1 data was used to generate pseudo-CT scans. ph1 Then, through segmentation operations, the mask GTV of the gross tumor volume (GTV) is obtained. ph1 In the context of model training, GTV ph1 (Static target mask under reference phase) is usually determined by clinicians using actual CT scans. ph1 The value obtained by manually drawing the above is used as the most accurate initial truth value.

[0210] S1004. Based on the deformation registration field, perform spatial transformation on the static target area mask under the reference time phase to generate dynamic target area labels under the other time phases, obtain the training dataset, and use the training dataset for model training.

[0211] In the embodiments of this application, such as Figure 1 As shown, the deformation field 1 generated in S1002 is applied to the GTV determined in S1003. ph1 By using spatial resampling technology, the tumor contour at time ph1 is "extended" to time ph2, thereby automatically generating GTV. ph2 Similarly, using the corresponding deformation fields, GTV is respectively... ph1 Propagate to ph3...ph10, generating GTV ph3 To GTV ph10 .thus, Figure 1 The image shown is from GTV ph1 Go to GTV ph10The full-sequence mask was established. These dynamic GTV sequences, calculated based on real CT deformation, can be used as training datasets for model training. The prediction error of the cross-level and cross-spatial feature aggregation prediction network is calculated, thereby driving the update of model parameters and completing the model training of the cross-level and cross-spatial feature aggregation prediction network.

[0212] In some embodiments of this application, a composite loss function is used for parameter optimization in the model training process of the cross-level and cross-spatial feature aggregation prediction network. This composite loss function is a weighted combination of mutual information similarity loss and continuity loss.

[0213] Mutual information similarity loss measures the statistical correlation between the predicted image (or predicted mask) and the true target image (or true mask) after the model-generated deformation registration field is applied to the source image. Mutual information is a metric in information theory used to measure the interdependence between two random variables. Compared to mean squared error (MSE), which only focuses on the absolute difference in pixel values, mutual information can capture the more complex nonlinear gray-level mapping relationships between images. Maximizing the mutual information between the predicted and true results is equivalent to minimizing this mutual information similarity loss, thereby driving the model to learn a deformation field that accurately represents the correspondence of anatomical structures.

[0214] The continuity loss acts as a regularization term, constraining the smoothness of the generated deformation registration field and its gradient. In biological tissue deformation, the displacement of adjacent pixels should be gradual rather than abrupt. This loss function is typically implemented by calculating the L2 norm of the spatial gradient of the deformation field, penalizing non-physical deformations that involve drastic spatial transformations that could lead to image tearing or overlap.

[0215] In some embodiments of this application, to balance registration accuracy and the physical plausibility of the deformation field, specific weighting coefficients are set to weight the sum of the two losses. Specifically, the weighting coefficient for similarity loss is set to 10, and the weighting coefficient for continuity loss is set to 2. This configuration indicates that the model prioritizes matching accuracy of anatomical structures (high-weight mutual information) during training, while also applying appropriate smoothing constraints (low-weight continuity) to prevent overfitting. Regarding the training strategy, the batch size is set to 2, the initial learning rate is 1e-4, and a StepLR (Step Learning Rate) scheduler is used, decaying the learning rate to 0.9 times its original value every 5 epochs. The total training epochs are set to 50 to ensure the model fully converges to the optimal solution.

[0216] By constructing a weighted combination of similarity loss based on mutual information and continuity loss based on gradient, we can not only ensure that the predicted target area is highly consistent with the real situation in terms of morphology, but also ensure that the predicted deformation field conforms to the physical deformation law of biological tissues. Thus, while ensuring high-precision registration, we can effectively avoid the generation of non-realistic deformations and improve the reliability and robustness of the model in clinical applications.

[0217] Secondly, embodiments of this application provide a pseudo-CT image target region prediction system with cross-level and cross-spatial feature aggregation. The pseudo-CT image target region prediction system with cross-level and cross-spatial feature aggregation is used to perform the pseudo-CT image target region prediction method with cross-level and cross-spatial feature aggregation as described in any of the above embodiments.

[0218] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for predicting target regions in pseudo-CT images by cross-level and cross-spatial feature aggregation, characterized in that, The pseudo-CT image target region prediction method involving cross-level and cross-spatial feature aggregation includes: Based on optical surface images, pseudo-CT image data is generated. Feature extraction is performed on the pseudo-CT image data to obtain multi-level original features; The pseudo-CT image data is processed using a three-dimensional convolutional neural network-long short-term memory network to obtain spatial-temporal integrated features, and the spatial-temporal integrated features are then fused into the multi-level original features. By utilizing a cross-level and cross-spatial feature aggregation pyramid network, and based on the fused multi-level original features, the target area prediction features of the pseudo-CT image data are determined. Based on the target region prediction features, a static target region mask for a single time phase is generated; Based on the static target mask, the target prediction location at the future prediction time phase is determined.

2. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 1, characterized in that, The method of using a cross-level, cross-spatial feature aggregation pyramid network to determine the target region prediction features of the pseudo-CT image data based on the fused multi-level original features includes: In the fused multi-level original features, the first-level features, second-level features, and third-level features with successively decreasing levels are identified; Using the first cross-level and cross-space feature aggregation module, a first intermediate feature is determined based on the first-level feature and the second-level feature; Using the second cross-level and cross-space feature aggregation module, the second intermediate feature is determined based on the first intermediate feature and the third-level feature; Using the third cross-level and cross-space feature aggregation module, the third intermediate feature is determined based on the first intermediate feature, the second intermediate feature, and the first level feature; Based on the third intermediate feature, the first-level output feature corresponding to the first-level feature is determined; based on the first intermediate feature and the second intermediate feature, the second-level output feature corresponding to the second-level feature is determined; and based on the second intermediate feature, the third-level output feature corresponding to the third-level feature is determined. The target area prediction features of the pseudo-CT image data are determined based on the first-level output features, the second-level output features, and the third-level output features.

3. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 2, characterized in that, The first cross-level, cross-spatial feature aggregation module, based on the first-level features and the second-level features, determines the first intermediate feature, including: Based on the first-level features, determine the global semantic association weights; The first-level features are weighted and adjusted using the global semantic association weights to obtain the semantically enhanced first-level features. Based on the second-level features, determine the multi-scale spatial dependency weights; The second-level features are weighted and adjusted using the multi-scale spatial dependency weights to obtain the second-level features after selective spatial enhancement. The first intermediate feature is determined based on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement.

4. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 3, characterized in that, The step of determining the global semantic association weight based on the first-level features includes: The first-level features are subjected to global standard deviation pooling and global average pooling respectively to obtain two types of global feature vectors; The two types of global feature vectors are stacked along the channel dimension to generate a global information summarization vector. The global semantic association weights are generated based on the global information induction vector.

5. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 3, characterized in that, The step of determining multi-scale spatial dependency weights based on the second-level features includes: The second-level features are input in parallel into three deep-dilated convolutional modules to obtain three spatial feature responses with different receptive fields. Among them, the kernel size or dilation rate of any two deep-dilated convolutional modules is different. By using a grouping channel aggregation strategy, the spatial feature responses of the three different receptive fields are grouped and recombined along the channel dimension to generate multi-scale spatial information aggregation features. Based on the multi-scale spatial information aggregation features, the multi-scale spatial dependency weights are generated.

6. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 3, characterized in that, The determination of the first intermediate feature based on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement includes: The first-level semantic matching process is performed on the first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement, respectively, to obtain the first semantic matching features and the second semantic matching features. Based on the first semantic matching feature and the second semantic matching feature, determine the semantic alignment feature; The first-level features after explicit semantic enhancement and the second-level features after selective spatial enhancement are subjected to second-level spatial matching processing to obtain the first spatial matching features and the second spatial matching features. Based on the first spatial matching feature and the second spatial matching feature, spatial alignment features are determined; Based on the semantic alignment features and the spatial alignment features, they are concatenated along the channel dimension to generate cross-level and cross-space aggregated features, which are then used as the first intermediate features.

7. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 6, characterized in that, The first-level semantic matching process is performed on the explicitly semantically enhanced first-level features and the selectively spatially enhanced second-level features to obtain first semantic matching features and second semantic matching features, including: The first semantic matching feature is obtained by using the first convolutional layer to perform semantic feature transformation on the first-level features after explicit semantic enhancement, while keeping the semantic level of the first-level features after explicit semantic enhancement unchanged. The second convolutional layer is used to perform semantic feature transformation on the second-level features after selective space enhancement, and semantic level enhancement is performed on the second-level features after selective space enhancement to obtain the second semantic matching feature; The second-level spatial matching processing is performed on the semantically explicitly enhanced first-level features and the selectively spatially enhanced second-level features to obtain first spatial matching features and second spatial matching features, including: The first spatial matching feature is obtained by using the third convolutional layer to perform spatial feature transformation on the semantically explicitly enhanced first-level features and improving the spatial resolution of the semantically explicitly enhanced first-level features. The second spatial matching feature is obtained by using the fourth convolutional layer to perform spatial feature transformation on the second-level features after selective spatial enhancement, while keeping the spatial resolution of the second-level features after selective spatial enhancement unchanged.

8. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 2, characterized in that, The method of using the third cross-level and cross-spatial feature aggregation module to determine the third intermediate feature based on the first intermediate feature, the second intermediate feature, and the first level feature includes: Based on the second intermediate feature, spatial downsampling is performed to obtain the fourth intermediate feature; Based on the fourth intermediate feature and the first intermediate feature, the fifth intermediate feature is obtained by stacking them in the channel dimension; Based on the fifth intermediate feature, spatial downsampling is performed to obtain the sixth intermediate feature; The third intermediate feature is determined using the third cross-level and cross-space feature aggregation module, based on the sixth intermediate feature and the first level feature.

9. The pseudo-CT image target region prediction method based on cross-level and cross-spatial feature aggregation as described in claim 1, characterized in that, The step of determining the predicted target location in the future prediction phase based on the static target mask includes: Using the regression prediction layer in the three-dimensional convolutional neural network-long short-term memory network, a deformation registration field pointing to the future prediction phase is generated; Based on the deformation registration field pointing to the future prediction phase, the static target mask is deformed to obtain the target prediction position under the future prediction phase.

10. A pseudo-CT image target region prediction system based on cross-level and cross-spatial feature aggregation, characterized in that, The pseudo-CT image target region prediction system with cross-level and cross-spatial feature aggregation is used to execute the pseudo-CT image target region prediction method with cross-level and cross-spatial feature aggregation as described in any one of claims 1 to 9.

Images