Radiotherapy image information fusion method based on neural network model

Abstract

The invention relates to the technical field of medical data processing, and discloses a radiotherapy image information fusion method based on a neural network model, and the method comprises the steps: constructing a dual-domain neural network model comprising an image domain and a feature domain, and carrying out the combined training of multi-modal radiotherapy image data through a self-supervised learning method, and deep information fusion from a global structure to local features is realized. A multi-resolution pyramid fusion mechanism is introduced into an image domain to enhance spatial consistency among multi-modal images, and deep feature expression is optimized by combining radiation dose distribution physical constraints and beam path feature constraints in a feature domain. According to the invention, accurate fusion of multi-modal radiotherapy image information is realized on the whole, the accuracy and safety of radiotherapy plan making are improved, the risk of image information distortion of a key area is reduced, and the higher requirement of tumor radiotherapy on the image fusion effect is met.

Description

Technical Field

[0001] The present invention relates to the technical field of medical data processing, and in particular to a method for fusing radiotherapy image information based on a neural network model. Background Art

[0002] During tumor radiotherapy, multimodal medical imaging data (such as CT, MRI, and PET) has become an essential basis for radiotherapy planning. However, existing multimodal image fusion methods generally have limitations. On the one hand, spatial registration errors and contrast differences exist between images of different modalities, resulting in insufficient structural integrity and detail fidelity in the fusion results. On the other hand, existing methods struggle to maintain local stability of image information in radiosensitive areas and high-dose distribution regions, which can easily lead to image quality degradation in critical areas. Furthermore, traditional fusion methods lack adaptability to complex anatomical structures and cannot effectively address the dose distribution changes and anatomical morphological shifts that may occur during radiotherapy. These limitations increase the uncertainty of radiotherapy planning and affect the accuracy and safety of treatment. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a method for fusing radiotherapy image information based on a neural network model to solve technical problems such as the poor accuracy of existing multimodal radiotherapy image data fusion results, which leads to uncertainty in radiotherapy planning and further affects the accuracy and safety of treatment.

[0004] The present invention discloses a method for fusing radiotherapy image information based on a neural network model, the method comprising the following steps: S1. Constructing a first dual-domain neural network model; the dual domains include an image domain and a feature domain; S2. Using a self-supervised learning method to train the first dual-domain neural network model based on historical patient radiotherapy image data to obtain a second dual-domain neural network model; S3. Based on the second dual-domain neural network model, perform a multimodal radiotherapy image fusion operation in the image domain, introduce beam path feature constraints in the fusion process, obtain a first fused radiotherapy image, and input the first fused radiotherapy image into the feature domain; S4. Perform deep semantic feature fusion operations in the feature domain, introduce physical constraints of radiation dose distribution and beam path feature constraints in the fusion process, and generate fused radiotherapy image information.

[0005] Furthermore, the step S1 includes the following sub-steps: S11, constructing an image domain subnetwork; the image domain subnetwork is integrated by a multi-scale convolution unit, a cross-modal feature alignment unit, an upsampling unit, and a feature normalization unit according to a preset hierarchical connection relationship; S12, constructing a feature domain sub-network; the feature domain sub-network is integrated by a multi-layer encoding-decoding network unit and a cross-channel feature fusion unit; S13. Setting an inter-domain residual connection unit and a multi-scale feature transfer unit between the image domain sub-network and the feature domain sub-network to enable dual-domain feature information flow and interaction; S14. Initialize the weight parameters and network hyperparameters of the dual-domain neural network model to obtain a first dual-domain neural network model.

[0006] Furthermore, the step S3 specifically includes: S31. Based on the second dual-domain neural network model, a multi-resolution pyramid fusion network is used in the image domain sub-network to extract global and local features of multimodal radiotherapy images through convolutional layers of different scales. S32, performing a cross-modal spatial structure and grayscale distribution consistency discrimination operation on the extracted features, and performing a preliminary feature fusion operation based on the discrimination results to generate a multi-resolution fusion feature map; S33. Perform upsampling and reconstruction operations on the multi-resolution fusion feature map to generate a preliminary fusion radiotherapy image.

[0007] Furthermore, the step S3 further includes: S34, generating a beam path weight map according to the patient's individual radiotherapy plan data; S35. Based on the preliminary fused radiotherapy image, introduce beam path feature constraints, embed the beam path weight map into the fusion network, optimize the image quality of the beam path coverage area through the weighted attention mechanism, output the first fused radiotherapy image, and input it into the feature domain.

[0008] Furthermore, the step S4 specifically includes: S41. In the feature domain, extract deep semantic features of the first fused radiotherapy image through a multi-layer encoding-decoding network unit, and perform a semantic consistency operation in a feature reconstruction process to generate a first fused feature map; S42. Introducing a physical constraint on radiation dose distribution into the cross-channel feature fusion unit, dynamically adjusting the channel weights of the high-dose region and the low-dose region in the first fused feature map to obtain a second fused feature map; S43. Based on the second fusion feature map, the beam path feature constraint is introduced, the feature expression of the beam path intersection area is optimized through the weighted attention mechanism and the local dose sensitive area correction factor, and the second fusion radiotherapy image information is output.

[0009] Furthermore, the step S42 specifically includes: Generate radiation dose distribution maps based on individual patient radiotherapy plan data; In the cross-channel feature fusion unit, the radiation dose distribution weight map is embedded as a modulation factor in the deep semantic feature fusion process, and the weight distribution of the feature channels in each dose region of the first fused feature map is optimized through back propagation; A residual correction path is constructed based on the residual feedback of the fusion error to correct the local dose characteristic deviation that occurs during the fusion process and obtain the second fusion feature map.

[0010] Furthermore, the step S43 specifically includes: performing a weighted fusion operation based on the second fused feature map and the beam path weight map; The differential attention mechanism is used to detect abnormal gradient responses of features in the intersection area of ​​beam paths, and the abnormal gradient suppression algorithm is used to dynamically fine-tune the weight distribution of feature channels in the intersection area. According to the dose accumulation in the intersection area of ​​the beam paths, the feature expression in the intersection area is corrected and recalibrated based on the local dose sensitive area correction factor, and the second fusion radiotherapy image information is output.

[0011] Furthermore, step S2 includes the following sub-steps: S21, inputting historical patient multimodal radiotherapy image data into the first dual-domain neural network model, and performing feature representation optimization in the image domain subnetwork through a self-supervised contrastive learning method; S22. In the feature domain sub-network, feature reconstruction loss and cross-modal consistency loss are used to jointly train network parameters; S23. Update the weight parameters and hyperparameters of the first dual-domain neural network model through iterative training to obtain a second dual-domain neural network model.

[0012] Furthermore, the self-supervised contrastive learning method specifically includes: Constructing positive sample pairs and negative sample pairs; wherein, features from the same anatomical structure region in radiotherapy images of different modalities are regarded as positive samples, and features from different anatomical structure regions are regarded as negative samples; The image domain sub-network is optimized based on the improved InfoNCE loss function, and the weights of positive and negative samples are adjusted based on the sample confidence distribution.

[0013] Furthermore, the step S2 further includes: S24. During the training process, the loss weight distribution of each sample in the training batch is dynamically adjusted by integrating the patient's historical body position change data, radiation dose distribution, and beam path characteristics; S25. After completing all training rounds, verify the effectiveness of the multimodal information fusion effect and physical consistency constraints of the second dual-domain neural network model through feature visualization.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention constructs a dual-domain neural network model including the image domain and the feature domain, and adopts a self-supervised learning method to jointly train multimodal radiotherapy image data, thereby realizing deep information fusion from global structure to local features. A multi-resolution pyramid fusion mechanism is introduced in the image domain to enhance the spatial consistency between multimodal images, and the physical constraints of radiation dose distribution and beam path feature constraints are combined in the feature domain to optimize the deep feature expression. In addition, a weighting mechanism is introduced in the fusion process to dynamically adjust the feature contribution weights of each modality data, effectively improving the image quality of high-dose sensitive areas and complex anatomical structures. Overall, the precise fusion of multimodal radiotherapy image information is achieved, the accuracy and safety of radiotherapy planning are improved, the risk of image information distortion in key areas is reduced, and the higher requirements of tumor radiotherapy for image fusion effects are met. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of the application, and do not constitute a limitation of the embodiments of the present invention. In the drawings: Figure 1 The figure is a flow chart of a method for fusing radiotherapy image information based on a neural network model disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0017] Example 1 The first aspect of the present invention discloses a method for fusing radiotherapy image information based on a neural network model. Figure 1 , Figure 1 1 is a flow chart of a method for fusing radiotherapy image information based on a neural network model disclosed in an embodiment of the present invention. The method includes the following steps: S1. Constructing a first dual-domain neural network model; the dual domains include an image domain and a feature domain; S2. Using a self-supervised learning method to train the first dual-domain neural network model based on historical patient radiotherapy image data to obtain a second dual-domain neural network model; S3. Based on the second dual-domain neural network model, perform a multimodal radiotherapy image fusion operation in the image domain, introduce beam path feature constraints in the fusion process, obtain a first fused radiotherapy image, and input the first fused radiotherapy image into the feature domain; S4. Perform deep semantic feature fusion operations in the feature domain, introduce physical constraints of radiation dose distribution and beam path feature constraints in the fusion process, and generate fused radiotherapy image information.

[0018] Furthermore, the step S1 includes the following sub-steps: S11, constructing an image domain subnetwork; the image domain subnetwork is integrated by a multi-scale convolution unit, a cross-modal feature alignment unit, an upsampling unit, and a feature normalization unit according to a preset hierarchical connection relationship; S12, constructing a feature domain sub-network; the feature domain sub-network is integrated by a multi-layer encoding-decoding network unit and a cross-channel feature fusion unit; S13. Setting an inter-domain residual connection unit and a multi-scale feature transfer unit between the image domain sub-network and the feature domain sub-network to enable dual-domain feature information flow and interaction; S14. Initialize the weight parameters and network hyperparameters of the dual-domain neural network model to obtain a first dual-domain neural network model.

[0019] Specifically, the image-domain subnetwork in the dual-domain neural network model is primarily designed to perform spatial alignment and preliminary feature fusion of multimodal radiotherapy images. The multiscale convolutional unit simultaneously captures both local details and global structural information from radiotherapy images, enabling features at different resolution levels to work synergistically and avoiding the blurring of edge details or distortion of anatomical structures caused by single-scale processing during the fusion process. The cross-modal feature alignment unit addresses differences in spatial offset and grayscale distribution between radiotherapy images from different modalities, such as CT and MRI. By introducing learnable alignment parameters, it ensures geometric and grayscale consistency within the feature space of the multimodal data, thus preventing structural distortion or signal artifacts in the fused image. The upsampling unit restores the spatial resolution of the lower-resolution modality through feature upsampling, enabling subsequent network layers to optimize feature representation at higher resolutions. The feature normalization unit normalizes the distribution of features from different modalities during fusion, mitigating gradient instability caused by differing dynamic ranges between modalities and thereby improving the convergence speed and generalization capability of the fused network.

[0020] The main purpose of setting up the feature domain subnetwork is to extract and fuse multimodal radiotherapy image data at a deep semantic level. Specifically, the multi-layer encoding-decoding structure can extract high-order semantic features of the image through downsampling and reconstruct the spatial distribution through upsampling, so that the model can take into account both global consistency and local detail expression when extracting radiobiological information (such as tumor heterogeneity). The cross-channel feature fusion unit enhances the expression of the importance of each modality in different clinical areas by dynamically adjusting the weights between feature channels of different modalities. Without cross-channel fusion, the model may not be able to accurately capture key features in high-dose radiation areas or complex anatomical areas.

[0021] In addition, an inter-domain residual connection unit and a multi-scale feature transfer unit are introduced between the image domain sub-network and the feature domain sub-network to support efficient information exchange between the two domains. The inter-domain residual connection unit directly supplements the shallow fusion features of the image domain into the deep fusion process of the feature domain, thereby alleviating the problem of vanishing gradients in the deep network and enhancing the model's sensitivity to boundary structures and fine-grained features. The multi-scale feature transfer unit establishes contextual connections between different resolutions, enabling the coordinated optimization of shallow and deep features, which would otherwise result in feature expression fragmentation or multi-level information loss.

[0022] After completing the dual-domain structure, the weight parameters and network hyperparameters are initialized to ensure that the image domain and feature domain achieve gradient flow balance and feature distribution stability in the early stage of training, providing a solid network structure foundation for the subsequent self-supervised learning stage.

[0023] Through this setup, the image domain and feature domain work together to ensure the geometric consistency of multimodal radiotherapy images at the spatial level while also enhancing the expressive power of deep semantic features. This dual-domain architecture effectively overcomes the problem of traditional fusion models that often lose critical cross-modal information or introduce artifacts when processing complex radiotherapy images, thereby improving the overall accuracy and clinical usability of fused radiotherapy images.

[0024] Furthermore, the step S3 specifically includes: S31. Based on the second dual-domain neural network model, a multi-resolution pyramid fusion network is used in the image domain sub-network to extract global and local features of multimodal radiotherapy images through convolutional layers of different scales. S32, performing a cross-modal spatial structure and grayscale distribution consistency discrimination operation on the extracted features, and performing a preliminary feature fusion operation based on the discrimination results to generate a multi-resolution fusion feature map; S33. Perform upsampling and reconstruction operations on the multi-resolution fusion feature map to generate a preliminary fusion radiotherapy image.

[0025] Furthermore, the step S3 further includes: S34, generating a beam path weight map according to the patient's individual radiotherapy plan data; S35. Based on the preliminary fused radiotherapy image, introduce beam path feature constraints, embed the beam path weight map into the fusion network, optimize the image quality of the beam path coverage area through the weighted attention mechanism, output the first fused radiotherapy image, and input it into the feature domain.

[0026] First of all, it should be noted that in the present invention, although the image domain sub-network has completed the learning of preliminary multimodal fusion capabilities in the training phase, there may still be differences in anatomical structure details or uneven distribution of modal features in the multimodal radiotherapy images input from different patients during the inference phase. To this end, a multi-resolution pyramid fusion network is introduced within the image domain sub-network. By constructing a multi-level feature pyramid, a dynamic trade-off and optimization of low-resolution global structural information and high-resolution detail texture information is achieved. The multi-resolution pyramid fusion network is not an independently trained network, but an integrated structure of the image domain sub-network, which is used to temporarily call its feature fusion strategy during the inference phase. This setting ensures the spatial consistency and grayscale smoothness of the fusion results, and improves the input quality of subsequent deep fusion of the feature domain.

[0027] Specifically, in this embodiment of the present invention, a multi-resolution pyramid fusion network is used to extract multi-scale feature information from radiotherapy images of different modalities during the feature inference phase. The multi-resolution pyramid fusion network is a hierarchical feature extraction structure that extracts image features at different resolution levels, enhancing the ability to express local details while preserving global context.

[0028] The present invention introduces a multi-resolution pyramid fusion network into the image domain subnetwork. The reason is that in traditional convolutional networks, due to the fixed receptive field, it is often difficult to simultaneously capture wide-area structural relationships and subtle texture features, resulting in blurred edge information or anatomical structure distortion during the image fusion stage. The radiotherapy image fusion task not only needs to maintain the spatial position relationship between the tumor and target area and the surrounding tissues on a global scale, but also needs to finely restore the soft tissue boundaries and small blood vessel structures on a local scale, so that high-risk areas can be accurately assessed during subsequent dose planning. If this structure is lacking, the fusion network will lose local feature details after modal alignment, thereby affecting the fidelity of the target area boundary.

[0029] In the multi-resolution pyramid fusion network, the input multimodal radiotherapy image first passes through a series of convolutional units at different scales to extract cross-modal global and local features. This information is then transferred between the scale levels to achieve complementary enhancement of features at different resolutions. This network structure effectively fuses the high-density tissue information of CT images with the soft tissue features of MRI images, resulting in a fusion result that retains both high-contrast bony structures and delicate soft tissue textures.

[0030] Furthermore, in embodiments of the present invention, multimodal radiotherapy images originate from different imaging devices and exhibit spatial offset and grayscale differences. To prevent artifacts or feature offset from being mistakenly used as fusion features during inference, a cross-modal spatial structure and grayscale distribution consistency check is performed after feature extraction to determine the consistency of the spatial structure and pixel grayscale distribution of the feature maps of different modalities. The core principle of this check is to compare the structural similarity indicators (such as the structural similarity index (SSIM)) and grayscale distribution differences (such as histogram mean differences) in the overlapping regions of the modal images. When the consistency reaches a preset threshold, the features of that region are allowed into the fusion calculation. Otherwise, the region is identified as an abnormal region and its features are weighted down. This prevents inter-modal grayscale conflicts from introducing artifacts or information loss in the fusion result, ensuring that the fused image maintains inter-modal consistency at both the structural and grayscale levels.

[0031] Subsequently, preliminary feature fusion is performed based on the consistency judgment results to generate a multi-resolution fused feature map. This fusion process calculates modality-weighted features at each scale through a convolutional fusion layer, forming a cross-modal complementary fused feature representation. The fused features are then restored to their original resolution using an upsampling unit, and feature normalization units adjust the feature strength to ensure stability during reconstruction, preventing grayscale drift and edge jaggedness during the fusion process. This ensures that the output preliminary fused radiotherapy image maintains structural smoothness and grayscale balance.

[0032] After generating the preliminary fused radiotherapy image, the present invention further introduces beam path feature constraints to enhance the image expression quality of the radiotherapy key area.

[0033] Among them, in an embodiment of the present invention, the radiotherapy plan data includes but is not limited to the patient's individual tumor target area contour, radiation beam path information and three-dimensional dose distribution map. The beam path refers to the spatial path from the transmitting end of the radiotherapy equipment to the target area during radiotherapy. The tissue along the path is usually exposed to a higher dose of radiation, and the image quality directly affects the dose assessment accuracy and target area positioning accuracy. By extracting the beam path coordinates and dose distribution map from the radiotherapy plan data, and using the voxel weighting algorithm to perform weighted integration on the voxels along the path, a beam path weight map is constructed. The weight map reflects the dose sensitivity of different voxels in the area covered by the beam path, where high-dose areas are assigned higher weight values, and low-dose or non-critical areas are assigned lower weight values, thereby establishing a regional priority map.

[0034] Within the fusion network, the beam path weight map is weighted pixel-by-pixel with the preliminary fused radiotherapy image through a feature weighting layer, providing dose-sensitive guidance in the spatial dimension for the fused features. In the weighted attention mechanism, the network first performs channel attention and spatial attention decomposition operations on the input feature map. The beam path weight map is then used to modulate the channel attention distribution, significantly enhancing the feature channel response in high-dose regions while moderately suppressing the response in low-dose regions. Subsequently, the spatial attention layer dynamically allocates pixel-level attention weights between the beam path coverage area and the surrounding area, achieving enhanced expression of local structural details.

[0035] Through the above operations, the present invention can effectively fuse multimodal radiotherapy image information, improve the consistency of spatial structure and grayscale distribution between images of different modalities, and focus on enhancing the image quality of the beam path coverage area. It also ensures the balance of global and local feature expression of the fused image, provides high-quality input for the subsequent deep feature fusion of the feature domain, significantly improves the overall accuracy and diagnostic value of radiotherapy images, and provides a reliable image basis for the formulation of precise radiotherapy plans.

[0036] Furthermore, the step S4 specifically includes: S41. In the feature domain, extract deep semantic features of the first fused radiotherapy image through a multi-layer encoding-decoding network unit, and perform a semantic consistency operation in a feature reconstruction process to generate a first fused feature map; S42. Introducing a physical constraint on radiation dose distribution into the cross-channel feature fusion unit, dynamically adjusting the channel weights of the high-dose region and the low-dose region in the first fused feature map to obtain a second fused feature map; S43. Based on the second fusion feature map, the beam path feature constraint is introduced, the feature expression of the beam path intersection area is optimized through the weighted attention mechanism and the local dose sensitive area correction factor, and the second fusion radiotherapy image information is output.

[0037] Furthermore, the step S42 specifically includes: Generate radiation dose distribution maps based on individual patient radiotherapy plan data; In the cross-channel feature fusion unit, the radiation dose distribution weight map is embedded as a modulation factor in the deep semantic feature fusion process, and the weight distribution of the feature channels in each dose region of the first fused feature map is optimized through back propagation; A residual correction path is constructed based on the residual feedback of the fusion error to correct the local dose characteristic deviation that occurs during the fusion process and obtain the second fusion feature map.

[0038] Furthermore, the step S43 specifically includes: performing a weighted fusion operation based on the second fused feature map and the beam path weight map; The differential attention mechanism is used to detect abnormal gradient responses of features in the intersection area of ​​beam paths, and the abnormal gradient suppression algorithm is used to dynamically fine-tune the weight distribution of feature channels in the intersection area. According to the dose accumulation in the intersection area of ​​the beam paths, the feature expression in the intersection area is corrected and recalibrated based on the local dose sensitive area correction factor, and the second fusion radiotherapy image information is output.

[0039] Specifically, in the deep feature fusion stage of the feature domain, the first fused radiotherapy image is processed by a multi-layer encoding-decoding network unit to extract deep semantic features. The encoding-decoding network adopts a variant of the U-Net structure. The encoder part gradually reduces the dimension of the input feature map through layer-by-layer convolution and pooling operations to extract abstract semantic information; the decoder part gradually restores the resolution through the deconvolution layer and performs jump connections to ensure that low-level spatial information is transmitted during the decoding process. This structure can effectively enhance the model's ability to express cross-modal differences, avoid the problem of traditional convolutional structures losing local details during deep abstraction, and maintain semantic richness while enabling the reconstructed features to effectively adapt to the detailed differences of multimodal radiotherapy images. The first fused feature map generated by this process not only contains global semantic relationships, but also retains the spatial information of the key structures of the radiotherapy image.

[0040] After generating the first fused feature map, the cross-channel feature fusion phase begins, focusing on optimizing the physical constraints of the radiation dose distribution. The patient's individual radiotherapy plan data includes dose matrices based on CT and MRI images, as well as the output of the radiotherapy planning end. Specifically, the raw dose data is first normalized to ensure that the dose value distribution is consistent with the scale of the feature domain neural network input. A spatial resampling algorithm is then used to align the dose matrix with the fused feature map, constructing a radiation dose distribution weight map to ensure that the dose constraints correctly cover the corresponding anatomical regions. This map reflects the differences in radiation dose received by different regions.

[0041] In the cross-channel feature fusion unit, the generated radiation dose distribution weight map is embedded as a modulation factor in the deep semantic feature fusion process. Specifically, the modulation factor is dynamically adjusted on the activation value of each convolutional channel. The channel response is modified through multiplicative attention weights, enhancing the features of high-dose regions and moderately suppressing low-dose regions. Backpropagation is then used to optimize the channel weights to ensure dynamic adaptation to different dose regions.

[0042] In order to further alleviate the dose characteristic drift problem that may be introduced during the training process, the present invention constructs a residual correction path based on the fusion error residual feedback, and feeds the local error residual back to the input end of the fusion unit through a bypass connection to achieve real-time correction of the local dose characteristic deviation.

[0043] During the cross-channel feature fusion process, the residual correction path is used to correct the local dose feature deviation accumulated during the fusion process. The core idea is to introduce a bypass to perform a difference operation between the intermediate features output during the fusion process and the original input features to form a residual error map. The residual error map reflects the feature drift or overfitting problems that may occur in the dose-sensitive area after the current deep fusion. The residual error map is processed by a lightweight convolution unit and then re-injected into the early layer of the cross-channel feature fusion unit to form a residual correction path. This feedback mechanism enables the model to perceive the feature expression errors of local areas in real time, and through multiple iterative adjustments, it efficiently corrects the feature offset caused by fusion in high-dose areas, avoiding the feature collapse phenomenon that occurs in traditional deep fusion architectures when fusing multiple dose areas, and ensuring that the feature information at different dose levels can maintain structural consistency.

[0044] Based on the optimized second fusion feature map, beam path feature constraints are introduced to further enhance the expressiveness of the fusion results in the key areas of radiotherapy.

[0045] In the later stage of deep fusion of feature domains, the differential attention mechanism is introduced to further enhance the feature expression capability of complex areas. The core of the differential attention mechanism is not only to pay attention to the absolute size of the eigenvalues, but also to calculate the gradient change rate in the feature map in real time to detect potential abnormal gradient responses in the intersection area of ​​the beam paths. These abnormal responses are characterized by drastic changes in the eigenvalues ​​in space, which may cause blurred details or dose assessment deviations in the critical intersection area of ​​the radiotherapy image. The second fused feature map is scanned by the differential convolution kernel to generate a local gradient response map, and after detecting the high gradient abnormal area, the abnormal gradient suppression algorithm is started. Specifically, the mean and variance of the feature gradients in the abnormal area are first calculated, and then the weights of each channel in the intersection area are dynamically adjusted through the gated attention unit.

[0046] Furthermore, considering the common phenomenon of dose accumulation in beam path intersections, the present invention further precisely corrects and recalibrates the feature representation within these intersections using a local dose-sensitive region correction factor. This correction differs from the residual correction path in step S42 in that it focuses on addressing the feature accumulation caused by multi-beam intersection. By redistributing feature weights within the intersection region, the fused image maintains smooth and realistic feature representation even in highly complex areas.

[0047] This series of operations achieves a leap forward in the fusion image from global consistency to precise expression of local key areas by performing deep semantic feature optimization, radiation dose constraint adjustment, and beam path feature enhancement in the feature domain. It effectively solves the problem of insufficient image quality in high-dose areas and multi-beam intersection areas with traditional methods, and provides solid data support for the formulation of clinical radiotherapy plans.

[0048] Furthermore, step S2 includes the following sub-steps: S21, inputting historical patient multimodal radiotherapy image data into the first dual-domain neural network model, and performing feature representation optimization in the image domain subnetwork through a self-supervised contrastive learning method; S22. In the feature domain sub-network, feature reconstruction loss and cross-modal consistency loss are used to jointly train network parameters; S23. Update the weight parameters and hyperparameters of the first dual-domain neural network model through iterative training to obtain a second dual-domain neural network model.

[0049] Furthermore, the self-supervised contrastive learning method specifically includes: Constructing positive sample pairs and negative sample pairs; wherein, features from the same anatomical structure region in radiotherapy images of different modalities are regarded as positive samples, and features from different anatomical structure regions are regarded as negative samples; The image domain sub-network is optimized based on the improved InfoNCE loss function, and the weights of positive and negative samples are adjusted based on the sample confidence distribution.

[0050] Furthermore, the step S2 further includes: S24. During the training process, the loss weight distribution of each sample in the training batch is dynamically adjusted by integrating the patient's historical body position change data, radiation dose distribution, and beam path characteristics; S25. After completing all training rounds, verify the effectiveness of the multimodal information fusion effect and physical consistency constraints of the second dual-domain neural network model through feature visualization.

[0051] Specifically, during the training phase of the dual-domain neural network model, feature representation optimization is performed using a self-supervised contrastive learning method. This method constructs pairs of positive and negative samples to guide the model in learning modality-independent, structure-sensitive latent feature representations. Self-supervised contrastive learning was introduced to overcome the label scarcity and modality differences in radiotherapy images, enabling the network to effectively capture key structural features under unsupervised conditions and avoiding feature drift or insufficient representation of key anatomical structures in multimodal information fusion, which can lead to poor fusion results.

[0052] In the design of the loss function, the present invention adopts an improved InfoNCE loss function, which not only considers the similarity between positive and negative samples, but also introduces spatial neighborhood consistency weights and sample confidence adjustment factors to dynamically adjust the training contribution of each sample pair. The reason for setting the spatial neighborhood consistency weight is to ensure the coherence of the feature mapping of adjacent anatomical areas and avoid feature jumps caused by local anatomical differences. The purpose of introducing the confidence adjustment factor is to reduce the interference of low-quality samples on model training, thereby improving the robustness of training.

[0053] In the feature domain subnetwork, feature reconstruction loss and cross-modal consistency loss are used to jointly optimize network parameters. The purpose is to guide the deep semantic features to maintain information integrity during the reconstruction process, and to enable different modalities to be mapped to similar distributions in the deep feature space, thereby avoiding inter-modal feature alignment deviation in the deep fusion stage of the model.

[0054] During the training process, the loss weight distribution is dynamically adjusted by fusing the patient's historical body position change data, radiation dose distribution, and beam path characteristics, so that the model focuses on high-dose sensitive areas and key anatomical structures with strong body position dependence during training, thereby improving clinical applicability and the physical consistency of the fusion results.

[0055] As a preferred embodiment, the improved InfoNCE loss function is specifically:

[0056] Specifically, in the self-supervised training process of the image domain sub-network, the neighborhood weight based on anatomical structure is introduced. and dynamic confidence adjustment factor , construct an improved contrast loss function. This loss function not only considers the similarity of sample pairs in the feature space, but also integrates the proximity information of the anatomical space and the sample prediction confidence, which helps to enhance the feature expression ability of key anatomical regions. For all positive sample pairs index set, each pair of samples They come from different modalities but correspond to the same anatomical region; is the embedded feature vector of sample i, j, k in the feature space; Represents the normalization operation of all candidate samples k of sample i (including positive samples and negative samples, k=j for positive samples and k≠j for negative samples); is the eigenvector The similarity measurement function uses normalized cosine similarity; is the temperature parameter used to adjust the smoothness of the similarity distribution, The smaller it is, the sharper the similarity distribution is.

[0057] in,

[0058]

[0059] in, is the Euclidean distance between samples i and j in the anatomical space; is the neighborhood attenuation coefficient, which is used to control the effect of distance on weight. The confidence score of the sample pair is obtained by the network prediction. The above setting can make the contrast weight of adjacent anatomical structures high, the contrast weight of distant structures low, and the high confidence sample pairs contribute more to the loss optimization.

[0060] Finally, it should be noted that the above-mentioned embodiments include multiple parallel implementation methods of the present invention, and deleting or otherwise adjusting one or more of the implementation methods will not affect the implementation of the scheme. In addition, the method for fusing radiotherapy image information based on a neural network model disclosed in the embodiments of the present invention is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, not to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for fusion of radiotherapy image information based on a neural network model, characterized in that: The method comprises the following steps: S1. Constructing a first dual-domain neural network model; the dual domains include an image domain and a feature domain; S2. Using a self-supervised learning method to train the first dual-domain neural network model based on historical patient radiotherapy image data to obtain a second dual-domain neural network model; S3. Based on the second dual-domain neural network model, perform a multimodal radiotherapy image fusion operation in the image domain, introduce beam path feature constraints in the fusion process, obtain a first fused radiotherapy image, and input the first fused radiotherapy image into the feature domain; S4. Perform deep semantic feature fusion operations in the feature domain, introduce physical constraints of radiation dose distribution and beam path feature constraints in the fusion process, and generate fused radiotherapy image information.

2. The radiotherapy image information fusion method based on the neural network model according to claim 1, characterized in that: The step S1 includes the following sub-steps: S11, constructing an image domain subnetwork; the image domain subnetwork is integrated by a multi-scale convolution unit, a cross-modal feature alignment unit, an upsampling unit, and a feature normalization unit according to a preset hierarchical connection relationship; S12, constructing feature domain sub-network; The feature domain sub-network is integrated by a multi-layer encoding-decoding network unit and a cross-channel feature fusion unit; S13. Setting an inter-domain residual connection unit and a multi-scale feature transfer unit between the image domain sub-network and the feature domain sub-network to enable dual-domain feature information flow and interaction; S14. Initialize the weight parameters and network hyperparameters of the dual-domain neural network model to obtain a first dual-domain neural network model.

3. The radiotherapy image information fusion method based on a neural network model according to claim 1, characterized in that: The step S3 specifically includes: S31. Based on the second dual-domain neural network model, a multi-resolution pyramid fusion network is used in the image domain sub-network to extract global and local features of multimodal radiotherapy images through convolutional layers of different scales. S32, performing a cross-modal spatial structure and grayscale distribution consistency discrimination operation on the extracted features, and performing a preliminary feature fusion operation based on the discrimination results to generate a multi-resolution fusion feature map; S33. Perform upsampling and reconstruction operations on the multi-resolution fusion feature map to generate a preliminary fusion radiotherapy image.

4. The method for fusion of radiotherapy image information based on a neural network model according to claim 3, characterized in that: The step S3 further comprises: S34, generating a beam path weight map according to the patient's individual radiotherapy plan data; S35. Based on the preliminary fused radiotherapy image, introduce beam path feature constraints, embed the beam path weight map into the fusion network, optimize the image quality of the beam path coverage area through the weighted attention mechanism, output the first fused radiotherapy image, and input it into the feature domain.

5. The radiotherapy image information fusion method based on a neural network model according to claim 4, characterized in that: The step S4 specifically includes: S41. In the feature domain, extract deep semantic features of the first fused radiotherapy image through a multi-layer encoding-decoding network unit, and perform a semantic consistency operation in a feature reconstruction process to generate a first fused feature map; S42. Introducing a physical constraint on radiation dose distribution into the cross-channel feature fusion unit, dynamically adjusting the channel weights of the high-dose region and the low-dose region in the first fused feature map to obtain a second fused feature map; S43. Based on the second fusion feature map, the beam path feature constraint is introduced, the feature expression of the beam path intersection area is optimized through the weighted attention mechanism and the local dose sensitive area correction factor, and the second fusion radiotherapy image information is output.

6. The radiotherapy image information fusion method based on a neural network model according to claim 5, characterized in that: The step S42 specifically includes: Generate radiation dose distribution maps based on individual patient radiotherapy plan data; In the cross-channel feature fusion unit, the radiation dose distribution weight map is embedded as a modulation factor in the deep semantic feature fusion process, and the weight distribution of the feature channels in each dose region of the first fused feature map is optimized through back propagation; A residual correction path is constructed based on the residual feedback of the fusion error to correct the local dose characteristic deviation that occurs during the fusion process and obtain the second fusion feature map.

7. The radiotherapy image information fusion method based on a neural network model according to claim 6, characterized in that: The step S43 specifically includes: performing a weighted fusion operation based on the second fused feature map and the beam path weight map; The differential attention mechanism is used to detect abnormal gradient responses of features in the intersection area of ​​beam paths, and the abnormal gradient suppression algorithm is used to dynamically fine-tune the weight distribution of feature channels in the intersection area. According to the dose accumulation in the intersection area of ​​the beam paths, the feature expression in the intersection area is corrected and recalibrated based on the local dose sensitive area correction factor, and the second fusion radiotherapy image information is output.

8. The method for fusing radiotherapy image information based on a neural network model according to any one of claims 1 to 7, characterized in that: The step S2 includes the following sub-steps: S21, inputting historical patient multimodal radiotherapy image data into the first dual-domain neural network model, and performing feature representation optimization in the image domain subnetwork through a self-supervised contrastive learning method; S22. In the feature domain sub-network, feature reconstruction loss and cross-modal consistency loss are used to jointly train network parameters; S23. Update the weight parameters and hyperparameters of the first dual-domain neural network model through iterative training to obtain a second dual-domain neural network model.

9. The radiotherapy image information fusion method based on a neural network model according to claim 8, characterized in that: The self-supervised contrastive learning method specifically includes: Constructing positive sample pairs and negative sample pairs; wherein, features from the same anatomical structure region in radiotherapy images of different modalities are regarded as positive samples, and features from different anatomical structure regions are regarded as negative samples; The image domain sub-network is optimized based on the improved InfoNCE loss function, and the weights of positive and negative samples are adjusted based on the sample confidence distribution.

10. The radiotherapy image information fusion method based on a neural network model according to claim 9, characterized in that: The step S2 further includes: S24. During the training process, the loss weight distribution of each sample in the training batch is dynamically adjusted by integrating the patient's historical body position change data, radiation dose distribution, and beam path characteristics; S25. After completing all training rounds, verify the effectiveness of the multimodal information fusion effect and physical consistency constraints of the second dual-domain neural network model through feature visualization.

Images