Deep reinforcement learning model training method, computed tomography image reconstruction method, equipment, medium and product

Abstract

The invention discloses a training method of a deep reinforcement learning model, a computed tomography image reconstruction method, equipment, a medium and a product, and relates to the technical field of medical imaging, and the training method comprises the steps: extracting the features of a current state image through an upper path and a lower path which are processed in parallel by using a structure-detail collaborative feature extraction module, fusing the outputs of the two paths to obtain a fused feature map; outputting the action probability distribution of each predefined image processing operation selected by each pixel according to the input fusion feature map by utilizing a strategy network; and based on the motion probability distribution, determining a predefined image processing operation applied to each pixel by the current time step, and then applying the predefined image processing operation to the current state image to obtain a reconstructed image. According to the method, through adaptive sequential decision and multi-target composite reward optimization based on deep reinforcement learning, dynamic balance noise suppression and anatomical detail retention, the quality, adaptive ability and calculation efficiency of low-dose CT image reconstruction are significantly improved.

Description

A training method for a deep reinforcement learning model, a method for reconstructing computed tomography images, equipment, media, and products. Technical Field

[0001] This application relates to the field of medical imaging technology, and in particular to a training method for a deep reinforcement learning model, a method for reconstructing computed tomography images, an apparatus, a medium, and a product. Background Technology

[0002] Computed tomography (CT) is a widely used imaging technique in clinical diagnosis and treatment. In clinical practice, to avoid prolonged exposure of patients to high doses of X-rays, the X-ray dose is usually minimized. However, this often reduces image quality and leads to a decrease in the image signal-to-noise ratio. Therefore, how to recover high-quality, high-resolution images under low-dose radiation conditions has become a pressing problem to be solved in the field of CT image reconstruction.

[0003] Methods for high-resolution reconstruction of low-dose CT (LDCT) images can be categorized into two main types: traditional methods and deep learning methods. Traditional methods primarily include projection domain processing and iterative reconstruction. Projection domain methods are prone to losing high-frequency information, while iterative reconstruction is computationally complex, difficult to tune parameters, and relies on manual prior knowledge. Furthermore, it struggles to handle large-scale data and dynamic scenes with varying imaging conditions. Deep learning methods offer significant advantages in accuracy and efficiency, but each has its limitations: RED-CNN has insufficient generalization to strong noise; secondary autoencoders have weak edge preservation capabilities; CTformer has high computational costs; SWELNet relies on simulated data distribution matching, making training complex; MMCA's edge enhancement operator has limited adaptability; and generative adversarial networks reduce reliance on paired data through adversarial training, but most models still rely on static feedforward processing, lacking the ability to dynamically adjust based on local image features, and the balance between noise and detail depends on preset parameters, resulting in insufficient adaptability. Summary of the Invention

[0004] The purpose of this application is to provide a training method for a deep reinforcement learning model, a method for reconstructing computed tomography (CT) images, an apparatus, a medium, and a product. By using adaptive sequential decision-making and multi-objective composite reward optimization based on deep reinforcement learning, and dynamically balancing noise suppression and anatomical detail preservation, the quality, adaptability, and computational efficiency of low-dose CT image reconstruction are significantly improved.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a training method for a deep reinforcement learning model, comprising: acquiring training samples; the training samples include: computed tomography (CT) images and corresponding high-resolution reconstructed images; using the CT images as initial current state images, inputting them into a structure-detail co-extraction feature extraction module; utilizing the structure-detail co-extraction feature extraction module, extracting features of the current state image through parallel processing of upper and lower paths, and fusing the outputs of the two paths to obtain a fused feature map; wherein, the upper path performs the following operations: performing a basic convolution on the current state image, followed by processing a convolution with dilated convolution... The system employs a parallel structure of branches for identity mapping, detail enhancement, and noise suppression to extract multi-scale features. The outputs of each branch are fused, and channel attention and spatial attention are applied sequentially to obtain enhanced semantic features. The next path performs the following operations: edge detection is performed using the Sobel operator, followed by convolutional layer deepening to obtain enhanced edge features. The fusion operation specifically involves concatenating the enhanced semantic features with the enhanced edge features, generating a fused feature map through an adaptive gating mechanism. The fused feature map is then input into a policy network, which, based on the input fused feature map, outputs each predefined image processing operation for each pixel. The action probability distribution; the policy network includes a gated recurrent unit, which is used to model the spatial dependencies between pixels; based on the action probability distribution, a predefined image processing operation is determined for each pixel at the current time step; the determined predefined image processing operation for each pixel is applied to the current state image to obtain the reconstructed image; a composite reward is calculated based on the reconstructed image, the current state image, and the high-resolution reconstructed image; the composite reward is a weighted sum of pixel-level fidelity reward, gradient structure similarity reward, and local variance penalty term; the current state image is input into the network for evaluation to obtain the state value of each pixel, and combined with... The composite reward and the state value judgment satisfy the iteration termination condition; if the termination condition is not satisfied, the reconstructed image is used as the updated current state image, and the process returns to the step of "using the structure-detail co-feature extraction module to extract features of the current state image through parallel processing of the upper and lower paths, and fusing the outputs of the two paths to obtain a fused feature map"; if the termination condition is satisfied, the trained deep reinforcement learning model is obtained; the deep reinforcement learning model includes: a structure-detail co-feature extraction module, a policy network, and a value network; the trained deep reinforcement learning model is used for high-resolution reconstruction of low-dose computed tomography images.

[0006] Optionally, the step of using the computed tomography image as the initial current state image and inputting it into the structure-detail co-extraction module specifically includes: performing an Anscombe transform on the computed tomography image to convert the Poisson noise in it into Gaussian noise to obtain a preprocessed image; and using the preprocessed image as the initial current state image and inputting it into the structure-detail co-extraction module.

[0007] Optionally, the detail enhancement branch includes a small-scale branch and a medium-scale branch; the small-scale branch captures fine structures through cascaded 1×3 and 3×1 convolutional layers; the medium-scale branch processes medium anatomical structures through two cascaded 3×3 convolutional layers.

[0008] Optionally, based on the action probability distribution, determining the predefined image processing operation applied to each pixel at the current time step specifically includes: performing random sampling according to the action probability distribution to determine the predefined image processing operation applied to each pixel at the current time step.

[0009] Optionally, the predefined image processing operations include: no action, Gaussian filtering with different variances, bilateral filtering with different parameters, median filtering, nonlocal mean filtering, guided filtering, adaptive sharpening filtering, convolution enhancement, and residual enhancement.

[0010] Optionally, the value network is a fully convolutional network; the value network takes the current state image as input, processes it through at least one convolutional layer, and outputs a single-channel feature map, where the value at each position in the single-channel feature map represents the state value of the corresponding pixel; the value network specifically obtains the state value of each pixel through the following operations: performing dimensionality reduction and encoding on the input features through a 3×3 convolutional layer with a stride of 2; and performing feature integration through a 3×3 convolutional layer with a stride of 1 to output a single-channel feature map.

[0011] Secondly, this application provides a method for reconstructing computed tomography (CT) images, comprising: acquiring a low-dose CT image; inputting the low-dose CT image into a trained deep reinforcement learning model to obtain a high-resolution reconstructed image; wherein the trained deep reinforcement learning model is trained by the training method of the deep reinforcement learning model described in any of the preceding claims.

[0012] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method for the deep reinforcement learning model or the steps of the computed tomography image reconstruction method described in any one of the above.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the training method for the deep reinforcement learning model or the steps of the computed tomography image reconstruction method described in any one of the above.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the training method for the deep reinforcement learning model or the steps of the computed tomography image reconstruction method described in any one of the above.

[0015] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a training method for a deep reinforcement learning model, a method for reconstructing computed tomography (CT) images, an apparatus, a medium, and a product. The training method includes: acquiring training samples; the training samples include: CT images and corresponding high-resolution reconstructed images; using the CT images as initial current state images, inputting them into a structure-detail co-extraction module; using the structure-detail co-extraction module, extracting features from the current state images through parallel processing of upper and lower paths, and fusing the outputs of the two paths to obtain a fused feature map; wherein, the upper path performs the following operations: processing the... After performing basic convolution on the current state image, multi-scale features are extracted through a parallel structure including dilated convolution branches, identity mapping branches, detail enhancement branches, and noise suppression branches. The outputs of each branch are fused, and channel attention and spatial attention are applied sequentially to obtain enhanced semantic features. The next path performs the following operations: edge detection is performed using the Sobel operator, followed by convolutional layer deepening to obtain enhanced edge features. The fusion operation specifically involves concatenating the enhanced semantic features with the enhanced edge features, generating a fused feature map through an adaptive gating mechanism. The fused feature map is input into a policy network, which outputs each... Each pixel selects the action probability distribution for each predefined image processing operation; the policy network includes a gated recurrent unit, which is used to model the spatial dependencies between pixels; based on the action probability distribution, the predefined image processing operation applied to each pixel at the current time step is determined; the determined predefined image processing operation for each pixel is applied to the current state image to obtain the reconstructed image; a composite reward is calculated based on the reconstructed image, the current state image, and the high-resolution reconstructed image; the composite reward is a weighted sum of pixel-level fidelity reward, gradient structure similarity reward, and local variance penalty term; the current state image is input into the network for evaluation to obtain the result for each pixel. The state value is used, and the composite reward and the state value are combined to determine whether the iteration termination condition is met. If the termination condition is not met, the reconstructed image is used as the updated current state image, and the process returns to the step of "using the structure-detail co-feature extraction module to extract features of the current state image through parallel processing of the upper and lower paths, and fusing the outputs of the two paths to obtain a fused feature map". If the termination condition is met, the trained deep reinforcement learning model is obtained. The deep reinforcement learning model includes: a structure-detail co-feature extraction module, a policy network, and a value network. The trained deep reinforcement learning model is used for high-resolution reconstruction of low-dose computed tomography images.This application utilizes a sequential decision-making mechanism of reinforcement learning agents to dynamically select and combine various denoising and enhancement operations. Through a composite reward function, it simultaneously optimizes pixel fidelity, gradient structure similarity, and local statistical properties, guiding the network to precisely enhance key anatomical structures and tissue edges, such as blood vessels and small lesions, while suppressing noise. This overcomes the problems of insufficient generalization in RED-CNN and weak edge preservation in secondary autoencoders. The policy network in this application can evaluate the local state of each pixel in real time based on fused features and output the optimal probability distribution of image processing actions. This closed-loop mechanism of "state awareness-decision execution" allows the processing strategy to dynamically and adaptively adjust according to the noise level, texture complexity, and structural characteristics of different regions of the image, without relying on fixed preset parameters. This application uses a parallel multi-branch structure to extract and fuse multi-scale features in one go, avoiding redundant computation. Through a deep reinforcement learning framework, it innovatively combines adaptive decision-making, multi-objective optimization, and efficient parallel training, comprehensively solving multiple challenges in high-resolution reconstruction of low-dose CT images, including noise suppression, detail preservation, adaptability, and computational efficiency. Ultimately, it achieves image quality and clinical applicability significantly superior to existing technologies. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is an application environment diagram of a deep reinforcement learning model training method or a computed tomography image reconstruction method according to an embodiment of this application.

[0018] Figure 2 is a flowchart illustrating a training method for a deep reinforcement learning model according to an embodiment of this application.

[0019] Figure 3 is a schematic diagram of the structure of the HRDRL-Net model provided in an embodiment of this application.

[0020] Figure 4 is a schematic diagram of the structure-detail collaborative feature extraction module provided in an embodiment of this application.

[0021] Figure 5 is a schematic diagram of the execution flow of the intelligent agent convolution module provided in an embodiment of this application.

[0022] Figure 6 is a schematic diagram of the training process of HRDRL-Net provided in an embodiment of this application.

[0023] Figure 7 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] This application first converts Poisson noise in LDCT images into Gaussian noise using Anscombe transform to improve the numerical stability of subsequent processing. Then, a High Resolution Deep Reinforcement Learning Network (HRDRL-Net) model is constructed, employing shared convolutional modules for multi-level feature extraction. For complex tissue structures in the images, a hybrid architecture combining multi-scale dual attention mechanisms and the Sobel operator is introduced. The former enhances the representation of key anatomical regions through spatial-channel dual attention weighting, while the latter strengthens the geometric features of tissue edges through gradient operations. In the reinforcement learning decision-making stage, a composite reward function is designed to comprehensively evaluate pixel-level fidelity, gradient structure similarity, and local variance constraints, providing multi-scale feedback signals for the dynamic optimization of the policy network. Finally, a real-time state evaluation module compares the current reconstruction result with the target image to intelligently determine the iteration termination condition. Ultimately, a high-quality reconstructed image is output through inverse Anscombe transform, achieving an optimal balance between noise suppression and detail preservation.

[0026] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] The deep reinforcement learning model training method and computed tomography image reconstruction method provided in this application embodiment can be applied to the application environment shown in Figure 1. The terminal 102 communicates with the server 104 via a network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated into the server 104, or placed in the cloud or on other servers.

[0028] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0029] In an exemplary embodiment, as shown in FIG2, a training method for a deep reinforcement learning model is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is described using the server 104 in FIG1 as an example, and includes the following steps 201 to 208. Wherein: S1, obtaining training samples; the training samples include: computed tomography images and corresponding high-resolution reconstructed images.

[0030] S2. The computed tomography image is used as the initial current state image and input into the structure-detail co-feature extraction module.

[0031] The computed tomography images in this embodiment are LDCT images. Since LDCT images mainly contain Poisson noise and their structural information is more complex than that of natural images, the original images are first subjected to Anscombe transform to convert the Poisson noise in the images into Gaussian noise.

[0032] S3. Using the structure-details collaborative feature extraction module, features of the current state image are extracted through parallel processing of the upper and lower paths, and the outputs of the two paths are fused to obtain a fused feature map. The upper path performs the following operations: after performing basic convolution on the current state image, multi-scale features are extracted through a parallel structure including dilated convolution branches, identity mapping branches, detail enhancement branches, and noise suppression branches. The outputs of each branch are fused, and channel attention and spatial attention are applied sequentially to obtain enhanced semantic features. The lower path performs the following operations: edge detection is performed using the Sobel operator, followed by convolutional layer deepening processing to obtain enhanced edge features. The fusion operation specifically involves concatenating the enhanced semantic features with the enhanced edge features and generating a fused feature map through an adaptive gating mechanism.

[0033] The deep reinforcement learning model used in this embodiment is the HRDRL-Net model, and its overall structure is shown in Figure 3. The agent consists of three parts: a Structure-Detail Collaborative Feature Extractor (SDCFE), a policy network, and a value network. The network input is the LDCT image after noise transformation, and the output is the reconstructed high-resolution image. The agent continuously interacts with the environment, learns decision-making strategies to improve the quality of the current image based on the current image state and the composite reward signal, and dynamically selects the action sequence that maximizes the improvement in image quality.

[0034] The SDCFE module is based on a multi-scale feature fusion design. By combining multi-scale convolution, edge enhancement, and dual attention, it can effectively suppress noise while maintaining and enhancing key structures and details such as blood vessels and micro lesions.

[0035] As shown in Figure 4, the input image undergoes feature extraction through dual-path parallel processing. The upper path captures multi-scale semantic information of the image, while the lower path extracts and enhances the structural boundary information. This decoupled design ensures that edge features are not lost in the deep network, providing important structural guidance for subsequent detail enhancement.

[0036] In the above path, the input image first passes through a 3×3 convolutional layer for basic feature extraction. Then, the feature map is input into the Low-Dose Noise Mitigation (LDNM) module, where multi-scale feature processing is performed through a four-branch parallel architecture: Dilated convolution branch: adopts a parallelized asymmetric dilation rate design (dilation=1,2,5), and integrates multi-scale receptive field information through feature concatenation to effectively expand the receptive field range.

[0037] Identity mapping branch: Preserves original feature information through 1×1 convolution, prevents gradient vanishing, and maintains information integrity.

[0038] Detail Enhancement Branch: Employing a multi-scale design, the small-scale branch captures fine structures such as blood vessels and small lesions through cascaded convolutions of 1×3 and 3×1, while the medium-scale branch processes medium-sized anatomical structures through two cascaded convolutions of 3×3. Features from both branches are concatenated along the channel dimension and then integrated through a fusion convolutional layer to precisely enhance important detail regions.

[0039] Noise suppression branch: Deep features are constructed through two consecutive 3×3 convolutional layers to suppress noise patterns in LDCT images.

[0040] The outputs of the four branches are fused through element-wise addition to form a feature map rich in multi-scale information. Furthermore, a dual attention strategy is employed to collaboratively optimize feature representation capabilities from both channel and spatial dimensions: the Efficient Channel Attention (ECA) branch uses one-dimensional convolution to adaptively adjust channel weights, enhancing the expression of important feature channels; the Spatial Attention (SA) branch combines spatial descriptors of average pooling and max pooling, generating a spatial attention weight map through a 7×7 convolution, accurately reflecting the importance distribution at each location in the image.

[0041] In the next path, edge detection is performed on the input image using the predefined Sobel operator, followed by two 3×3 convolutional layers to deepen the edge feature representation.

[0042] Finally, the feature map output from the upper path and the edge features extracted from the lower path are fused through an adaptive gating mechanism to dynamically balance the contribution ratio of semantic features and edge features, avoid the loss of details caused by feature conflicts, and provide a high-precision state representation for subsequent reinforcement learning decisions.

[0043] S4. Input the fused feature map into the policy network, and use the policy network to output the action probability distribution of each pixel for each predefined image processing operation based on the input fused feature map; the policy network includes a gated recurrent unit, which is used to model the spatial dependency between pixels.

[0044] S5. Based on the action probability distribution, determine the predefined image processing operation applied to each pixel at the current time step.

[0045] In this embodiment, random sampling is performed based on the action probability distribution to determine the predefined image processing operation applied to each pixel at the current time step.

[0046] In this embodiment, the action set designed for LDCT images includes 14 image processing operations, which are divided into two categories: basic denoising and structure preservation. Together, they constitute the decision space of the agent, as shown in Table 1.

[0047] Table 1. List of predefined image processing operations

[0048] The basic denoising component comprises six core actions: a baseline option of inaction, maintaining the current state and providing the agent with a "do nothing" option; two 5×5 Gaussian filters with different variances (σ = 0.5 or σ = 1.5) to handle noise regions of varying intensities; and two 5×5 bilateral filters that adjust the color space parameters (color space standard deviation σ). c ) and coordinate space parameters (coordinate space standard deviation σ) s To achieve differentiated smoothing effects, two bilateral filters with different parameter configurations are used in combination to adapt to the density differences of different tissues in CT images; the 5×5 median filter is specifically designed for impulse noise.

[0049] Nonlocal mean filtering (filter strength parameter h = 5, 10, 15) utilizes image self-similarity to protect tissue structure; guided filtering (regularization parameter eps = 0.01) maintains edge sharpness while smoothing uniform regions; adaptive sharpening filtering (sharpening factor α = 1.5 or 2.5) enhances edges and details using the difference of Gaussians method; convolutional enhancement uses a three-layer convolutional network to improve sharpness, and residual enhancement uses a 20-layer residual network to restore details.

[0050] During the image restoration process, the algorithm adopts an adaptive denoising strategy, which first performs global noise suppression, then moves to local detail repair, and finally completes edge structure optimization, forming a hierarchical processing flow S6. The predefined image processing operations for each pixel are applied to the current state image to obtain the reconstructed image.

[0051] S7. Calculate a composite reward based on the reconstructed image, the current state image, and the high-resolution reconstructed image; the composite reward is a weighted sum of pixel-level fidelity reward, gradient structure similarity reward, and local variance penalty term.

[0052] S8. The current state image input value network is evaluated to obtain the state value of each pixel, and the composite reward and the state value are combined to determine whether the iteration termination condition is met.

[0053] The termination condition can be set as follows: calculate the advantage function based on the composite reward and the state value; calculate the policy loss of the policy network based on the advantage function and the action probability distribution; simultaneously calculate the value function loss of the value network based on the composite reward and the state value; update the parameters of the policy network and the value network through backpropagation using the policy loss and the value function loss; determine whether a preset training termination condition has been met, the training termination condition including the current training step count reaching a preset maximum value.

[0054] Specifically, both the policy network and the value network are constructed using end-to-end fully convolutional networks (FCNs). The policy network processes the states of all pixels simultaneously, outputting the state transition probability distribution π(a) for each pixel. i,t |s i,t ;θ π (i=1,2,...,N, where N is the total number of pixels, a) i,t and s i,t These are the current action and current state of the i-th pixel, respectively, θ π The parameters of the policy network are used to model the spatial dependencies between pixels through Gated Recurrent Units (GRUs), while eliminating mutual interference between adjacent agents. The last layer of the network generates the corresponding feature map for each possible action, which is normalized by the Softmax function to obtain the action probability distribution for each pixel. During training, this distribution is randomly sampled to select a specific denoising operation for each pixel. The agents consist of a structure-details co-extraction module, a policy network, and a value network. Each agent processes information for one pixel, but agents can interfere with each other; the GRU can eliminate this interference.

[0055] The value network outputs the expected cumulative reward V(s) for each pixel state. i,t ;θ v ), where θ v These are the parameters of the value network. In the initial state, the state of each pixel is represented as s. i,0 =I(i), where I(i) is the intensity value of the i-th pixel in the input image I.

[0056] Both the policy network and the value network take the current image state as input, but they optimize different objective functions: the policy network focuses on optimizing action selection, while the value network accurately evaluates the value of the state. The two work together to help the agent learn the optimal denoising strategy.

[0057] Specifically, the policy network processes the state of all pixels simultaneously. Taking the multi-scale feature map output by the SDCFE module as input, the forward computation first undergoes two convolutional operations: the first layer is a 3×3 convolution with a stride of 3 (outputting 64 channels), completing the initial compression and encoding of features; the second layer is a 3×3 convolution with a stride of 2 (outputting 64 channels), further refining the feature representation and adjusting the feature map size. Subsequently, a gated recurrent unit (GRU) layer with a stride of 1 is connected to model the spatial dependencies between pixels and eliminate interference between adjacent agents. The final layer of the network generates feature maps corresponding to the actions through convolution. After normalization using the Softmax function, the action probability distribution of each pixel is obtained. (1) In the formula, i=1,2,...,N represents the pixel index, N is the total number of pixels, t is the time step, and a i,t and s i,t These are the current action and current state of the i-th pixel, respectively, θ π These are policy parameters, where k=1,2,...,14 represents the action index, and L... k,i The action a corresponds to the i-th pixel. k The original input scores. During training... Random sampling is used to match an appropriate denoising operation for each pixel, which, while ensuring the diversity of exploration, drives the strategy to gradually converge to the optimal value.

[0058] The first layer of the value network is a 3×3 convolution with a stride of 2 (outputting 32 channels), which performs initial dimensionality reduction and encoding of the input features; the second layer is a 3×3 convolution with a stride of 1 (outputting 1 channel), which completes the final integration of features. The output of the value network is the expected cumulative reward for each pixel state. , where θ v These are the parameters of the value network, and its output 1-channel feature map corresponds to the state value of each pixel. In the initial state, the state of each pixel is represented as... , where I(i) is the intensity value of the i-th pixel in the input image I. During training, the discounted total reward at time step t is defined as: (2) In the formula, r (t) The immediate reward is the reward received at time step t; r (t+1) This is the immediate reward obtained at time step t+1; γ is the normalized discount factor, generally ranging from 0 ≤ γ ≤ 1, representing the importance of future rewards relative to current rewards. The smaller γ is, the more emphasis is placed on recent rewards; r (t+2) It represents the instant reward at time step t+2, and overall represents the reward value after two steps of discounting; γ n-1 r (t+n-1)This represents the reward contribution at time step t+n-1 after n-1 discount steps; γ n r (t+n) This represents the reward contribution at time step t+n after n discount steps. By minimizing the mean squared error between the predicted state value and the target value (current reward + discount value of the next state value), the accuracy of state value assessment is improved, providing a reliable value reference for the action selection of the policy network.

[0059] This embodiment designs a multi-dimensional composite reward function (i.e., composite reward), which achieves higher quality denoising effect by synergistically optimizing pixel-level fidelity, structural similarity and local statistical characteristics, and solves the problem of balancing noise suppression and detail preservation.

[0060] The composite reward function is defined as: (3) In the formula, r pixel r gradient and r var These are pixel-level rewards, gradient similarity rewards, and local variance penalties, respectively. λ1, λ2, and λ3 are weighting factors, determined through grid search optimization.

[0061] Pixel-level reward directly measures the difference between the current denoised image and the target image, and is defined as follows: (4) In the formula, and These are the pixel errors of the previous and current states, respectively, and scale is a factor for adjusting the reward magnitude. pixel By providing real-time feedback through the temporal differential form of the error, the agent is prompted to continuously optimize the denoising effect.

[0062] Gradient similarity reward is defined as: (5) In the formula, SSIM(·,·) represents the Structural Similarity Index Measure. Represents the gradient of an image. current Indicates the current image gradient. target Represents the gradient of the target image. gradient By using SSIM to measure the similarity between the current image and the target image in the gradient domain, image structural information can be better preserved, especially in noisy regions.

[0063] The local variance penalty term is defined as: (6) In the formula, Var(I current) represents the local variance of the current image, and mask is a smoothing mask used to reduce the penalty intensity in high-gradient regions. The local variance penalty mechanism identifies noisy regions by calculating the variance of local regions. It can adaptively distinguish between real anatomical texture (high variance, high gradient) and noise (high variance, low gradient), and reduce its impact on the overall image quality through a smoothing mask, achieving more accurate noise suppression while maintaining reasonable texture features.

[0064] S9. If the termination condition is not met, the reconstructed image is used as the updated current state image, and the process returns to the step of "using the structure-detail co-feature extraction module to extract features of the current state image through parallel processing of the upper and lower paths, and fusing the outputs of the two paths to obtain a fused feature map"; if the termination condition is met, the trained deep reinforcement learning model is obtained; the deep reinforcement learning model includes: a structure-detail co-feature extraction module, a policy network, and a value network; the trained deep reinforcement learning model is used for high-resolution reconstruction of low-dose computed tomography images.

[0065] As a specific implementation method, this embodiment also proposes network execution steps and training steps when using the deep reinforcement learning model for image reconstruction, as follows: Network execution steps: As shown in Figure 5, the network execution steps are as follows: Step 1, Initialize network parameters: The number of input feature channels and the number of output feature channels are 1 and 64 respectively, the number of convolutions conv=0, the number of poolings pool=0, and the number of upsamplings upsample=0.

[0066] SDCFE module execution: Main line A module inputs a feature map with dimensions H×W×64.

[0067] Step A1: Use 3×3 convolution to obtain 64-channel features, and then pass them in parallel through dilated convolution branch, identity mapping branch, detail enhancement branch and noise suppression branch.

[0068] Step A2: Concatenate the feature maps output from the four branches in the channel dimension to obtain a feature map with dimensions H×W×256.

[0069] Step A3: Use a 1×1 convolution to reduce the number of channels from 256 to 64 to obtain the basic fused feature map F. base .

[0070] Step A4, (Channel Attention) for F base Global average pooling is performed, and channel attention weights are obtained through Conv1D and the Sigmoid function, and then compared with F. base Multiplication yields the channel attention feature F CA .

[0071] Step A5 (Spatial Attention) for F CA Perform average pooling and max pooling along the channel dimension, concatenate the data, and then pass it through a 7×7 convolution and a sigmoid function to obtain spatial attention weights, which are then compared with F. CA Multiplying them together yields the final output F of the main line. A The dimensions are H×W×64.

[0072] Main line B shares the same input as main line A, which is a module input feature map with dimensions of H×W×64.

[0073] Step B1: Convolve the input features using fixed Sobel operators (horizontal kernel and vertical kernel) to extract gradient information in both directions.

[0074] Step B2: Segment the obtained horizontal gradient map and vertical gradient map along the channel dimension.

[0075] Step B3: The concatenated gradient features are passed through two consecutive 3×3 convolutional layers (each with a stride of 1 and padding of 1, outputting 64 channels). Each convolutional layer is followed by a batch normalization (BN) layer to stabilize the training and introduce non-linear transformation capabilities, ultimately outputting a 64-channel Sobel structure feature F. B The dimensions are H×W×64.

[0076] Step 2: Concatenate the outputs of the two main lines to obtain a feature map with dimensions H×W×128. Perform a 3×3 convolution on the concatenated feature map (stride of 1 and padding of 1 per layer), and apply the Softmax function to obtain F. A and F B The fusion weights at this position are calculated by weighting the outputs of the two main lines using the generated adaptive weight map to obtain the fused feature map. Finally, the fused feature map is obtained by passing it through a BatchNorm layer and a Dropout layer.

[0077] Step 6: Pass F through dilated convolution blocks with dilation factors of 2 and 3 to obtain F1.

[0078] Policy network execution: Step 7, F1 performs 3×3 convolutions (stride 1, padding 1) through 64 feature channels, and calculates update gate, reset gate and candidate state respectively.

[0079] Step 8: Weigh and fuse the updated gate, the hidden state of the previous time step, and the current candidate state to obtain the hidden state of the current time step.

[0080] Step 9: Perform a 3×3 convolution (stride 1, padding 1) on the current hidden state through 64 feature channels to obtain the probability distribution corresponding to the number of actions. .

[0081] Step 10: Pass F through dilated convolution blocks with dilation factors of 2 and 3 to obtain F2.

[0082] Value network execution: Step 11, F2 performs one 3×3 convolution (stride 1, padding 1) through 64 feature channels to obtain the expected total reward of the current state. The number of output channels becomes 1.

[0083] Training the network: (1) Loss function: The total loss function of the network is composed of the policy loss. Sum-value function loss The weighted summation consists of: (7) In the formula, η is the weighting factor for balancing the two losses, and its specific value is determined by cross-validation.

[0084] The policy loss is constructed based on policy gradient theory and aims to maximize the expected cumulative reward. It is defined as follows: (8) In this equation, N is the total number of pixels in the image. This represents the state-action transition probability output by a policy network with parameter θ. It is the dominance function.

[0085] The optimization objective of the value network is to minimize the error in the estimated state value, and its loss function is: (9)(2) Training steps: As shown in Figure 6, the training steps of the network are as follows: Step 1: Initialize the number of times to traverse the training set episode=0.

[0086] Step 2: Initialize global parameters , , and Initialize the parameters for each thread , and Initialize the thread time t=1.

[0087] Step 3: Reset gradients , , .

[0088] Step 4: Synchronize global parameters to thread-specific parameters , , .

[0089] Step 5: Obtain the current state for all threads. .

[0090] Step 6: If the boolean parameter is True, select the action with the highest probability; otherwise, randomly sample an action.

[0091] Step 7: Perform the action Each thread receives a reward. and the next state .

[0092] Step 8, let , .

[0093] Step 9, Judgment Is it the final state, or Check if the condition is true. If not, return to step 5; otherwise, set episode←episode+1 and execute step 10. If this is the final state, then the total discount reward is... If it is not the final state, then: In formula (10), It is a local region centered on the i-th pixel. It represents the weight, indicating the degree to which the value of adjacent pixels is considered in the next time step.

[0094] Step 10: If episode < 30000, calculate the policy loss and value function loss, perform backpropagation to calculate the gradient, update the parameters, and return to step 4; otherwise, stop the iteration, and the training process ends. The gradient update calculation is as follows: (11) (12) (13).

[0095] Based on the same inventive concept, this application also provides a method for reconstructing computed tomography (CT) images, including: acquiring low-dose CT images.

[0096] The low-dose computed tomography image is input into the trained deep reinforcement learning model to obtain a high-resolution reconstructed image; the trained deep reinforcement learning model is trained by the deep reinforcement learning model training method described above.

[0097] In an exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram is shown in Figure 7. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a deep reinforcement learning model training method or a computed tomography image reconstruction method.

[0098] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. A specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0099] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0100] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0101] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of the relevant data are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.

[0102] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0103] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0104] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0105] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method for a deep reinforcement learning model, characterized in that, include: Obtain training samples; The training samples include: computed tomography (CT) images and corresponding high-resolution reconstructed images; the CT images are used as the initial current state images and input into the structure-detail co-feature extraction module; the structure-detail co-feature extraction module extracts features from the current state images through parallel processing of the upper and lower paths, and fuses the outputs of the two paths to obtain a fused feature map; wherein, the upper path performs the following operations: after performing basic convolution on the current state image, it extracts multi-scale features through a parallel structure including dilated convolution branches, identity mapping branches, detail enhancement branches, and noise suppression branches, and merges the outputs of each branch. After fusion, channel attention and spatial attention are applied sequentially to obtain enhanced semantic features. The next path performs the following operations: edge detection is performed using the Sobel operator, followed by convolutional layer deepening to obtain enhanced edge features. The fusion operation specifically involves concatenating the enhanced semantic features with the enhanced edge features and generating a fused feature map through an adaptive gating mechanism. The fused feature map is input into a policy network, which outputs the probability distribution of actions for each pixel to select each predefined image processing operation based on the input fused feature map. The policy network includes a gated recurrent unit. The loop unit is used to model the spatial dependencies between pixels; based on the action probability distribution, a predefined image processing operation is determined for each pixel at the current time step; the determined predefined image processing operation for each pixel is applied to the current state image to obtain the reconstructed image; a composite reward is calculated based on the reconstructed image, the current state image, and the high-resolution reconstructed image; the composite reward is a weighted sum of pixel-level fidelity reward, gradient structure similarity reward, and local variance penalty term; the current state image is input into the value network for evaluation to obtain the state value of each pixel, and the composite reward and the state value are combined for judgment. The algorithm checks whether the iteration termination condition is met. If the termination condition is not met, the reconstructed image is used as the updated current state image, and the algorithm returns to the step of "using the structure-detail co-extraction module to extract features of the current state image through parallel processing of the upper and lower paths, and fusing the outputs of the two paths to obtain a fused feature map". If the termination condition is met, the trained deep reinforcement learning model is obtained. The deep reinforcement learning model includes: a structure-detail co-extraction module, a policy network, and a value network. The trained deep reinforcement learning model is used for high-resolution reconstruction of low-dose computed tomography images.

2. The training method for the deep reinforcement learning model according to claim 1, characterized in that, The step of using the computed tomography image as the initial current state image and inputting it into the structure-detail co-extraction module specifically includes: performing an Anscombe transform on the computed tomography image to convert the Poisson noise in it into Gaussian noise to obtain a preprocessed image; and using the preprocessed image as the initial current state image and inputting it into the structure-detail co-extraction module.

3. The training method for the deep reinforcement learning model according to claim 1, characterized in that, The detail enhancement branch includes a small-scale branch and a medium-scale branch; the small-scale branch captures fine structures through cascaded 1×3 and 3×1 convolutional layers; the medium-scale branch processes medium anatomical structures through two cascaded 3×3 convolutional layers.

4. The training method for the deep reinforcement learning model according to claim 1, characterized in that, Based on the action probability distribution, the predefined image processing operation applied to each pixel at the current time step is determined, specifically including: random sampling according to the action probability distribution to determine the predefined image processing operation applied to each pixel at the current time step.

5. The training method for the deep reinforcement learning model according to claim 1, characterized in that, The predefined image processing operations include: no action, Gaussian filtering with different variances, bilateral filtering with different parameters, median filtering, nonlocal mean filtering, guided filtering, adaptive sharpening filtering, convolution enhancement, and residual enhancement.

6. The training method for the deep reinforcement learning model according to claim 1, characterized in that, The value network is a fully convolutional network. The value network takes the current state image as input, processes it through at least one convolutional layer, and outputs a single-channel feature map. The value at each position in the single-channel feature map represents the state value of the corresponding pixel. Specifically, the value network obtains the state value of each pixel through the following operations: performing dimensionality reduction and encoding on the input features through a 3×3 convolutional layer with a stride of 2; and performing feature integration through a 3×3 convolutional layer with a stride of 1 to output a single-channel feature map.

7. A method for reconstructing computed tomography (CT) images, characterized in that, include: Acquire low-dose computed tomography (CT) images; input the low-dose CT images into a trained deep reinforcement learning model to obtain a high-resolution reconstructed image; the trained deep reinforcement learning model is trained by the training method of the deep reinforcement learning model according to any one of claims 1-6.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that the processor executes the computer program to implement the training method of the deep reinforcement learning model according to any one of claims 1-6 or the computed tomography image reconstruction method according to claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the training method of the deep reinforcement learning model according to any one of claims 1-6 or the computed tomography image reconstruction method according to claim 7.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the training method of the deep reinforcement learning model according to any one of claims 1-6 or the computed tomography image reconstruction method according to claim 7.

Images