A deep learning-based temperature imaging method for photothermal therapy target area

Abstract

The present invention discloses a method for temperature imaging of target areas for photothermal therapy based on deep learning. The method comprises the following steps: Step 1: hardware composition and configuration of a multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities; Step 2: signal acquisition and preprocessing; Step 3: construction and training of a TEM-UNet deep learning network; and Step 4: using a traditional reconstruction algorithm to generate multiple individual modal data images from the preprocessed signals in Step 2. The individual modal data images are then processed through the TEM-Unet deep learning network for precise target segmentation, temperature image reconstruction, and multimodal image information fusion. By introducing the deep learning TEM-UNet network architecture, this method effectively segments tumor targets and achieves selective temperature image reconstruction, significantly improving temperature measurement accuracy during photothermal therapy while reducing the complexity of the system hardware. The method is suitable for clinical use in tumor photothermal therapy.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a method for temperature imaging of a target area for photothermal therapy, and specifically to a method for temperature imaging of a target area for photothermal therapy based on deep learning. Background Art

[0002] In recent years, photothermal therapy, a promising treatment approach with the advantages of non-invasiveness and high selectivity, has attracted widespread attention in the field of tumor treatment. However, due to the lack of effective non-contact temperature measurement technology, accurate and real-time temperature measurement of the target area is difficult. As a result, photothermal therapy for tumors can cause damage to healthy tissue surrounding the target area. The rapid development of deep learning technology, particularly in the fields of image reconstruction and signal processing, has provided new solutions. Summary of the Invention

[0003] To overcome the lack of high-precision, non-invasive temperature measurement methods in existing photothermal therapy, this paper provides a deep learning-based method for temperature imaging of target areas in photothermal therapy. By incorporating the deep learning TEM-UNet network architecture, this method effectively segments the tumor target and enables selective temperature image reconstruction. This significantly improves temperature measurement accuracy during photothermal therapy while reducing system hardware complexity, making it suitable for clinical use in tumor photothermal therapy.

[0004] The purpose of the present invention is achieved through the following technical solutions:

[0005] A method for temperature imaging of a photothermal therapy target area based on deep learning, comprising the following steps:

[0006] Step 1. Hardware composition and configuration of the multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities:

[0007] The multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities consists of a pulsed laser, a linear array ultrasonic transducer, a multi-channel data acquisition card, and a central computer. The pulsed laser is used to excite the light absorber in the target area to generate a photoacoustic signal; the linear array ultrasonic transducer is used to acquire the ultrasound and photoacoustic dual modal signals; and the multi-channel data acquisition card is used to digitize the ultrasound and photoacoustic dual modal signals received by the linear array ultrasonic transducer and transmit the processed data to the central computer for further analysis and processing.

[0008] Step 2: Signal acquisition and preprocessing:

[0009] Step 2.1: The pulsed laser emits short pulses of light, which are irradiated on the tissue within the target imaging area, stimulating a photoacoustic signal. The generation of the photoacoustic signal is expressed by the following formula:

[0010] p0(r)=Γ·μ a (r)·F(r)

[0011] where p0(r) represents the initial sound pressure distribution at position r, Γ is the Grüneisen coefficient, and μ a (r) is the light absorption coefficient of the tissue, and F(r) represents the local light energy density;

[0012] Step 2.2: Through timing design, a single linear array ultrasound transducer receives the photoacoustic signal and ultrasound signal of the tumor target area. These signals are collected by a multi-channel data acquisition card and stored in a central computer.

[0013] Step 2.3: Preliminary preprocessing is performed on the signals collected by the multi-channel data acquisition card in the central computer to remove noise and enhance the effective part of the signal;

[0014] Step 3: Construction and training of TEM-UNet deep learning network:

[0015] Step 3.1, Construction of TEM-UNet deep learning network:

[0016] The first half of the TEM-UNet deep learning network is the encoder, which consists of four sub-modules, each of which contains two convolutional layers for extracting features from the input image; each sub-module is followed by a maximum pooling layer for downsampling, which extracts more abstract features by reducing the spatial size of the feature map and doubling the feature channels; the second half is the decoder, which consists of four sub-modules, each of which receives the feature map from the corresponding layer of the encoder through a jump connection. Each sub-module starts with an upsampling operation, and the upsampled feature map is fused with the feature map of the corresponding layer in the encoder. The fused feature map undergoes processing through two convolutional layers, and each convolutional layer is followed by a nonlinear activation function ReLU for extracting higher-level features and reducing the number of channels of the feature map. During the upsampling process, the network gradually restores the resolution of the medical image, making it gradually The resolution of the original input image is gradually approached. Through this process, the network can finely locate and segment different areas in the image, thereby achieving the same resolution as the input image and generating high-quality image segmentation results. On this basis, the eSR-TF module is introduced at the jump connection to strengthen the fusion of image features, and the self-attention mechanism is used to upsample the filtered feature image pixel reorganization. After feature fusion, the WCSA module is used to fuse the encoder feature x1 and the decoder feature x2 channels, and the channel attention mechanism is used for feature extraction. The feature map is added with the decoder feature x3 upsampled by the eSR-TF module, and finally the feature map x4 is output after spatial attention feature extraction, thereby enhancing the interaction of image features in the channel and spatial dimensions, and compensating for the loss of spatial information that may occur during the downsampling process, thereby improving the quality of ultrasound image segmentation results.

[0017] Step 3.2, TEM-UNet deep learning network training: During the training process, the cross entropy loss function is used to minimize the difference between the final output image and the label image. The cross entropy loss function is expressed as:

[0018]

[0019] Where n represents the batch size, p i and y i Represent the probability prediction and true label of the i-th category respectively;

[0020] In step 4 and step 2, the preprocessed signals are reconstructed using traditional reconstruction algorithms to generate multiple individual modal data images. The individual modal data images are then processed through the TEM-Unet deep learning network for precise target segmentation, temperature image reconstruction, and multimodal image information fusion.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] 1. This paper proposes a new deep learning network (TEM-UNet) for photoacoustic temperature measurement and temperature image reconstruction, which significantly improves the accuracy of non-invasive temperature measurement based on the photoacoustic effect.

[0023] 2. The present invention adds a new eSR-TF module at the jump connection to fully integrate image features, reorganizes the upsampled pixels of the filtered feature image through the self-attention mechanism, and combines it with the WCSA module to compensate for the loss of spatial information that may be caused by the downsampling process.

[0024] 3. The present invention reduces the number of model channels and optimizes model parameters, so that better segmentation effects can be achieved even with a small amount of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Schematic diagram of the deep learning segmentation model TEM-UNet proposed in this invention;

[0026] Figure 2 Schematic diagram of the eSR-TF module;

[0027] Figure 3 Schematic diagram of the WCSA module. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0029] The present invention provides a method for temperature imaging of a target area for photothermal therapy based on deep learning, the method comprising the following steps:

[0030] Step 1. Hardware composition and configuration of the multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities:

[0031] The present invention provides a multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities, which is intended to provide the basic information required for subsequent temperature reconstruction and multi-modal image fusion. The multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities is composed of the following main components: a pulsed laser, a linear array ultrasonic transducer, a multi-channel data acquisition card, and a central computer, wherein: the pulsed laser is used to excite the light absorber in the target area to generate a photoacoustic signal; the linear array ultrasonic transducer uses a linear array probe with a center frequency of 7.5MHz to obtain ultrasound and photoacoustic dual modal signals. The multi-channel data acquisition card digitizes the ultrasound and photoacoustic dual modal signals received by the linear array ultrasonic transducer and transmits the processed data to the central computer for further analysis and processing. Through a special timing design, the multi-bandwidth photoacoustic imaging system can realize real-time ultrasound / photoacoustic dual modal image reconstruction of the target area, thereby providing rich tumor structure and function information.

[0032] Step 2: Signal acquisition and preprocessing:

[0033] During the experimental operation, the pulsed laser emits short pulses of light, which irradiate the tissue within the target imaging area and stimulate the generation of photoacoustic signals. The generation of photoacoustic signals can be expressed by the following formula:

[0034] p0(r)=Γ·μ a (r)·F(r)

[0035] where p0(r) represents the initial sound pressure distribution at position r, Γ is the Grüneisen coefficient, and μ a (r) is the light absorption coefficient of the tissue, and F(r) represents the local light energy density.

[0036] The linear variation of the Grüneisen coefficient with temperature is the basis for non-invasive temperature measurement using the photoacoustic effect, which can be expressed as:

[0037] Γ=A+BT

[0038] Among them, A and B are two constants that depend on the properties of the tissue and do not change with the temperature T. Therefore, the intensity of the photoacoustic signal reflects the local temperature of the tissue to a certain extent.

[0039] At different times, a pulsed laser provides external triggering, while the system provides internal triggering. Sound propagation time is calculated, and the system's control timing is designed based on the two trigger signals. This allows a single linear array probe to receive both photoacoustic and ultrasound signals from the tumor target. These signals are acquired by a multi-channel data acquisition card and stored in a central computer. To ensure the accuracy and efficiency of subsequent image reconstruction, the signals collected by the multi-channel data acquisition card undergo preliminary preprocessing in the central computer to remove noise and enhance the effective signal portion, ensuring efficient subsequent dual-modality image reconstruction. The preprocessed signals are then reconstructed using traditional algorithms such as delayed sum (DAS) to generate multiple separate modal image data. These separate modal image data are then processed using the TEM-Unet deep learning network for precise target segmentation, temperature image reconstruction, and multimodal image information fusion.

[0040] Step 3: Construction and training of the TEM-UNet deep learning network, a model for automatic segmentation of the photothermal therapy target area:

[0041] To achieve high-quality image segmentation, this paper improves on the traditional U-Net model and constructs a multi-scale, cross-dimensional information interaction-guided automatic photothermal therapy target segmentation model (TEM-UNet deep learning network) to automatically extract tumor regions from ultrasound images. This model, based on the traditional U-Net model, introduces two specialized modules: the eSR-TF (edge-SR TRansformer) module and the WCSA (Weighted Channel-Spatial Attention) module.

[0042] like Figure 1As shown in Figure 1, the first half of the TEM-UNet deep learning network is the encoder, which consists of four submodules. Each submodule contains two convolutional layers to extract features from the input image. Each submodule is followed by a max pooling layer for downsampling, which reduces the spatial size of the feature map and doubles the number of channels to extract more abstract features. The second half of the network is the decoder, which also contains four submodules. Each submodule begins with an upsampling operation, typically implemented using a transposed convolution (also called deconvolution). The goal is to increase the spatial resolution of the output feature map of the previous submodule to approach the size of the original input image. The upsampled feature map is then fused with the feature map of the corresponding level in the encoder. This fusion is typically achieved by resizing the encoder feature map to match the decoder feature map and then concatenating the two in the channel dimension, thus transferring information from the encoder to the decoder. The fused feature map undergoes two convolutional layers, each of which is typically followed by a ReLU nonlinear activation function to extract higher-level features and reduce the number of channels in the feature map. Each submodule of the decoder receives feature maps from the corresponding layer of the encoder via skip connections. This helps preserve detailed information captured by the encoder in the decoder, thereby restoring greater detail and accuracy in the final segmentation result. During the upsampling process, the network gradually restores the resolution of the medical image, gradually approaching that of the original input image. This process enables the network to finely locate and segment different regions within the image, achieving the same resolution as the input image and producing high-quality image segmentation results. Furthermore, an eSR-TF module is added to the skip connections to fully fuse image features. The filtered feature image is then upsampled and reconstructed using a self-attention mechanism. After feature fusion, a WCSA module is used to channel-wise fuse the encoder features x1 and decoder features x2. Feature extraction is then performed using a channel-wise attention mechanism. These features are then added to the decoder features x3 upsampled by the eSR-TF module. Finally, spatial attention is used to extract the feature map x4. This enhances the interaction of image features in both the channel and spatial dimensions, compensating for the spatial information loss that may occur during downsampling, thereby improving the quality of ultrasound image segmentation results. By introducing the eSR-TF module and the WCSA module, TEM-UNet can significantly improve the accuracy of image segmentation.

[0043] Because a small amount of training data facilitates optimization of model parameters, the number of model channels was reduced. During network training, the batch_size was set to 4, the number of training epochs was set to 100, and the model used 256×256 pixel images for both label images and final output. Training and validation were performed on an NVIDIA GTX1080Ti GPU platform with 11GB of memory and the PyTorch deep learning framework. During training, a cross-entropy loss function was used to minimize the difference between the final output image and the label image. This cross-entropy loss function is expressed as:

[0044]

[0045] Where n represents the batch size, p i and y i Represent the probability prediction and true label of class i, respectively. To validate the performance of the TEM-UNet deep learning network, the segmentation results of the proposed network were compared with those of several commonly used end-to-end image segmentation networks, including U-Net, U-Net++ based on U-Net with nested skip connections, and CE-Net using a context encoder module and a densely connected attention module. Three common evaluation metrics were used to evaluate the semantic segmentation performance: pixel accuracy (PAC), average intersection over union (IoU), and the Dice coefficient.

[0046] The Adam optimizer was used to optimize the network parameters. The learning rate during training was set to an initial value of 0.0001 and was optimized using a step-by-step reduction method to ensure the stability and convergence of the training.

[0047] Step 4: System Verification and Application

[0048] In order to verify the effectiveness of the present invention, relevant experiments were carried out to verify its reliability and applicability in practical applications.

[0049] The following examples demonstrate the image segmentation metrics and visual temperature measurement results of our model, demonstrating its superior performance in practical applications. Experimental results show that the proposed TEM-UNet deep learning network demonstrates excellent segmentation performance for target area segmentation. Compared to other networks, it also outperforms image quality metrics such as PAC, IoU, and Dice, as shown in Table 1.

[0050] Table 1 Segmentation indicators

[0051]

[0052] Experimental results show that TEM-UNet performs well in multi-bandwidth photoacoustic imaging tasks. In in vivo experiments, the system was successfully applied to non-invasive temperature measurement of mouse tumor target areas that were locally injected with photothermal nanoprobes. After segmenting the target area using TEM-UNet, the system performed well in indicators such as local temperature measurement accuracy and temperature correlation, verifying its reliability and wide applicability in practical applications. The present invention demonstrates high-precision non-invasive temperature measurement capabilities and provides a more advanced non-invasive temperature monitoring method for photothermal treatment of tumors. The present invention integrates imaging, segmentation, temperature measurement, treatment and other functional modules, accurately identifies the treatment area through the TEM-UNet deep learning network, filters out irrelevant information in the background position, and significantly improves the temperature measurement accuracy. Assisted by multimodal image fusion methods, it ensures that multimodal information of the target area can be obtained during the treatment process, provides staff with more intuitive visual feedback on the tumor status, improves the treatment effect, and shows significant advantages and huge application potential in the application of photothermal treatment.

Claims

1. A method for temperature imaging of target area in photothermal therapy based on deep learning, characterized in that The method comprises the following steps: Step 1. Hardware composition and configuration of the multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities: The multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasound dual modalities consists of a pulsed laser, a linear array ultrasonic transducer, a multi-channel data acquisition card, and a central computer. The pulsed laser is used to excite the light absorber in the target area to generate a photoacoustic signal; the linear array ultrasonic transducer is used to acquire the ultrasound and photoacoustic dual modal signals; and the multi-channel data acquisition card is used to digitize the ultrasound and photoacoustic dual modal signals received by the linear array ultrasonic transducer and transmit the processed data to the central computer for further analysis and processing. Step 2: Signal acquisition and preprocessing: Step 2.1: The pulsed laser emits short pulses of light, which are irradiated on the tissue within the target imaging area, stimulating a photoacoustic signal. The generation of the photoacoustic signal is expressed by the following formula: in, represents the initial sound pressure distribution at position r, is the Grüneisen coefficient, is the light absorption coefficient of the tissue, represents the local light energy density; Step 2.2: Through timing design, a single linear array ultrasound transducer receives the photoacoustic signal and ultrasound signal of the tumor target area. These signals are collected by a multi-channel data acquisition card and stored in a central computer. Step 2.3: Preliminary preprocessing is performed on the signals collected by the multi-channel data acquisition card in the central computer to remove noise and enhance the effective part of the signal; Step 3: Construction and training of TEM-UNet deep learning network: Step 3.1, Construction of TEM-UNet deep learning network: The first half of the TEM-UNet deep learning network is the encoder, which consists of four sub-modules, each of which contains two convolutional layers for extracting features from the input image; each sub-module is followed by a maximum pooling layer for downsampling, which extracts more abstract features by reducing the spatial size of the feature map and doubling the feature channels; the second half is the decoder, which consists of four sub-modules, each of which receives the feature map from the corresponding layer of the encoder through a jump connection. Each sub-module starts with an upsampling operation, and the upsampled feature map is fused with the feature map of the corresponding level in the encoder. The fused feature map undergoes two convolutional layers, each of which is followed by a nonlinear activation function ReLU for extracting higher-level features and reducing the number of channels of the feature map. During the upsampling process, the network gradually restores the resolution of the medical image, making it gradually approach the resolution of the original input image. Through this process, the network can finely locate and segment different areas in the image, thereby achieving the same resolution as the input image and generating high-quality image segmentation results; on this basis, edge-SR is introduced at the jump connection. The TRansformer module strengthens the fusion of image features and uses the self-attention mechanism to reconstruct the upsampled pixels of the filtered feature image; after feature fusion, the WCSA module is used to transform the encoder features into and decoder features After channel fusion, the channel attention mechanism is used for feature extraction and combined with the decoder features sampled by the edge-SR TRansformer module. Add the feature maps and finally output the feature map after spatial attention feature extraction , thereby enhancing the interaction of image features in both channel and spatial dimensions, and compensating for the loss of spatial information that may result from the downsampling process, thereby improving the quality of ultrasound image segmentation results; Step 3.2: Training the TEM-UNet deep learning network: During the training process, the cross entropy loss function is used to minimize the difference between the final output image and the label image; In step 4 and step 2, the preprocessed signals are reconstructed using traditional reconstruction algorithms to generate multiple individual modal data images. The individual modal data images are then processed through the TEM-Unet deep learning network for precise target segmentation, temperature image reconstruction, and multimodal image information fusion.

2. The method for photothermal therapy target area temperature imaging based on deep learning according to claim 1 is characterized in that The linear array ultrasonic transducer adopts a linear array probe with a center frequency of 7.5 MHz.

3. The method for photothermal therapy target area temperature imaging based on deep learning according to claim 1, characterized in that The Grüneisen coefficient is expressed as: Among them, A and B are two constants that depend on the properties of the tissue and do not change with temperature T.

4. The method for temperature imaging of target area of ​​photothermal therapy based on deep learning according to claim 1, characterized in that The cross entropy loss function is expressed as: Where, represents the batch size, and Represent the probability prediction and true label of the i-th category respectively.

Images