A Deep Learning-Based Quantitative Photoacoustic Imaging Method and System

Abstract

This invention discloses a quantitative photoacoustic imaging method and system based on deep learning. By constructing a Transformer module, this model combines the advantages of Transformer and U-Net, enhancing the modeling ability of global information through deep learning. It overcomes the problem of existing deep learning techniques failing to fully capture the correlation between different wavelengths in multi-wavelength data processing, thus limiting the quantitative accuracy of blood oxygen saturation. Simultaneously, this method also addresses the insufficient generalization ability of deep learning models across data domains (such as simulation and experimental data), improving the model's adaptability to practical clinical applications. Through comprehensive optimization of the photoacoustic imaging quantification process, it achieves higher accuracy, greater robustness, and wider applicability in the reconstruction of blood oxygen saturation.

Description

Technical Field

[0001] This invention relates to the field of photoacoustic image processing and analysis technology, and in particular to a quantitative photoacoustic image method and system based on deep learning. Background Technology

[0002] Existing photoacoustic imaging techniques have the following limitations in quantitative blood oxygen saturation imaging. Traditional methods based on physical modeling typically assume uniform tissue optical properties, making it difficult to accurately reflect the complexity of biological tissues. These methods are highly sensitive to noise and data sampling density, with reconstruction accuracy significantly decreasing under sparse sampling and limited viewing angle conditions. Furthermore, their high computational complexity makes them unsuitable for real-time requirements.

[0003] In recent years, deep learning methods have been introduced to solve the above problems, but existing deep learning models are mostly limited to local feature extraction and cannot make full use of the global correlation of multi-wavelength photoacoustic data.

[0004] Furthermore, the performance of deep learning models relies on high-quality labeled data and computational resources, exhibiting limited generalization ability across data domains (such as simulation and experimental data). These limitations mean that current technologies still face challenges in the accurate quantification and widespread clinical application of blood oxygen saturation.

[0005] Therefore, a quantitative photoacoustic imaging method is needed to overcome the limitations of traditional physical modeling methods that assume uniformity of tissue optical properties. Summary of the Invention

[0006] In view of this, the purpose of this invention is to provide a quantitative photoacoustic imaging method based on deep learning, which solves the problem of accurately retrieving optical absorption coefficient and blood oxygen saturation in complex biological tissues.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The quantitative photoacoustic image method based on deep learning provided by this invention includes the following steps:

[0009] S1. Photoacoustic data preparation: acquire simulation data and experimental data, wherein the photoacoustic data includes photoacoustic image data of different wavelengths;

[0010] S2. Model training and test set partitioning: Divide the dataset into training dataset, validation dataset, and test dataset;

[0011] S3. Construct a neural network model, which adopts the TransUNet model. A Transformer module is added to the U-Net network structure to capture long-distance dependencies in the dataset.

[0012] S4. Model training: Train the TransUNet model using the training set; optimize the model hyperparameters.

[0013] S5. Model Validation and Evaluation: Use the test set to validate the model and calculate model performance metrics.

[0014] S6. Image processing is performed using the trained TransUNet model. The input photoacoustic image is used to estimate the absorption coefficient, and the processed image or absorption coefficient distribution map is output.

[0015] Furthermore, the TransUNet model includes an input module, an encoder module, a Transformer module, a decoder module, and an output module;

[0016] The multi-channel input module is used to input photoacoustic image data of different wavelengths in different channels;

[0017] The encoder module includes multiple concatenated convolutional blocks for extracting local features from the input data; each convolutional block includes multiple convolutional layers, batch normalization layers, activation functions, and max pooling layers; features are extracted by progressively downsampling through each convolutional block.

[0018] The Transformer module is used to capture global dependencies and the correlation between different wavelengths;

[0019] The decoder module is used to progressively restore the resolution of the feature map and fuse low-level and high-level features to obtain a two-dimensional distribution map representing the absorption coefficient and blood oxygen saturation.

[0020] The output module is used to output the reconstructed quantitative photoacoustic image, including the distribution map of absorption coefficient and blood oxygen saturation.

[0021] Furthermore, the Transformer module includes several Transformer layers, each Transformer layer including a multi-head attention mechanism module and a feedforward neural network. The multi-head attention mechanism module is used to calculate the feature dependencies and global correlations between wavelengths; the feedforward neural network is used to further process and transform features.

[0022] Furthermore, the decoder module includes several serially connected upsampling blocks; used to implement multiple upsampling operations, each upsampling block is followed by a skip connection, and features are fused with the layer corresponding to the encoder to obtain a two-dimensional distribution map representing the absorption coefficient and blood oxygen saturation.

[0023] Furthermore, the photoacoustic data in step S1 is preprocessed in the following manner:

[0024] S11: Data format: Loaded as multi-channel data with shape (H,W,C), where H and W are the spatial dimensions of the image, and C represents the number of channels for different wavelengths;

[0025] S12: Normalization: The data for each wavelength is normalized to the interval [0, 1].

[0026] Furthermore, the specific steps of the Transformer module are as follows:

[0027] Step 1: Calculate the query, key, and value. For each input feature, generate three vectors: query vector, key vector, and value vector.

[0028] Step 2: Calculate attention weights: Obtain attention weights by calculating the similarity between the query and the key;

[0029] Step 3: Cross-wavelength correlation: For multi-wavelength data, each wavelength corresponds to an input channel, and the correlation between different wavelengths is calculated through a self-attention mechanism.

[0030] Furthermore, the multi-head attention mechanism module employs a multi-head attention mechanism for parallel computation, specifically according to the following formula:

[0031]

[0032] in, It is the first One point of attention, It is the output weight matrix.

[0033] Furthermore, the decoder module outputs the absorption coefficient and according to the following formula: :

[0034]

[0035] in, This represents the output, specifically the output absorption coefficient. and blood oxygen saturation The two-dimensional distribution.

[0036] Furthermore, the loss function in the TransUNet model is calculated according to the following formula:

[0037]

[0038] Where L is the joint loss function, Lmse is the mean squared error used to evaluate the difference between the predicted and the true values, α and β are weighting coefficients, and absorption represents the absorption coefficient; This indicates blood oxygen saturation.

[0039] The present invention also provides a quantitative photoacoustic imaging system based on deep learning, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method when executing the program.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention provides a deep learning-based quantitative photoacoustic imaging method. By constructing a Transformer module, this model combines the advantages of Transformer and U-Net, enhancing the modeling ability of global information through deep learning. It overcomes the problem of existing deep learning techniques failing to fully capture the correlation between different wavelengths in multi-wavelength data processing, thus limiting the quantitative accuracy of blood oxygen saturation. Furthermore, this method addresses the insufficient generalization ability of deep learning models across data domains (such as simulation and experimental data), improving the model's adaptability to practical clinical applications. Through comprehensive optimization of the photoacoustic imaging quantification process, it achieves higher accuracy, greater robustness, and wider applicability in the reconstruction of blood oxygen saturation. This method has the following characteristics:

[0042] 1. More accurate quantitative results: Many traditional algorithms assume a linear correlation between the initial photoacoustic pressure distribution and the light absorption coefficient. However, this assumption is inaccurate due to light scattering and attenuation in tissues, especially in deep tissues where the error is significantly amplified. This can lead to inaccurate estimations of the light absorption coefficient and blood oxygen saturation. The multi-channel input of the TransUNet model effectively models the relationships between different wavelengths, thereby improving the accuracy of photoacoustic image reconstruction and blood oxygen saturation estimation.

[0043] 2. Enhanced Global Understanding: Combining the Advantages of Transformer and U-Net: The traditional U-Net model is widely used in medical image processing, especially in image segmentation tasks in medical imaging. It relies on convolutional operations for local feature extraction. While the U-Net model performs well in image reconstruction and quantitative tasks, it often struggles to capture long-range global dependencies in images. TransUNet, by introducing the Transformer into the U-Net model, adds a self-attention mechanism to the local convolutional operations, enabling the network to capture long-range dependencies. The Transformer module excels at handling global contextual information, effectively improving the model's understanding of global features, especially in multi-wavelength data tasks, where it can better capture feature correlations between different wavelengths.

[0044] 3. Higher Computational Performance and Efficiency: While traditional convolutional neural networks (such as U-Net) perform well in image reconstruction tasks, their computational cost is high, especially when dealing with high-resolution images or large-scale data, resulting in slow training and inference speeds. The parallel computing capabilities of the Transformer module allow TransUNet to improve computational efficiency while maintaining high accuracy. The multi-head self-attention mechanism can process multiple data channels simultaneously, improving training and inference speeds, and maintaining high efficiency, especially when dealing with large-scale datasets.

[0045] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. Attached Figure Description

[0046] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following drawings are provided for illustration.

[0047] Figure 1 This is a system flowchart.

[0048] Figure 2 This is a diagram showing the main structure of the TransUNet neural network model.

[0049] Figure 3 This is a structural diagram of the Transformer module.

[0050] Figure 4 1. Comparison of experimental results of different methods on test samples.

[0051] Figure 5 2. Comparison of experimental results of different methods on test samples.

[0052] Figure 6 3. Comparison of experimental results of different methods on test samples. Detailed Implementation

[0053] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0054] Example 1

[0055] like Figure 1 As shown, the quantitative photoacoustic image method based on deep learning provided in this embodiment includes the following steps:

[0056] S1. Photoacoustic data preparation: acquire simulation data and experimental data, and ensure that the photoacoustic data includes photoacoustic image data of different wavelengths;

[0057] S2. The model training set and test set are divided according to the proportion of the dataset (e.g., 80% training, 10% validation, 10% test);

[0058] S3. Construct a neural network model, which adopts the TransUNet model. A Transformer module is added to the U-Net network structure. The Transformer module captures long-distance dependencies.

[0059] S4. Model training: Train the TransUNet model using the training set; optimize model hyperparameters (such as learning rate, batch size, etc.).

[0060] S5. Model Validation and Evaluation: Use the test set to validate the model and calculate performance metrics (such as error, correlation, CNR, etc.).

[0061] S6. Image processing is completed using the trained TransUNet model, that is, the model is applied to the actual image processing process, the absorption coefficient is estimated from the input photoacoustic image, and the high-quality processed image or absorption coefficient distribution map is output.

[0062] The TransUNet model in this embodiment can effectively process multi-wavelength photoacoustic image data and achieve high-precision reconstruction of blood oxygen saturation and absorption coefficient in complex biological tissues.

[0063] like Figure 2 As shown, the TransUNet model structure provided in this embodiment includes an input module, an encoder module, a Transformer module, a decoder module, and an output module; the specific structure of each part is as follows:

[0064] The input module is used to input photoacoustic image data, which is usually multi-channel data, with each channel corresponding to a different wavelength; it is used to input photoacoustic image data of different wavelengths in a different channel manner.

[0065] This embodiment utilizes different channels corresponding to different wavelengths, allowing the model to fully leverage the unique information of each wavelength in tissue, thereby improving the accuracy of quantitative analysis of key indicators such as absorption coefficient and blood oxygen saturation. Specifically:

[0066] 1. Capturing Diverse Optical Properties: Different wavelengths of light exhibit varying absorption characteristics in biological tissues, with the optical absorption coefficient (μa) showing significant differences across wavelengths. Using multi-wavelength, multi-channel input can more comprehensively reflect the optical properties of tissues, contributing to improved accuracy in quantitative estimation of blood oxygen saturation and absorption coefficient. Furthermore, each wavelength provides different tissue contrast and detail information. By combining information from different wavelengths across multiple channels, the model can more accurately extract multi-layered features of tissues, thereby enhancing imaging quality and accuracy.

[0067] 2. Improve the accuracy of blood oxygen saturation estimation: Different wavelengths of light have different absorption characteristics on oxidized hemoglobin (HbO2) and deoxygenated hemoglobin (HbR), which allows data from different wavelengths to provide more information to distinguish between the two. By using multi-wavelength photoacoustic image input, the model can more accurately distinguish the oxygenation status in the blood, thereby improving the accuracy of blood oxygen saturation estimation.

[0068] 3. Improved Reconstruction of Depth and Tissue Structure: Because photoacoustic imaging involves the propagation and absorption of light, the imaging accuracy of deep tissues is often affected. Different wavelengths have different penetration capabilities into deep tissues. By utilizing multi-wavelength data, it is possible to synthesize the responses of different wavelengths to deep structures, thereby improving the ability to reconstruct deep tissues.

[0069] The encoder module includes multiple concatenated convolutional blocks (Conv Blocks) for extracting local features from the input data;

[0070] The convolutional block includes multiple convolutional layers, batch normalization layers, activation functions (such as ReLU), and max pooling layers; each convolutional block progressively downsamples and extracts higher-level feature representations; for example, the input is mapped from 21 channels to 64 channels through the first convolutional block, and the number of channels is multiplied sequentially through subsequent convolutional blocks (128, 256), and the final output feature map shape is (H / 8, W / 8, 256).

[0071] The Transformer module is used to capture global dependencies and the correlations between different wavelengths;

[0072] In this embodiment, the Transformer module can effectively capture global dependencies and correlations between different wavelengths, mainly due to its two major features: self-attention and multi-head attention. Specifically:

[0073] 1. The core idea of ​​self-attention is that the network can autonomously determine the correlation between features at each location in the input and features at other locations. This mechanism is not limited to local regions but can perform global modeling of the entire input data, capturing the dependencies between different features. In image processing tasks, each pixel (or each spatial location) may have important relationships with other distant regions in the image. For example, in blood oxygen saturation estimation, the morphology and structure of blood vessels may span a large area. Traditional convolutional neural networks (CNNs) usually rely on local convolutional kernels for processing, making it difficult to effectively capture long-distance contextual information. Self-attention, however, can capture global dependencies by calculating the correlation between each location and all other locations, fully understanding the global structure of the image. In multi-wavelength photoacoustic imaging, the information of each wavelength is different, so the interrelationships between wavelengths are crucial. Self-attention allows the model to calculate the similarity and correlation between different wavelengths, enabling the network to understand how different wavelengths synergistically affect the optical properties of tissues (such as absorption coefficient and blood oxygen saturation) on a global scale. This helps the model capture the relationships between wavelengths more accurately, rather than simply processing each wavelength independently.

[0074] 2. Multi-head attention mechanisms, by parallelizing the attention mechanism, enable the model to independently learn different relationships between input features in multiple subspaces, thereby enhancing the model's expressive power. In processing multi-wavelength data, multi-head attention allows the model to learn the dependencies between different wavelengths from multiple perspectives. For example, some wavelengths may play a dominant role in absorption coefficient estimation, while others provide supplementary information. Multi-head attention mechanisms allow the model to learn these different correlations between wavelengths in parallel through multiple different heads (subspaces), enabling the model to capture the contribution of different wavelengths to the final result from more dimensions. Multi-head attention allows the model to perform feature fusion on the outputs of different heads, thereby enhancing the model's expressiveness. The features learned by each head can be weighted and combined; this flexible feature fusion can effectively capture the complex dependencies between different wavelengths, improving the final accuracy of quantitative photoacoustic imaging.

[0075] like Figure 3 As shown, the Transformer module includes Transformer*12 (Multi-head + FFN); that is, it includes 12 Transformer layers, each of which contains a multi-head attention mechanism module and a feed-forward network (FFN).

[0076] The multi-head attention mechanism module is used to calculate the feature dependencies and global correlations between wavelengths;

[0077] In this embodiment, the process of calculating the feature dependencies between wavelengths mainly relies on the application of the self-attention mechanism in the Transformer module. This process helps the model capture the mutual influence and global dependencies between different wavelength data, thereby improving the accuracy in quantitative photoacoustic imaging. The specific steps are as follows:

[0078] Step 1: Calculate the query, key, and value. For each input feature (e.g., data for each pixel or wavelength in a multi-wavelength image), three vectors are first generated: a query vector, a key vector, and a value vector. These vectors are obtained by multiplying them with a learnable weight matrix. The following is the calculation of the query, key, and value: Given input features We obtain the query, key, and value through linear transformation:

[0079]

[0080] It is a learnable weight matrix. , and Represents a query, key, and value vector.

[0081] Step 2: Calculate Attention Weights: Attention weights are obtained by calculating the similarity between the query and the key. This similarity is typically measured using the dot product.

[0082]

[0083] in,

[0084] It is the dot product of the query vector and the key vector, which measures the relevance of the query to other location features;

[0085] It is the square root of the dimension of the key vector, used to prevent the gradient from vanishing due to an excessively large dot product result.

[0086] The softmax operation ensures that the weights are normalized across all positions, thus yielding the relative importance of each position.

[0087] Ultimately, the weights are applied to the value vector. The weighted output is obtained from the above.

[0088] Step 3: Cross-wavelength correlation:

[0089] For multi-wavelength data, each wavelength corresponds to one input channel, and the self-attention mechanism can calculate the correlation between different wavelengths. For example, suppose the input data contains... The photoacoustic images at each wavelength are analyzed using an attention mechanism that not only calculates the correlation within each wavelength but also the dependencies between different wavelengths. This allows the model to understand, from a global perspective, how different wavelengths synergistically influence the final estimation of the absorption coefficient and blood oxygen saturation.

[0090] The feedforward neural network is used for further processing and transformation of features, as follows:

[0091] 1. Feature processing and transformation of input data:

[0092] Photoacoustic imaging typically uses multi-wavelength lasers to acquire multi-channel data. In this case, the photoacoustic image of each wavelength represents different absorption characteristics of the photoacoustic signal. To enable the TransUNet model to effectively process this multi-wavelength data, it is first necessary to convert the image of each wavelength into different channels. Assume the input data contains... The data is at each wavelength, and the shape of the input photoacoustic image data is as follows: ,in, and These are the height and width of the image, respectively. This refers to the number of wavelengths (e.g., 21 wavelengths with a step size of 10 nm between 700 nm and 900 nm). Each wavelength corresponds to one channel; therefore, the entire dataset will be represented as a shape... The three-dimensional tensor. The meaning of a channel: Each channel corresponds to photoacoustic data at a different wavelength. For example, an image with a wavelength of 700 nm is in the first channel, an image with a wavelength of 710 nm is in the second channel, and so on.

[0093] To avoid the impact of value range differences between different wavelengths on model training, it is necessary to normalize the image for each wavelength:

[0094] Normalization formula:

[0095] in, It's location In the Pixel values ​​in each wavelength channel and They are the first Minimum and maximum values ​​for each wavelength channel. This normalization process ensures that data for all wavelengths are within the normal range. Within a certain range, it facilitates model training.

[0096] The data needs to be converted to a format suitable for TransUNet input. TransUNet models typically process input data as a two-dimensional sequence, while image data is three-dimensional. To accommodate the structure of TransUNet models, multi-wavelength image data needs to be converted to a suitable format, i.e., feature flattening.

[0097] Spatial flattening: First, flatten the spatial dimensions of the image. Flattened to one dimension. When processing multi-wavelength images, the input data will be from... Convert to This involves using multiple wavelengths of information at each location in the image as a feature vector. Thus, the feature of each pixel becomes a feature vector composed of... A vector consisting of wavelengths.

[0098] Output shape: The converted data shape is ,in This represents the number of pixels in the image. This represents the feature dimension (i.e., the number of wavelengths) of each pixel. This flattened input can be directly fed into TransUNet.

[0099] Feature processing in the Transformer module: Since the Transformer is a sequence-based model, it cannot directly capture spatial location information in the input data. Therefore, when converting image data into a one-dimensional sequence, positional encoding needs to be added to each location (i.e., each pixel) to preserve spatial information. Positional encoding ensures that the model can recognize spatial locations by embedding each location. For each pixel in the image, a positional code can be generated using sine and cosine functions:

[0100] ,

[0101] Where 𝑖 is the position index, 𝑘 is the dimension index, and d is the embedding dimension. In this way, positional encoding can provide unique spatial information for each pixel.

[0102] Multi-head self-attention mechanism

[0103] In the Transformer module, the multi-head self-attention mechanism is central, helping the model capture global dependencies between input features, including wavelength dependencies. For each input pixel location (or each wavelength), the Transformer calculates the similarity between that location and other locations (by calculating the similarity between the query and key vectors through a dot product), thus determining how each pixel focuses information from other pixels. In particular, the model can capture dependencies between different wavelengths. For example, some wavelengths may contribute significantly to the estimation of the absorption coefficient, while others provide supplementary information. The self-attention mechanism allows the network to compute these dependencies globally. The formula is as follows:

[0104]

[0105] in, These are query, key, and value vectors, respectively. It is the dimension of the key vector. This ensures that the correlation of each location is normalized and calculates the final weighted value.

[0106] In photoacoustic imaging, the dependence between different wavelengths is crucial, especially in the estimation of absorption coefficients and blood oxygen saturation. Through a self-attention mechanism, the model can calculate similarity between different wavelengths, automatically learning which wavelengths have a higher correlation with the final quantitative results, thereby enhancing the accuracy of quantitative imaging.

[0107] Output Feature Processing: After processing by the Transformer module, the image features are updated and passed to the decoder. The decoder progressively restores the spatial resolution of the image, ultimately generating distribution maps of absorption coefficient and blood oxygen saturation. The decoder uses skip connections corresponding to those in the encoder to combine low-level features with high-level features, recovering more details.

[0108] Finally, convolutional layers are used to map the feature map to the target output dimension (absorption coefficient) to complete quantitative reconstruction.

[0109] The decoder module is used to progressively restore the resolution of the feature map and fuse low-level and high-level features; it includes several cascaded up-sample blocks; it is used to implement multiple upsampling operations (such as deconvolution layers or transposed convolution layers), each upsampling block is followed by a skip connection, and features are fused with the corresponding layer of the encoder, finally obtaining a two-dimensional distribution of absorption coefficient and blood oxygen saturation (sO2) with dimensions (H, W, 2).

[0110] The output module is used to output the reconstructed quantitative photoacoustic image, including the distribution of absorption coefficient and blood oxygen saturation.

[0111] The TransUNet model provided in this embodiment combines the advantages of U-Net and Transformer to achieve efficient processing of multi-wavelength data. The encoder part extracts local features through convolutional blocks, the Transformer part captures global information through a multi-head attention mechanism, and the decoder part restores the original image size and fuses features through upsampling blocks, ultimately outputting a high-precision quantitative photoacoustic image.

[0112] The TransUNet model structure provided in this embodiment combines the advantages of U-Net and Transformer, fusing local features with global dependencies: it effectively extracts local features from the input data through convolutional operations in the encoder part, and introduces a self-attention mechanism in the Transformer part to capture feature dependencies between different wavelengths and global contextual information. This combination enables the model to not only handle local details but also understand the global information of the entire image, thereby improving the accuracy of photoacoustic image reconstruction.

[0113] Meanwhile, taking into account the characteristics of multi-wavelength photoacoustic data, the TransUNet model, utilizing its multi-channel input capability and the Transformer's multi-head attention mechanism, can more accurately model the relationships between different wavelengths. This helps improve the estimation accuracy of physiological parameters such as blood oxygen saturation, especially in deep tissues.

[0114] The input data preprocessing in this embodiment includes the following methods:

[0115] Data format: Loaded as multi-channel data with shape (H, W, C), where H and W are the spatial dimensions of the image, and C represents the number of channels for different wavelengths;

[0116] Normalization: The data for each wavelength is normalized to the [0, 1] interval to ensure that the dynamic range of all wavelength data is consistent.

[0117] Data classification: 80% is used for training and 20% is used for testing (10% of the training set is randomly divided as a validation set).

[0118] The encoder module in this embodiment includes: Feature Extraction Convolution: It comprises three convolutional blocks, each consisting of a convolutional layer, batch normalization, ReLU activation function, and max pooling layer. The first convolutional block maps the input from 21 channels to 64 channels, and the number of channels in subsequent convolutional blocks increases sequentially (128, 256), resulting in a final output feature map shape of (H / 8, W / 8, 256). Feature Vectorization: The two-dimensional feature map is flattened into a one-dimensional vector with shape (N, P, D), where N is the batch size, P is the number of feature maps, and D is the feature dimension.

[0119] Transformer Module: Encoding Location: Adds sine and cosine positional encoding to preserve spatial information. Multi-head Attention Mechanism: Parameters include 8 attention heads, 12 Transformer layers, and 256 hidden layers. This mechanism calculates feature dependencies and global correlations between wavelengths. Feedforward Neural Network (FFN): Each Transformer layer is followed by two fully connected layers. The first layer expands the feature dimension from D to 4D, and the second layer compresses it back to D.

[0120] Decoder Module: Upsampling and Fusion: Features corresponding to the encoder are progressively upsampled until the original image size is restored. Each upsampling stage includes a deconvolutional layer, skip connections (concatenating feature maps with the corresponding layers of the encoder), and a convolutional block (structured the same as the encoder).

[0121] Output layer: High-dimensional features are mapped to the target output dimension (absorption coefficient and sO2) using a separate convolutional layer, and the final output shape is (H, W, 2).

[0122] The loss function design in this embodiment is as follows: Multi-task loss: Define a joint loss function L, which combines mean squared error and other weighting coefficients to evaluate the difference between the predicted and actual values. Wavelength consistency: Introduce a wavelength regularization term to ensure global consistency of multi-wavelength data.

[0123] Training and optimization strategy: Use the Adam optimizer with an initial learning rate of 10. -4 A learning rate decay strategy was adopted; the batch size was set to 16. Data augmentation techniques included random pruning, rotation, horizontal flipping, and adding random noise to the wavelength data to improve model robustness.

[0124] The Transformer module provided in this embodiment improves the accuracy and efficiency of photoacoustic image reconstruction by combining the advantages of U-Net and Transformer, and performs particularly well when processing deep tissues and large-scale data.

[0125] Example 2

[0126] The photoacoustic image quantitative analysis method based on the TransUNet model provided in this embodiment has high quantitative accuracy and image reconstruction quality, and can be widely used in the field of medical imaging, providing effective support for clinical diagnosis and treatment. First, the neural network of the TransUNet model is trained and obtained using a photoacoustic image training dataset. Then, the trained TransUNet model is used to reconstruct the quantitative photoacoustic image, specifically following these steps:

[0127] Step 1: Input data preprocessing:

[0128] Data format preparation: The obtained simulation data and experimental data are loaded into 21 wavelength multi-channel data with the shape (H, W, 21), where H and W are the spatial dimensions of the image, and 21 represents the number of channels for the wavelength.

[0129] Experimental data: Experimental data typically comes from actual photoacoustic imaging experiments, using real photoacoustic imaging equipment to image biological tissues. The advantage of experimental data is that it reflects the characteristics of real-world biological tissues, including tissue heterogeneity, noise, and measurement errors. It directly reflects the optical absorption and scattering characteristics of real tissues, providing a realistic reflection of actual pathological conditions (such as tumors and blood vessels); furthermore, training and evaluation using experimental data can help models be better applied in clinical settings, improving their practicality. However, experimental data often contains noise and measurement errors, and in some cases, data acquisition may be incomplete or limited (e.g., limited viewing angles or sparse sampling); in addition, experimental data is usually expensive and difficult to obtain, especially labeled data, which limits model training and validation.

[0130] Simulation data, generated through physical models or simulation platforms, simulates processes such as light wave propagation, tissue absorption, and scattering in photoacoustic imaging. Its advantages include the ability to generate a large number of diverse samples from models, making it suitable for training deep learning models, especially when experimental data is scarce; simulation data allows researchers to precisely control factors such as the optical properties of tissues, scattering effects, and noise, enabling training and testing under different experimental conditions; furthermore, the labels for simulation data (such as absorption coefficients and blood oxygen saturation) can be accurately generated through physical models, avoiding the problems of human labeling errors during experiments. However, a domain gap often exists between simulation data and actual experimental data, meaning that simulation models may not fully capture the complexity and variability of actual biological tissues. This gap may cause models to perform worse on real data than on simulation data. By inputting simulation data and experimental data in different proportions, the advantages of both can be leveraged, and their respective shortcomings can be compensated for.

[0131] Enhancing the diversity of training data: Simulation data provides large-scale, accurately labeled, and diverse training samples, while experimental data reflects the complexity of the real world. Combining these two allows the model to learn more diverse features and variations, improving its generalization ability.

[0132] Reduce the impact of experimental noise: By combining simulation data, the model can be trained on cleaner, more clearly labeled data, thereby reducing the interference of noise in the experimental data on training and improving the robustness of the model.

[0133] Improving the model's generalization ability: Pre-training with simulation data and fine-tuning with experimental data allows the model to learn the general patterns obtained from the simulation data while retaining the specific characteristics of the experimental data, thereby enhancing the model's performance and robustness in real-world scenarios.

[0134] Optimizing quantitative imaging accuracy: Combining these two types of data can effectively improve the accuracy of quantitative photoacoustic imaging tasks, especially in the estimation of blood oxygen saturation and absorption coefficient. Simulation data can help the model identify patterns under different tissues and conditions, while experimental data enables the model to better adapt to the complex situations in real-world applications.

[0135] Normalization: The data for each wavelength is normalized to [0, 1] using the following formula to eliminate the influence of different wavelength data scales and ensure that the dynamic range of all wavelength data is consistent.

[0136]

[0137] In this formula, the meanings of each letter are as follows:

[0138] : Represents the value of the original data, that is, a specific value in the input data;

[0139] : Represents the minimum value in the original dataset xxx;

[0140] : Represents the maximum value in the original dataset xxx;

[0141] This represents the normalized data value, that is, the result after normalizing the original data using this formula.

[0142] Data classification: All data are divided into 80% training set and 20% test set (10% of the training set is randomly allocated as a validation set).

[0143] Model parameter settings: Input data processing: Input data includes multi-wavelength photoacoustic images, in the form of... ,in: and It is the spatial dimension of the image; This refers to the number of wavelengths, consisting of 21 wavelengths ranging from 700 nm to 900 nm with a stride of 10 nm. Features are extracted from the data channels of each wavelength through convolution operations. The image is flattened and fed into the TransUNet module for further processing. The input data needs to be normalized to ensure it has a uniform scale range.

[0144] Step 2, Encoder

[0145] The encoder extracts features from the input image step-by-step using multiple convolutional blocks. Each convolutional block includes the following steps:

[0146] Convolutional layers: using Convolutional kernels are used for feature extraction, and the number of convolutional kernels increases layer by layer (e.g., 64 -> 128 -> 256).

[0147]

[0148] in, These are input features. It is a convolution kernel. It is a bias.

[0149] Batch Normalization: Normalizes the convolutional output to speed up training and reduce overfitting.

[0150] Activation function (ReLU): The ReLU activation function ensures the non-linear expression of the network.

[0151]

[0152] Max Pooling: employs Pooling layers perform downsampling, reducing spatial dimensions and retaining the most salient features.

[0153] Feature flattening: After the convolutional layer, the feature maps are flattened into one-dimensional vectors to fit the format of the Transformer input. ,in It is the number of pixels in the image. It is the feature dimension of each pixel.

[0154] Step 3: Transformer Module

[0155] The self-attention mechanism is the core of the Transformer module. It determines the degree of attention each feature pays to other features by calculating the similarity between the query, key, and value of the input features. The specific steps are as follows:

[0156] Calculate the query, key, and value vectors:

[0157]

[0158] in, It is the input feature matrix. It is a learnable weight matrix. , , These are the query, key, and value vectors, respectively.

[0159] Calculating attention weights: The similarity between the query and the key is calculated using a dot product, and then the attention weights are obtained after a softmax operation.

[0160]

[0161] in, It is the dimension of the key vector. It is the dot product of the query and the key, measuring the similarity between features. Through The operation normalizes the weights, ensuring that the sum of the attention weights for all positions is 1.

[0162] Capturing wavelength dependencies: The Transformer module captures the interactions between different wavelengths by calculating the similarity (dot product) of different wavelengths, thereby modeling wavelength dependencies. The self-attention mechanism can process multi-wavelength image data, enabling the model to integrate information from various wavelengths and improve the quantitative estimation ability of absorption coefficient and blood oxygen saturation.

[0163] A multi-head attention mechanism is employed for parallel computation, enhancing the model's ability to model relationships between features at different wavelengths. Each attention head learns different relationships and patterns, strengthening the model's understanding of the data.

[0164]

[0165] in, It is One's attention, It is the output weight matrix.

[0166] Each layer in the Transformer module is followed by a feedforward neural network for further feature processing. The feedforward neural network typically consists of two fully connected layers.

[0167]

[0168] in, and It is a learnable weight matrix. and It is a bias term. Activation functions are used to introduce nonlinearity.

[0169] Step 3: Decoder Section

[0170] The goal of the decoder in this embodiment is to progressively restore the spatial resolution of the image. At each layer, the decoder first performs deconvolution, that is, through... Upsampling is performed using convolutional kernels and convolutions with a stride of 2. Then, features from the encoder are concatenated with feature maps from the decoder via skip connections, combining low-level and high-level features.

[0171]

[0172] By gradually upsampling, the image is eventually restored to its original size.

[0173] Step 4, Output Layer

[0174] The final step of the decoder is to map the recovered feature map through convolutional layers to the final target output dimension (such as the absorption coefficient and the sum). A precise estimate.

[0175]

[0176] in, This represents the output, specifically the output absorption coefficient. and blood oxygen saturation The two-dimensional distribution has 2 output channels.

[0177] In this embodiment, the model training uses mean squared error (MSE) as the loss function to optimize the absorption coefficient and The accuracy of the estimation.

[0178]

[0179] in, It is a predicted value. N is the true value, and N is the number of samples.

[0180] The encoder portion of this embodiment:

[0181] ① Feature Extraction Convolution: Three convolutional blocks are used to extract features from multi-wavelength data in sequence. Each convolutional block includes: a convolutional layer (kernel=3x3, stride=1, padding=1), batch normalization, activation function (ReLU), and a max pooling layer (MaxPooling, kernel=2x2, stride=2). The first convolutional block maps 21 channels of input to 64 channels, and the number of channels increases sequentially (128, 256). The output feature shape is (H / 8, W / 8, 256).

[0182] ② Feature vectorization: Flatten the two-dimensional feature map into a one-dimensional vector with the shape (N, P, D).

[0183] Where: N = batchsize; p = H / 8 * W / 8; number of patches in the image; D = 256: feature dimension;

[0184] The Transformer part of this embodiment:

[0185] ① Encoding position: Add positional encoding to preserve the spatial information of the image, using sine and cosine positional encoding methods;

[0186] ② Multi-head attention mechanism: Set the core parameters of Transformer: number of attention heads (num_heads) = 8; number of Transformer layers (num_layers) = 12; hidden layer size (hidden_size) = 256; calculate the feature dependencies and global correlations between wavelengths through the multi-head attention mechanism;

[0187] In this embodiment, the feedforward neural network (FFN) is followed by each Transformer layer, using a two-layer fully connected network.

[0188]

[0189] Among them, the dimensions of W1 and W2 are D*4D and 4D*D, respectively; and Indicates the bias term; Indicates input;

[0190] The decoder section of this embodiment:

[0191] ① Upsampling and fusion: The decoder structure upsamples the corresponding encoded features step by step until it is restored to the size of the original image;

[0192] ② Each upsampling stage includes: a deconvolution layer (transposed convolution, kernel=2x2, stride=2), a skip connection (concatenating feature maps with the corresponding layer of the encoder), a convolutional block (same structure as the encoder), and a final output dimension (H, W, 2), representing the two-dimensional distribution of absorption coefficients and sO2;

[0193] ③ Output layer: Use a separate convolutional layer to map the number of channels from high-dimensional features to the target output dimension (absorption coefficient and sO2);

[0194] Loss function design in this embodiment

[0195] (1). Multi-task loss: Define the joint loss function L:

[0196]

[0197] Where Lmse is the mean squared error, used to assess the difference between the predicted and the true values; α and β are weighting coefficients, with an initial value of 1; and absorption represents the absorption coefficient. Indicates blood oxygen saturation;

[0198] (2) Wavelength consistency: Introduce a wavelength regularization term to constrain the global consistency of multi-arm long data;

[0199] Training and optimization in this embodiment:

[0200] (1). Parameter settings:

[0201] ① Optimizer: Adam, learning rate 10 -4 And use a learning rate decay strategy; batch size=16; number of training rounds;

[0202] ② Data augmentation: The following augmentations are added during training: random cropping, rotation, horizontal flipping, and spectral interference (adding random noise to wavelength data to improve robustness);

[0203] Verification and testing of the model in this embodiment:

[0204] (1). Testing: Compare the model reconstruction with the actual distribution in the test set;

[0205] (2). Validation: The performance of the model in reconstructing the distribution of SO2 and absorption coefficient was evaluated using an independent validation set;

[0206] Example 3

[0207] This embodiment illustrates three representative cases, using line cross-sections of the inclusions at a wavelength of 800 nm, as detailed below:

[0208] First, examine the performance of the µa (absorption coefficient) estimation images in all test samples;

[0209] like Figure 4 As shown, the GT-φ (Monte Carlo luminous flux correction) method used in this embodiment may overestimate µa in the deep region of the test object, while the DL-Exp method can reconstruct a smoother and more uniform background.

[0210] Figure 4 The meanings are as follows:

[0211] Figure 4Image A: Optical absorption coefficient (µa) estimated by the linear calibration method (Cal.), with sample lines shown in yellow. This method generally yields µa estimates lower than reference values ​​in both background materials and inclusions.

[0212] Figure 4 Image B: µa image estimated based on the Monte Carlo flux correction method (GT-φ), with sample lines shown in green. This method has an excessively high bias in estimating µa in deep regions, and the edges of inclusions are distorted.

[0213] Figure 4 C in the image represents the estimated µa image by a deep learning model (DL-Sim) trained on simulated data, with sample lines indicated in blue. This model overestimates µa in the top region of the samples and produces artifacts in the coupling medium region (outside the samples).

[0214] Figure 4 In the image, D represents the µa image estimated by a deep learning model (DL-Exp) trained on experimental data, with sample lines indicated in red. This model generates clear boundaries at sample boundaries and the edges of inclusions, providing a smoother and more accurate µa estimate in space.

[0215] Figure 4 E in the figure shows a comparison of the sample line profiles using the four methods (Cal., GT-φ, DL-Sim, DL-Exp). The colors are consistent with those in Figures A to D, and are used to show the estimates of different methods at different depths.

[0216] like Figure 5 As shown, this embodiment uses a challenging test sample where a halo artifact exists around an inclusion. In this test, DL-Exp was able to reconstruct a relatively smooth and uniform background.

[0217] Figure 5 The meanings are as follows:

[0218] Figure 5 Figure A: The µa image estimated by the linear calibration (Cal.) method for the second test sample, with the sample line indicated in yellow. This sample is more complex than the sample in Figure A, containing more halo artifacts;

[0219] Figure 5 B in the image: The µa image of the second test sample based on Monte Carlo flux correction (GT-φ) estimation, with the sample lines shown in green. This method produces excessively high µa estimates at the edges of inclusions and in background regions, leading to distortion.

[0220] Figure 5C in the image represents the µa image estimated by a deep learning model (DL-Sim) trained on simulated data for the second test sample; the sample lines are indicated in blue. This method overestimates µa in the background region and exhibits artifacts, particularly in the coupling medium region.

[0221] Figure 5 In the diagram, D represents the µa image estimated by a deep learning model (DL-Exp) trained on experimental data for the second test sample; the sample lines are indicated in red. This method provides smooth background estimation and the estimates for most inclusions are close to the reference values.

[0222] Figure 5 E in the figure: Line profile of the second test sample, showing the µa estimates of the inclusion region by different methods. Red (DL-Exp) is close to the reference value and performs the best, while blue (DL-Sim) and green (GT-φ) have larger deviations.

[0223] like Figure 6 As shown, this embodiment uses a test sample that is not within the training data distribution range, and the geometry of its contents has not appeared in the training set. In this case, the µa estimation results of DL-Sim and DL-Exp methods show a highly uneven distribution inside the contents, but DL-Exp is slightly better than DL-Sim.

[0224] Figure 6 The meanings are as follows:

[0225] Figure 6 A in the image shows the µa image estimated by linear calibration (Cal.) for the third test sample, with the sample lines indicated in yellow. The geometry of this sample differs from the training data, and the distribution of inclusions is more complex.

[0226] Figure 6 B in the image: The µa image of the third test sample based on Monte Carlo flux correction (GT-φ) estimation, with the sample lines shown in green. This method overestimates µa in deep regions and distorts the boundaries of inclusions;

[0227] Figure 6 C in the image represents the µa image estimated by a deep learning model (DL-Sim) trained on simulated data for the third test sample. The sample lines are shown in blue. Because the geometry is not present in the training data, this method produces non-uniform µa estimates within the inclusions and introduces artifacts in certain regions.

[0228] Figure 6In the diagram, D represents the µa image estimated by a deep learning model (DL-Exp) trained on the experimental data for the third test sample; the sample lines are indicated in red. This method provides the smoothest and most accurate estimate, closely approximating the reference value.

[0229] Figure 6 E in the figure: Line profile of the third test sample, showing the µa estimates of the inclusion region by different methods. Red (DL-Exp) performs best among all methods, while blue (DL-Sim) and green (GT-φ) estimate µa in a less uniform manner and with larger errors.

[0230] Overall, the specific methods for comparison are as follows:

[0231] The Cal method (linear calibration) systematically underestimates the µa estimates for both inclusion and background regions;

[0232] The GT-φ method (based on Monte Carlo flux correction) shows excessively high estimates of µa in larger inclusions and in the central region of the background material, resulting in distortion at the edges of the inclusions.

[0233] The DL-Sim method (a deep learning model trained on simulated data) systematically overestimates µa in the top region of the sample.

[0234] The DL-Exp method (a deep learning model trained on experimental data) can generate smoother µa estimates and provide clearer boundaries at sample edges and inclusion boundaries.

[0235] The quantitative analysis results show that the deep learning method has significantly better accuracy in estimating µa in the inclusion region than other methods, and these results are consistent across all test samples.

[0236] Furthermore, the results show that the DL-Exp method outperforms DL-Sim in terms of spatial accuracy and quantitative estimation, highlighting the advantages of deep learning models trained on experimental data.

[0237] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A quantitative photoacoustic imaging method based on deep learning, characterized in that: Includes the following steps: S1. Photoacoustic data preparation: acquire simulation data and experimental data, wherein the photoacoustic data includes photoacoustic image data of different wavelengths; S2. Model training and test set partitioning: Divide the dataset into training dataset, validation dataset, and test dataset; S3. Construct a neural network model, which adopts the TransUNet model. A Transformer module is added to the U-Net network structure to capture long-distance dependencies in the dataset. S4. Model training: Train the TransUNet model using the training set; optimize the model hyperparameters. S5. Model Validation and Evaluation: Use the test set to validate the model and calculate model performance metrics. S6. Image processing is performed using the trained TransUNet model. The input photoacoustic image is used to estimate the absorption coefficient, and the processed image or absorption coefficient distribution map is output. The TransUNet model includes an input module, an encoder module, a Transformer module, a decoder module, and an output module. The multi-channel input module is used to input photoacoustic image data of different wavelengths in different channels; The encoder module includes multiple concatenated convolutional blocks for extracting local features from the input data; each convolutional block includes multiple convolutional layers, batch normalization layers, activation functions, and max pooling layers; features are extracted by progressively downsampling through each convolutional block. The Transformer module is used to capture global dependencies and the correlation between different wavelengths; The decoder module is used to progressively restore the resolution of the feature map and fuse low-level and high-level features; A two-dimensional distribution map representing the absorption coefficient and blood oxygen saturation was obtained; The output module is used to output the reconstructed quantitative photoacoustic image, including the distribution map of absorption coefficient and blood oxygen saturation.

2. The quantitative photoacoustic image method based on deep learning as described in claim 1, characterized in that: The Transformer module includes several Transformer layers. Each Transformer layer includes a multi-head attention mechanism module and a feedforward neural network. The multi-head attention mechanism module is used to calculate the feature dependencies and global correlations between wavelengths. The feedforward neural network is used to further process and transform features.

3. The quantitative photoacoustic image method based on deep learning as described in claim 1, characterized in that: The decoder module includes several serially connected upsampling blocks; used to implement multiple upsampling operations, each upsampling block is followed by a skip connection, and features are fused with the layer corresponding to the encoder to obtain a two-dimensional distribution map representing the absorption coefficient and blood oxygen saturation.

4. The quantitative photoacoustic image method based on deep learning as described in claim 1, characterized in that: The photoacoustic data in step S1 is preprocessed in the following manner: S11: Data format: Loaded as multi-channel data with shape (H,W,C), where H and W are the spatial dimensions of the image, and C represents the number of channels for different wavelengths; S12： Normalization: The data for each wavelength is normalized to the interval [0, 1].

5. The quantitative photoacoustic image method based on deep learning as described in claim 1, characterized in that: The specific steps of the Transformer module are as follows: Step 1: Calculate the query, key, and value. For each input feature, generate three vectors: query vector, key vector, and value vector. Step 2: Calculate attention weights: Obtain attention weights by calculating the similarity between the query and the key; Step 3: Cross-wavelength correlation: For multi-wavelength data, each wavelength corresponds to an input channel, and the correlation between different wavelengths is calculated through a self-attention mechanism.

6. The quantitative photoacoustic image method based on deep learning as described in claim 2, characterized in that: The multi-head attention mechanism module uses a multi-head attention mechanism for parallel computation, specifically according to the following formula: in, It is the first One point of attention, It is the output weight matrix.

7. The quantitative photoacoustic image method based on deep learning as described in claim 1, characterized in that: The decoder module outputs the absorption coefficient and according to the following formulas: : in, This represents the output, specifically the output absorption coefficient. and blood oxygen saturation The two-dimensional distribution.

8. The quantitative photoacoustic image method based on deep learning as described in claim 1, characterized in that: The loss function in the TransUNet model is calculated according to the following formula: Where L is the joint loss function, Lmse is the mean squared error used to evaluate the difference between the predicted and the true values, α and β are weighting coefficients, and absorption represents the absorption coefficient; This indicates blood oxygen saturation.

9. A quantitative photoacoustic imaging system based on deep learning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method described in any one of claims 1 to 8.

Images