A photoacoustic multi-modal segmentation method and system based on boundary information and a medium

Abstract

The application discloses a kind of based on boundary information's photoacoustic multimodal segmentation method, system and medium, it is related to photoacoustic imaging technical field, photoacoustic image is input to the photoacoustic multimodal segmentation network after training, obtain segmentation result;The construction process of photoacoustic multimodal segmentation network is as follows: construct CLIP signal encoding network and CLIP image encoding network, with "photoacoustic signal-boundary signal" data to fine-tune text encoder in CLIP signal encoding network, with "photoacoustic image-boundary image" data to fine-tune image encoder in CLIP image encoding network;Setting the image segmentation network of single mode, with "photoacoustic image-boundary image" data to pre-training image segmentation network;The decoder in the image encoder and text encoder after fine-tuning and pre-training image segmentation network are spliced and used as photoacoustic multimodal segmentation network;The segmentation method realizes high-precision photoacoustic image segmentation.

Description

Technical Field

[0001] This invention relates to the field of photoacoustic imaging technology, and in particular to a photoacoustic multimodal segmentation method, system, and medium based on boundary information. Background Technology

[0002] Photoacoustic computed tomography (PACT) is a novel non-invasive hybrid imaging technique based on the photoacoustic effect (PAE). It achieves optically contrastive ultrasound imaging by detecting the ultrasound signals generated when biological tissue absorbs pulsed laser light. This technique combines the deep penetration capability of ultrasound imaging (typically reaching several centimeters) with the high molecular-specific contrast of optical imaging, making it particularly suitable for high-resolution imaging of endogenous light absorbers such as hemoglobin and melanin. It has been widely applied in biomedical research fields such as molecular imaging and functional imaging. In PACT research, accurate image segmentation is a key technical step for achieving quantitative analysis: precise segmentation of specific structures such as vascular networks and organ boundaries can effectively extract key information from the image. Based on this, the technique can be further applied to important research areas such as precise localization and volume measurement of lesions, correction and compensation of light flux distribution, image reconstruction optimization under multi-velocity conditions, and dynamic monitoring and analysis of organ function.

[0003] In the early stages of photoacoustic imaging technology development, target region contour extraction mainly relied on manual segmentation. This method requires operators to have extensive professional experience in photoacoustic imaging and the ability to make subjective judgments and delineate contours based on image features. Although manual segmentation can obtain relatively accurate results, it is extremely time-consuming when dealing with complex photoacoustic images and large-scale photoacoustic image data, making it difficult to meet the needs of modern biomedical research for efficient and standardized analysis of massive photoacoustic images.

[0004] In 2016, Razansky et al. used an active contour model to segment photoacoustic images of phantoms and mice, and constructed a dual-chamber tissue model (representing the acoustic and optical parameters of the examined tissues respectively) based on the segmentation mask, thereby improving the accuracy of the image reconstruction algorithm. In 2019, Gao et al. systematically compared the segmentation performance of three models, namely support vector machine, AlexNet and GoogLeNet, on breast tumors, confirming that the combination of deep learning and photoacoustic imaging has important clinical value. In 2021, Razansky et al. used a convolutional neural network (CNN) model to segment ex vivo organs of mice. Compared with the active contour model, the CNN model improved the segmentation performance by about 40%.

[0005] As an important branch of medical imaging, the development of photoacoustic imaging segmentation methods is closely related to mainstream medical image segmentation techniques. Currently, the field of medical image segmentation mainly employs methods based on CNN generative models (such as UNet), Transformer-based architectures (such as Swin-Unet), and emerging diffusion probability models. These methods have achieved significant results in traditional medical imaging such as CT and MRI.

[0006] However, most existing medical image segmentation methods rely solely on the features of the reconstructed image itself for segmentation, failing to effectively utilize the most direct raw photoacoustic signal information in photoacoustic imaging. If these methods are directly applied to photoacoustic image segmentation, they often fail to fully explore the correspondence between the physical features of the signal layer and the tissue boundary, resulting in a significant decrease in segmentation accuracy when the tissue structure is complex or the boundary is blurred. Summary of the Invention

[0007] Based on the technical problems existing in the background technology, this invention proposes a photoacoustic multimodal segmentation method, system and medium based on boundary information, which realizes high-precision photoacoustic image segmentation.

[0008] This invention proposes a photoacoustic multimodal segmentation method based on boundary information, which inputs photoacoustic signals and photoacoustic images into a trained photoacoustic multimodal segmentation network to obtain segmentation results;

[0009] The construction process of the photoacoustic multimodal segmentation network is as follows:

[0010] The CLIP signal encoding network is constructed by connecting the decoder of the pre-trained signal autoencoder network to the text encoder of the CLIP model, and the CLIP image encoding network is constructed by connecting the decoder of the pre-trained image autoencoder network to the image encoder of the CLIP model.

[0011] Using "photoacoustic signal-boundary signal" as the input to the CLIP signal coding network and "photoacoustic image-boundary image" as the input to the CLIP image coding network, the text encoder in the CLIP signal coding network and the image encoder in the CLIP image coding network are jointly fine-tuned based on self-supervised learning and contrastive learning.

[0012] Set up a single-modal image segmentation network and use "photoacoustic image-boundary image" data to segment the pre-trained image network;

[0013] The fine-tuned text encoder, the fine-tuned image encoder, and the decoder from the pre-trained image segmentation network are fused to construct a photoacoustic multimodal segmentation network.

[0014] Furthermore, the pre-training process of the signal autoencoder network is as follows:

[0015] The photoacoustic signal-photoacoustic signal data pair is used as the input to the signal self-encoding network;

[0016] The photoacoustic image-photoacoustic image data pairs are used as input to the image autoencoder network;

[0017] Based on self-supervised learning and contrastive learning, signal autoencoder networks and image autoencoder networks are jointly pre-trained.

[0018] Furthermore, the boundary signal is generated from the boundary image through k-Wave sound field simulation. The sound field propagation simulation is performed using the boundary image as the initial sound pressure distribution to establish a supervisory mapping relationship from the signal domain to the boundary information.

[0019] Furthermore, the network structure of the image segmentation network is arranged from top to bottom as follows: In an image segmentation network, each layer has a symmetrical encoder-decoder structure. The output jumper of the layer encoder is connected to the first The input to the layer decoder, It is an integer.

[0020] Furthermore, the encoder in the image segmentation network is a downsampling connection, and the decoder in the image segmentation network is an upsampling connection;

[0021] The photoacoustic image is used as the input to the first encoder layer in the image segmentation network, and the real boundary image is used as the label for supervised training. The first decoder layer in the image segmentation network outputs the predicted boundary image.

[0022] Furthermore, the process of fusing the fine-tuned text encoder, the fine-tuned image encoder, and the decoder from the pre-trained image segmentation network to construct a photoacoustic multimodal segmentation network specifically involves:

[0023] The CLIP model's text encoder and image encoder are both divided into layers. The number of groups in the text encoder, the number of groups in the image encoder, and the number of layers in the decoder in the image segmentation network are all equal;

[0024] The first text encoder Group output and the first image encoder The group outputs are concatenated and then connected to the decoder in the image segmentation network. layer, It is an integer.

[0025] Furthermore, the training process of the photoacoustic multimodal segmentation network is as follows:

[0026] The original photoacoustic signal is used as input to the text encoder of the CLIP model to obtain the signal domain boundary features of each group;

[0027] The photoacoustic image is used as input to the image encoder of the CLIP model to obtain the image domain boundary features of each group;

[0028] The first Signal domain boundary features of the group and the first The image domain boundary features of the group are fused using an adaptive fusion weight mechanism, and after feature dimension alignment, they are connected to the first group in the image segmentation network. The input to the layer decoder;

[0029] An end-to-end optimization is performed using a collaborative training strategy with boundary augmentation, where training employs a combined loss function that includes segmentation loss, boundary augmentation loss, and consistency loss.

[0030] Furthermore, the combined loss function The formula is as follows:

[0031] ;

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] in, It is a segmentation loss. It is a boundary enhancement loss. It is a loss of consistency; It is the multi-class Dice loss, a loss function used in image segmentation tasks, primarily to measure the similarity between the predicted segmentation result and the true label. It is a multi-class cross-entropy loss; It represents the total number of segmentation categories (including background). It is a category index. , It is the first prediction of the photoacoustic multimodal segmentation network. Class segmentation probability graph It is the first manually labeled Class segmentation tags, It is a smoothing term used to prevent the denominator from being zero; It is a location The pixel at that location belongs to the first Class label (one-hot) encoding, It is a location The pixel at that location belongs to the first The predicted probability of a class; It is a location Does the pixel at this location belong to the first...? Class boundaries, It is a location Pixel belongs to the Predicted probability of class boundaries It is a weighting factor for the boundary region, imposing a higher penalty on boundary pixels; For position Is it in the first Within the class boundary region, For the first Class segmentation probability map at position Spatial gradient magnitude, For the first Class boundary prediction probability.

[0038] A computer system includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described above.

[0039] A computer-readable storage medium storing a plurality of classification programs, the plurality of classification programs being invoked by a processor to execute the method described above.

[0040] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0041] The advantages of the photoacoustic multimodal segmentation method, system, and medium based on boundary information provided by this invention are as follows: Addressing the technical bottleneck of existing photoacoustic image segmentation methods that rely solely on reconstructed images, resulting in low segmentation accuracy in blurred boundary regions, this invention proposes for the first time to use boundary information as the core supervisory signal to align the photoacoustic signal and the photoacoustic image in the semantic space of the boundary information, and completes end-to-end segmentation through dual-modal collaborative fusion. A progressive learning strategy is employed to construct the photoacoustic multimodal segmentation network, fully leveraging the model's multimodal feature extraction and collaborative capabilities. Specifically, by simulating the acoustic field of the target region contour in the photoacoustic image, the photoacoustic signal of the target region boundary is obtained, thereby establishing the relationship between the photoacoustic signal and image segmentation. Simultaneously, the image encoder and text encoder in the CLIP model are combined to extract features from the image and signal respectively, constructing a multimodal segmentation model and performing supervised training to improve the boundary recognition, semantic understanding, and generalization capabilities of photoacoustic image segmentation. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the process of the present invention;

[0043] Figure 2 These are schematic diagrams of autoencoder networks, where (a) is a schematic diagram of a signal autoencoder network and (b) is a schematic diagram of an image autoencoder network.

[0044] Figure 3 The diagrams show the fine-tuning frameworks for the CLIP signal coding network and the CLIP image coding network, where (a) is a diagram of the CLIP signal coding network fine-tuning framework and (b) is a diagram of the CLIP image coding network fine-tuning framework.

[0045] Figure 4 This is a schematic diagram of the structure of an image segmentation network;

[0046] Figure 5 A schematic diagram of the structure of PAMSNet, a photoacoustic multimodal segmentation network based on boundary information;

[0047] Figure 6 This is a schematic diagram of the pre-training process for signal autoencoder networks and image autoencoder networks;

[0048] Figure 7 A schematic diagram of the fine-tuning process for the CLIP signal coding network and the CLIP image coding network;

[0049] Figure 8 This is a schematic diagram of the pre-training process for a single-modal image segmentation network.

[0050] Figure 9 This is a schematic diagram of the training process for the photoacoustic multimodal segmentation network PAMSNet.

[0051] Figure 10This is a schematic diagram showing the comparison results between the photoacoustic multimodal segmentation network PAMSNet and the single-modal segmentation network. Detailed Implementation

[0052] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0053] like Figures 1 to 10 As shown, this invention proposes a photoacoustic multimodal segmentation method based on boundary information. The photoacoustic signal and photoacoustic image are input into a trained photoacoustic multimodal segmentation network (PAMSNet) to obtain the segmentation result. The construction process of the PAMSNet photoacoustic multimodal segmentation network is as follows:

[0054] Step 1: Connect the decoder of the pre-trained signal autoencoder network to the text encoder of the CLIP model to construct the CLIP signal encoding network; connect the decoder of the pre-trained image autoencoder network to the image encoder of the CLIP model to construct the CLIP image encoding network.

[0055] Step 2: Using "photoacoustic signal-boundary signal" as input to the CLIP signal coding network and "photoacoustic image-boundary image" as input to the CLIP image coding network, the text encoder in the CLIP signal coding network and the image encoder in the CLIP image coding network are jointly fine-tuned based on self-supervised learning and contrastive learning.

[0056] Step 3: Set up a single-modal image segmentation network and use the "photoacoustic image-boundary image" data to segment the pre-trained image network;

[0057] Step 4: Fuse the fine-tuned text encoder, the fine-tuned image encoder, and the decoder of the pre-trained image segmentation network to construct an opto-acoustic multimodal segmentation network.

[0058] This embodiment addresses the problems of inaccurate image segmentation and incomplete segmentation regions caused by low signal-to-noise ratio, blurred boundaries, and large field-of-view organization in photoacoustic images. It utilizes the characteristic that photoacoustic signals contain boundary signals of segmentation regions to construct a photoacoustic multimodal segmentation network. Simultaneously, to maximize the use of useful information in the signal and image, this embodiment employs a large multimodal model (CLIP model) and adopts a multi-step pre-training and fine-tuning strategy to achieve photoacoustic image segmentation with high accuracy.

[0059] That is, this embodiment obtains the photoacoustic signal of the target region boundary by simulating the sound field of the target region contour in the photoacoustic image, thereby establishing the relationship between the photoacoustic signal and image segmentation; at the same time, the image encoder and text encoder in the CLIP model are combined to extract features from the image and signal respectively, construct a multimodal segmentation model and perform supervised training, so as to improve the ability of photoacoustic image segmentation boundary recognition, semantic understanding and generalization.

[0060] In one embodiment, step one, after connecting the decoder of the pre-trained signal autoencoder network to the text encoder of the CLIP model to construct the CLIP signal encoding network, and after connecting the decoder of the pre-trained image autoencoder network to the image encoder of the CLIP model to construct the CLIP image encoding network, specifically involves:

[0061] To address the lack of a decoder in the CLIP model, which prevents direct supervised training using "photoacoustic signal-boundary signal" and "photoacoustic image-boundary image" data pairs, this embodiment first constructs two autoencoder network modules: a signal autoencoder network and an image autoencoder network, as follows: Figure 2 As shown, the signal autoencoder network in this embodiment is obtained by concatenating one-dimensional convolution and fully connected operations, while the image autoencoder network is constructed using two-dimensional convolution.

[0062] The pre-training process for the signal autoencoder network and the image autoencoder network in this embodiment is as follows:

[0063] The photoacoustic signal-photoacoustic signal data pair is used as the input to the signal self-encoding network;

[0064] The photoacoustic image-photoacoustic image data pairs are used as input to the image autoencoder network;

[0065] Based on self-supervised learning and contrastive learning, signal autoencoder networks and image autoencoder networks are jointly pre-trained.

[0066] In this embodiment, the data pairs "photoacoustic signal-photoacoustic signal" and "photoacoustic image-photoacoustic image" specifically involve using the photoacoustic signal as the input and label of a signal autoencoder network to guide the signal autoencoder network to generate a photoacoustic signal, and using the photoacoustic image as the input and label of an image autoencoder network to guide the image autoencoder network to generate a photoacoustic image.

[0067] It should be noted that in order for the decoders of the signal autoencoder network and the image autoencoder network to be concatenated with the CLIP model, the features output by the decoder of the signal autoencoder network must have the same feature dimension as the input feature dimension of the text encoder of the CLIP model, and the features output by the decoder of the image autoencoder network must have the same feature dimension as the input feature dimension of the image encoder of the CLIP model.

[0068] The signal autoencoder network and the image autoencoder network are pre-trained using self-supervised learning. Self-supervised learning enables the encoders in the signal autoencoder network and the image autoencoder network to establish effective feature encoding capabilities in their respective modal spaces. During the pre-training process, cross-modal contrastive learning is introduced to jointly pre-train the signal autoencoder network and the image autoencoder network, so as to initially narrow the distance between signal features and image features in the semantic space, reduce the modal gap, and lay the feature representation foundation for subsequent fine-tuning of the CLIP model using boundary information to achieve accurate cross-modal semantic alignment.

[0069] After pre-training the signal autoencoder network and the image autoencoder network, the decoder of the signal autoencoder network is connected to the CLIP text encoder to construct the CLIP signal encoding network. Similarly, the decoder of the image autoencoder network is connected to the CLIP image encoder to construct the CLIP image encoding network. Figure 3 As shown, the text encoder of the CLIP signal encoding network is fine-tuned using "photoacoustic signal-boundary signal" data, and the image encoder of the CLIP image encoding network is fine-tuned using "photoacoustic image-boundary image" data, enabling it to have preliminary professional encoding capabilities for photoacoustic images and photoacoustic signals, that is, fine-tuning the text encoder and image encoder of the CLIP model.

[0070] In one embodiment, step two involves using the "photoacoustic signal-boundary signal" as the input to the CLIP signal coding network and the "photoacoustic image-boundary image" as the input to the CLIP image coding network. Based on self-supervised learning and contrastive learning, the text encoder in the CLIP signal coding network and the image encoder in the CLIP image coding network are jointly fine-tuned. Specifically:

[0071] (a1) Perform boundary signal simulation on the labeled boundary image to obtain the boundary signal. The boundary signal is generated from the boundary image through k-Wave sound field simulation. The sound field propagation simulation is performed with the boundary image as the initial sound pressure distribution to establish a supervisory mapping relationship from the signal domain to the boundary information.

[0072] Edge extraction algorithms (such as the Canny edge detection algorithm) are applied to the labeled boundary image to extract the boundary information of the target region. This boundary information is then used to set the initial sound pressure distribution in the k-Wave toolbox. Based on this, and combined with the actual imaging conditions, key simulation elements such as the arrangement of the detector array and the physical parameters of the acoustic medium are accurately set to simulate photoacoustic signal propagation. Finally, a boundary signal can be generated, serving as the semantic guidance feature for subsequent CLIP signal coding network training.

[0073] (a2) The photoacoustic signal is used as the input of the CLIP signal coding network, the real boundary signal is used as the label, and the supervised loss is used to fine-tune the CLIP signal coding network;

[0074] (a3) The photoacoustic image is used as the input of the CLIP image coding network, and the real boundary image is used as the label. The supervised loss is used to fine-tune the CLIP image coding network.

[0075] (a4) The text encoder and image encoder of the CLIP model are jointly fine-tuned using contrastive loss.

[0076] According to steps (a1) to (a4), the text encoder and image encoder of the CLIP model are professionally fine-tuned based on the photoacoustic imaging features. At the same time, the contrast loss is used to jointly fine-tune the two modes of the text encoder and image encoder of the CLIP model, thereby fine-tuning to obtain a feature alignment relationship between the text encoder and image encoder of the CLIP model.

[0077] In one embodiment, step three involves setting up a single-modal image segmentation network, using "photoacoustic image-boundary image" data to train the pre-trained image segmentation network, specifically as follows:

[0078] After performing steps one and two, a CLIP model with preliminary professional coding capabilities for photoacoustic images and initial photoacoustic signals can be obtained. In one embodiment, the output of the CLIP model is a 1×512 dimensional vector, utilizing only... Figure 3 The network structure shown is insufficient to achieve high-precision photoacoustic image segmentation.

[0079] Therefore, this embodiment designs a dedicated single-modal image segmentation network. The network structure of the image segmentation network is arranged from top to bottom as follows: In an image segmentation network, each layer has a symmetrical encoder-decoder structure. The output jumper of the layer encoder is connected to the first The input to the layer decoder.

[0080] It is understandable that the encoder in the image segmentation network is a downsampling connection, and the decoder in the image segmentation network is an upsampling connection; the photoacoustic image is used as the input of the first layer encoder in the image segmentation network, the real boundary image is used as the label for supervised training, and the output of the first layer decoder in the image segmentation network is the predicted boundary image.

[0081] In addition, this embodiment introduces the first The output jumper of the layer encoder is connected to the first The input to the layer decoder is used to achieve multi-scale feature fusion, such as... Figure 4As shown. The main purpose of designing this U-shaped image segmentation network is to replace the connections in the image segmentation network with decoders. Figure 3 The CLIP model is followed by a pre-trained signal autoencoder network decoder, and a skip connection mechanism is used to connect the CLIP model with the decoder of the image segmentation network to achieve high-precision photoacoustic image segmentation.

[0082] This unimodal image segmentation network is trained under supervised supervision using photoacoustic image-boundary image data pairs. That is, it uses photoacoustic images as input and real boundary images as labels, enabling the network's decoder to reconstruct high-quality boundary images from encoded features. After training, the parameters of the decoder part of the image segmentation network are saved to prepare for the subsequent training of a photoacoustic multimodal segmentation network.

[0083] In one embodiment, step four, fusing the fine-tuned text encoder, the fine-tuned image encoder, and the decoder of the pre-trained image segmentation network to construct an opto-acoustic multimodal segmentation network, specifically involves:

[0084] After completing step three, a decoder capable of restoring encoded features to a high-quality boundary image (i.e., the decoder of the image segmentation network) is obtained. However, since the CLIP model can only output the final encoded result and not intermediate encoded features, this embodiment makes a slight modification to the CLIP encoder. Specifically, both the text encoder and image encoder of the CLIP model are divided into layers. The number of groups in the text encoder, the number of groups in the image encoder, and the number of layers in the decoder of the image segmentation network are all equal. The text encoder's first... Group output and the first image encoder The group outputs are concatenated and then connected to the decoder in the image segmentation network. layer, It is an integer.

[0085] It should be noted that the number of layers in the encoder-decoder structure of the image segmentation network is equal to or a multiple of the number of layers in the CLIP model. Both the text encoder and image encoder in the CLIP model are divided into layers. The number of groups in the text encoder, the number of groups in the image encoder, and the number of layers in the decoder in the image segmentation network are all equal. For example, in this embodiment, the text encoder and image encoder in the CLIP model both have 12 layers, and the encoder and decoder in the image segmentation network both have 6 layers. Due to the multiple setting between the number of layers, the number of layers of the text encoder and image encoder in the CLIP model can be grouped into groups of 2 layers each. The resulting group of 6 corresponds exactly to the 6-layer CLIP model (i.e., the text encoder and the image encoder), thereby achieving the goal of combining the decoder in the pre-trained image segmentation network with the fine-tuned CLIP model (including the text encoder and the image encoder) to achieve high-precision photoacoustic image segmentation.

[0086] The CLIP model, with its 12-layer text encoder and image encoder, is illustrated below. Specifically, the CLIP model's 12-layer encoder (each layer including both text and image encoders) outputs encoded features every two layers, corresponding to the 6-layer decoder in the image segmentation network. Since the output dimensions of the CLIP model's 12-layer encoder are identical, after concatenating and fusing the output image and signal features, a feature dimension transformation is required before they can be combined with the decoder of the image segmentation network obtained in step two via skip connections. Figure 5 As shown.

[0087] After constructing the photoacoustic multimodal segmentation network, it is trained using "photoacoustic signal + photoacoustic image" as input and real boundary images as labels; the training process of the photoacoustic multimodal segmentation network is as follows (b1) to (b4):

[0088] (b1) The original photoacoustic signal is used as the input to the text encoder of the CLIP model to obtain the signal domain boundary features of each group;

[0089] (b2) The photoacoustic image is used as the input to the image encoder of the CLIP model to obtain the image domain boundary features of each group;

[0090] (b3) will be the first Signal domain boundary features of the group and the first The image domain boundary features of the group are fused using an adaptive fusion weight mechanism, and after feature dimension alignment, they are connected to the first group in the image segmentation network. The input to the layer decoder;

[0091] (b4) End-to-end optimization is performed using a collaborative training strategy with boundary enhancement, wherein the training uses a combined loss function, which includes segmentation loss, boundary enhancement loss and consistency loss;

[0092] The segmentation loss optimizes the photoacoustic multimodal segmentation network using "photoacoustic signal and photoacoustic image-boundary image," with real boundary images as supervisory labels to guide the network in learning complete target region segmentation capabilities. The boundary enhancement loss also optimizes the network using "photoacoustic signal and photoacoustic image-boundary image," applying additional supervisory constraints to the boundary regions with real boundary images as supervisory labels, ensuring the network focuses on improving boundary accuracy. The consistency loss ensures consistency between the predicted and real boundary images in the boundary regions, avoiding conflicts between boundary prediction and overall segmentation. The synergistic optimization of these three loss mechanisms ensures the network accurately segments the overall target while significantly enhancing the segmentation accuracy of boundary regions, achieving a unification of global semantic understanding and local boundary refinement.

[0093] Combination loss function The formula is as follows:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] ;

[0100] in, It is a segmentation loss. It is a boundary enhancement loss. It is a loss of consistency; It is the multi-class Dice loss, a loss function used in image segmentation tasks. It primarily measures the similarity between the predicted segmentation result and the ground truth label (i.e., the predicted segmented image and the actual segmented image). It is a multi-class cross-entropy loss; It represents the total number of segmentation categories (including background). It is a category index. , It is the first prediction of the photoacoustic multimodal segmentation network. Class segmentation probability graph It is the first manually labeled Class segmentation tags, It is a smoothing term used to prevent the denominator from being zero; It is a location The pixel at that location belongs to the first Class label (one-hot) encoding, It is a location The pixel at that location belongs to the first The predicted probability of a class; It is a location Does the pixel at this location belong to the first...? Class boundaries, It is a location Pixel belongs to the Predicted probability of class boundaries It is a weighting factor for the boundary region, imposing a higher penalty on boundary pixels; For position Is it in the first Within the class boundary region, For the first Class segmentation probability map at position Spatial gradient magnitude, For the first Class boundary prediction probability.

[0101] Since the CLIP model fine-tuning based on boundary information in step two has achieved alignment of the text encoder and image encoder in the semantic space of boundary information, the fusion process in step four can achieve dynamic complementarity between the temporal boundary features of the signal domain and the spatial semantic features of the image domain in the boundary region, significantly improving the accuracy of boundary segmentation.

[0102] After completing the construction and training of the photoacoustic multimodal segmentation network, when performing photoacoustic image segmentation tasks, it is only necessary to utilize... Figure 5 By using the photoacoustic multimodal segmentation network shown and loading the stored final segmentation network weights, end-to-end photoacoustic image intelligent segmentation with cross-modal and high semantic understanding capabilities can be achieved.

[0103] Example 1;

[0104] This embodiment uses a cross-sectional photoacoustic image of a mouse abdominal kidney as the segmentation object, and provides a detailed introduction to the segmentation method proposed in this embodiment. It is also compared with existing methods UNet, Swin-UNet and DC-UNet. UNet is a deep learning model for image segmentation, named after its symmetrical "U"-shaped network structure. Swin-UNet is a pure Transformer image segmentation network that combines the Transformer architecture with the U-Net structure. DC-UNet is a deep learning network architecture for medical image segmentation.

[0105] The operating environment for this embodiment is shown in Table 1:

[0106] Table 1

[0107]

[0108] This embodiment contains 1000 sets of data, including one photoacoustic image, 512 photoacoustic signals, and one labeled boundary image. In order to quantitatively evaluate the performance of the segmentation algorithm proposed in this embodiment, the dataset is first divided into a training set and a validation set in an 8:2 ratio. The training set is used to train the parameters of the photoacoustic multimodal segmentation network, while the validation set is used to evaluate the segmentation performance.

[0109] The specific photoacoustic multimodal segmentation method in this embodiment is as follows: steps S1 to S5:

[0110] S1: Construction of a bimodal pairing dataset driven by boundary information;

[0111] Boundary information was extracted from manually annotated boundary images, and two paired datasets were constructed:

[0112] The first group is a paired dataset of "photoacoustic image-boundary image" to establish a supervised mapping relationship from the image domain to boundary information;

[0113] The second set is a paired dataset of "photoacoustic signal-boundary signal". The boundary signal is generated from the boundary image through k-Wave sound field simulation. The sound field propagation simulation is performed with the boundary image as the initial sound pressure distribution to establish a supervised mapping relationship from the signal domain to the boundary information.

[0114] The two sets of bimodal paired datasets mentioned above provide boundary information supervision for the subsequent fine-tuning of the CLIP model.

[0115] In this embodiment, the Canny edge detection algorithm is applied to the boundary images in the 900 training and validation sets to obtain boundary images containing only boundary information. Subsequently, the extracted boundary information is used to set the initial sound pressure distribution in the k-Wave toolbox, and the radius of the ring transducer is set to 40 mm, the number of transducer elements is set to 512, and then sound field simulation is performed to obtain the boundary signal.

[0116] S2: Cross-modal feature space initialization;

[0117] For the two heterogeneous modalities of photoacoustic signals and photoacoustic images, signal autoencoder networks and image autoencoder networks are constructed respectively. These networks are trained using "photoacoustic signal-photoacoustic signal" and "photoacoustic image-photoacoustic image" data. Figure 6 As shown.

[0118] By using a contrastive learning pre-training strategy, while performing intra-modal self-supervised reconstruction, a cross-modal contrastive loss is introduced to initially establish the correspondence between signal features and image features, laying the foundation for feature representation for subsequent cross-modal alignment.

[0119] Specifically, the signal autoencoder network and the image autoencoder network are pre-trained in a single mode using self-supervised loss. At the same time, the two modes of the signal autoencoder network and the image autoencoder network are jointly pre-trained using contrastive loss. The signal autoencoder network and the image autoencoder network trained in this way have a feature alignment relationship.

[0120] S3: Fine-tuning of the CLIP model based on boundary information;

[0121] A CLIP signal encoding network is constructed based on the text encoder of the signal autoencoder network and the CLIP model, and a CLIP image encoding network is constructed based on the image encoder of the image autoencoder network and the CLIP model. The CLIP signal encoding network and the CLIP image encoding network are trained using "photoacoustic signal-boundary signal" and "photoacoustic image-boundary image".

[0122] Based on the CLIP contrastive learning framework, the CLIP fine-tuning network is initialized using the encoder weights pre-trained in step S2, and the text encoder and image encoder of CLIP are fine-tuned using the bimodal pairing dataset constructed in step S1. Figure 7 As shown.

[0123] Contrastive Loss: Using boundary information as a common supervisory signal, the text encoder learns the boundary representation of the signal domain from the pairing of "photoacoustic signal-boundary signal", and the image encoder learns the boundary representation of the image domain from the pairing of "photoacoustic image-boundary image". By using contrastive learning loss to narrow the distance between the signal features and image features of the same sample in the semantic space, the text encoder and the image encoder are aligned in the semantic space of boundary information. This alignment process realizes the unified expression of the signal domain and the image domain in the common semantics of "boundary information".

[0124] S4: Segment decoder construction;

[0125] A six-layer symmetric encoder-decoder structure was constructed to provide a segmentation decoder for subsequent fusion with the CLIP model.

[0126] like Figure 8 As shown, photoacoustic images are used as input and real boundary images are used as labels for training. By jointly optimizing the segmentation loss and boundary enhancement loss, the decoder of the image segmentation network is made to have boundary awareness capability. After training, the decoder weights of the image segmentation network are retained and used to fuse with the CLIP model in step S5 to construct a complete photoacoustic multimodal segmentation network.

[0127] The segmentation loss used here differs from that used in the aforementioned photoacoustic multimodal segmentation network in that it involves an additional calculation step: using a "ray-based method" to determine which pixels are inside the boundary and which are outside. The "ray-based method" is a common approach for determining whether an image pixel is inside a boundary. It involves emitting a virtual ray horizontally (usually to the right) from the point to be determined and counting the number of intersections between this ray and the polygon boundary. If the number is odd, the pixel is inside; otherwise, it is outside.

[0128] The boundary enhancement loss here is consistent with the boundary enhancement loss used in the aforementioned photoacoustic multimodal segmentation network.

[0129] S5: Bimodal collaborative segmentation based on boundary information alignment;

[0130] The text encoder and image encoder of the CLIP model after fine-tuning in step S3 are used as feature extraction modules and fused into the decoder of the image segmentation network after pre-training in step S4 to construct the photoacoustic multimodal segmentation network PAMSNet, as shown below. Figure 5 As shown.

[0131] The text encoder and image encoder are each divided into 6 groups according to feature extraction levels, with the same number of layers as the decoder in the image segmentation network, and fusion connections corresponding to the levels are established; the photoacoustic signal is input into the text encoder, and at the 1st... The output layer of the group obtains the signal domain boundary features; the photoacoustic image input image encoder is reconstructed in the first... The output layer obtains image domain boundary features; .

[0132] The signal domain features and image domain features at the same level are fused using an adaptive fusion weighting mechanism, and the first... Group obtains signal domain boundary features and the first The group obtains the image domain boundary features, which are then aligned by feature dimensions and connected to the decoder of the image segmentation network. Layer input; end-to-end optimization is performed using a boundary-enhancing collaborative training strategy, such as... Figure 9 As shown, the training uses a combined loss function, which includes segmentation loss, boundary augmentation loss, and consistency loss.

[0133] After training, the photoacoustic multimodal segmentation network (PAMSNet) of this embodiment was tested using the remaining 200 sets of data and compared with the segmentation methods UNet, Swin-UNet, and DC-UNet. The differences between the segmentation method proposed in this embodiment and the UNet, Swin-UNet, and DC-UNet segmentation methods were quantitatively evaluated using the Dice and IOU metrics, as shown in Table 2. Furthermore, their segmentation quality was qualitatively compared using the segmentation results of four photoacoustic images, as shown in Table 2. Figure 10 As shown, Dice is the Dicecoefficient, which is the most commonly used index in medical image segmentation. It is a metric function used to evaluate the similarity between two samples, with a value ranging from 0 to 1. The larger the value, the more similar the samples are. IOU is the intersection-union ratio, which corresponds to the intersection-union ratio between the true boundary image and the predicted boundary image.

[0134] Table 2

[0135]

[0136] According to Table 2 and Figure 10 Compared to existing photoacoustic image segmentation methods, the photoacoustic multimodal segmentation network in this embodiment uses boundary information as the core guide, running through the entire process of dataset construction ("photoacoustic image-boundary image" and "photoacoustic signal-boundary signal"), cross-modal alignment (CLIP signal encoding network and CLIP image encoding network), and bimodal fusion (using the text encoder and image encoder of the CLIP model in the fine-tuned CLIP signal encoding network as feature extraction modules, fusing them into the pre-trained segmentation decoder to construct the photoacoustic multimodal segmentation network). Signal domain training data is generated from boundary information through k-Wave simulation, using boundary information as common supervision to achieve alignment between the signal domain and image domain in the boundary semantic space. Boundary enhancement loss is used in the fusion segmentation stage to strengthen boundary accuracy. This end-to-end boundary information guidance strategy, combined with multimodal fusion and progressive learning, enables the photoacoustic multimodal segmentation network in this embodiment to achieve Dice and IOU metrics of 0.99 and 0.96 respectively, an improvement of 6-8 percentage points compared to single-modal methods. This scheme is particularly effective in regions with blurred boundaries: the multi-step progressive guidance of boundary information ensures the accurate coordination between the temporal boundary features in the signal domain and the spatial semantic features in the image domain in the boundary region, significantly improving the boundary segmentation accuracy and verifying the superiority of the boundary information-driven multimodal segmentation mechanism.

[0137] In summary, this embodiment introduces target region boundary signal simulation into photoacoustic image segmentation for the first time, establishing a connection between photoacoustic signals and image segmentation; it uses dual-modal information of photoacoustic signals and images for photoacoustic image segmentation; it employs a multi-step pre-training and fine-tuning strategy to achieve high-precision photoacoustic image segmentation; and it modifies the CLIP model to allow it to directly skip connections with the decoder of the image segmentation network for multi-scale feature fusion. This effectively improves the accuracy and boundary awareness of photoacoustic image segmentation, achieving the goal of high-precision photoacoustic image segmentation.

[0138] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A photoacoustic multimodal segmentation method based on boundary information, characterized in that, The photoacoustic signal and photoacoustic image are input into the trained photoacoustic multimodal segmentation network to obtain the segmentation result; The construction process of the photoacoustic multimodal segmentation network is as follows: The CLIP signal encoding network is constructed by connecting the decoder of the pre-trained signal autoencoder network to the text encoder of the CLIP model, and the CLIP image encoding network is constructed by connecting the decoder of the pre-trained image autoencoder network to the image encoder of the CLIP model. Using "photoacoustic signal-boundary signal" as the input to the CLIP signal coding network and "photoacoustic image-boundary image" as the input to the CLIP image coding network, the text encoder in the CLIP signal coding network and the image encoder in the CLIP image coding network are jointly fine-tuned based on self-supervised learning and contrastive learning. Set up a single-modal image segmentation network and use "photoacoustic image-boundary image" data to segment the pre-trained image network; The fine-tuned text encoder, the fine-tuned image encoder, and the decoder from the pre-trained image segmentation network are fused to construct a photoacoustic multimodal segmentation network.

2. The photoacoustic multimodal segmentation method according to claim 1, characterized in that, The pre-training process of the signal autoencoder network is as follows: The "photoacoustic signal-photoacoustic signal" data pair is used as the input to the signal self-encoding network; The photoacoustic image-photoacoustic image data pairs are used as input to the image autoencoder network; Based on self-supervised learning and contrastive learning, signal autoencoder networks and image autoencoder networks are jointly pre-trained.

3. The photoacoustic multimodal segmentation method according to claim 1, characterized in that, The boundary signal is generated from the boundary image through k-Wave sound field simulation. The sound field propagation simulation is performed using the boundary image as the initial sound pressure distribution to establish a supervisory mapping relationship from the signal domain to the boundary information.

4. The photoacoustic multimodal segmentation method according to claim 1, characterized in that, The image segmentation network structure is arranged from top to bottom as follows: In an image segmentation network, each layer has a symmetrical encoder-decoder structure. The output of the layer encoder is connected to the first... The input to the layer decoder, It is an integer.

5. The photoacoustic multimodal segmentation method according to claim 4, characterized in that, The encoder in the image segmentation network is a downsampling connection, and the decoder in the image segmentation network is an upsampling connection. The photoacoustic image is used as the input to the first encoder layer in the image segmentation network, and the real boundary image is used as the label for supervised training. The first decoder layer in the image segmentation network outputs the predicted boundary image.

6. The photoacoustic multimodal segmentation method according to claim 1, characterized in that, The process of fusing the fine-tuned text encoder, the fine-tuned image encoder, and the decoder from the pre-trained image segmentation network to construct a photoacoustic multimodal segmentation network is as follows: The CLIP model's text encoder and image encoder are both divided into layers. The number of groups in the text encoder, the number of groups in the image encoder, and the number of layers in the decoder in the image segmentation network are all equal; The first text encoder Group output and the first image encoder The group outputs are concatenated and then connected to the decoder in the image segmentation network. layer, It is an integer.

7. The photoacoustic multimodal segmentation method according to claim 6, characterized in that, The training process of the photoacoustic multimodal segmentation network is as follows: The original photoacoustic signal is used as input to the text encoder of the CLIP model to obtain the signal domain boundary features of each group; The photoacoustic image is used as input to the image encoder of the CLIP model to obtain the image domain boundary features of each group; The first Signal domain boundary features of the group and the first The image domain boundary features of the group are fused using an adaptive fusion weight mechanism, and after feature dimension alignment, they are connected to the first group in the image segmentation network. The input to the layer decoder; An end-to-end optimization is performed using a collaborative training strategy with boundary augmentation, where training employs a combined loss function that includes segmentation loss, boundary augmentation loss, and consistency loss.

8. The photoacoustic multimodal segmentation method according to claim 7, characterized in that, The combined loss function The formula is as follows: ； ； ； ； ； ； in, It is a segmentation loss. It is a boundary enhancement loss. It is a loss of consistency; It is a multi-class Dice loss. It is a multi-class cross-entropy loss. It represents the total number of segmentation categories (including background). It is a category index. , It is the first prediction of the photoacoustic multimodal segmentation network. Class segmentation probability graph It is the first manually labeled Class segmentation tags, It is a smoothing term; It is a location The pixel at that location belongs to the first Class annotation encoding, It is a location The pixel at that location belongs to the first The predicted probability of a class; It is a location Does the pixel at this location belong to the first...? Class boundaries, It is a location Pixel belongs to the Predicted probability of class boundaries It is a weighting factor for the boundary region; For position Is it in the first Within the class boundary region, For the first Class segmentation probability map at position Spatial gradient magnitude, For the first Class boundary prediction probability.

9. A computer system comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of classification programs, which are used by a processor to execute the method as described in any one of claims 1-8.

Images