A photoacoustic image enhancement method and apparatus combining Mamba and CNN

Abstract

This disclosure provides a photoacoustic image enhancement method and apparatus combining Mamba and CNN, relating to the field of image processing technology. The photoacoustic image enhancement method of this disclosure, combining Mamba and CNN, is implemented through a hybrid dual-branch model, which includes a CNN, a Mamba U-shaped network, and a fusion module. The method may include: obtaining a first pixel-level organ category probability map of the photoacoustic image using the CNN; obtaining a second pixel-level organ category probability map of the photoacoustic image using the Mamba U-shaped network; and weightedly fusing the first pixel-level organ category probability map and the second pixel-level organ category probability map through the fusion module to obtain a third pixel-level organ category probability map, wherein the third pixel-level organ category probability map includes the probability value of each pixel in the photoacoustic image belonging to a certain organ category. This disclosure combines high-precision organ segmentation with low computational overhead.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a photoacoustic image enhancement method and apparatus that combines a structured spatial model (Mamba) with a convolutional neural network (CNN). Background Technology

[0002] Photoacoustic imaging (PAI) is a hybrid medical imaging modality based on the photoacoustic effect. It uses pulsed lasers to excite biological tissues to generate ultrasound signals and reconstruct images. This technology combines the high contrast of optical imaging with the deep tissue penetration capability of ultrasound imaging, making it valuable in fields such as brain functional imaging and early tumor detection.

[0003] Currently, organ segmentation in photoacoustic imaging is mainly achieved through models such as CNNs and Vision Transformers (ViTs). However, CNNs, limited by their local receptive field, struggle to model long-range spatial dependencies, leading to blurred organ boundary segmentation, especially in high-noise photoacoustic images where segmentation accuracy significantly decreases. While Vision Transformers can capture global context, their self-attention mechanism has a computational complexity that increases quadratically with the input image size (O(*n*²)), resulting in high computational overhead and low efficiency, limiting their application to high-resolution photoacoustic images. Therefore, a photoacoustic image enhancement scheme that combines high-precision segmentation performance with low computational overhead is urgently needed. Summary of the Invention

[0004] In view of this, this disclosure provides a photoacoustic image enhancement method and apparatus that combines Mamba and CNN.

[0005] According to a first aspect of this disclosure, a photoacoustic image enhancement method combining Mamba and CNN is provided. The method is implemented through a pre-trained hybrid dual-branch model, which includes a CNN, a Mamba U-shaped network, and a fusion module. The method comprises:

[0006] Using CNN to obtain the first pixel-level organ category probability map of photoacoustic images;

[0007] The second pixel-level organ category probability map of the photoacoustic image was obtained using a Mamba U-shaped network;

[0008] The first pixel-level organ category probability map and the second pixel-level organ category probability map are weighted and fused by the fusion module to obtain a third pixel-level organ category probability map, which includes the probability value of each pixel in the photoacoustic image belonging to each type of organ.

[0009] In some embodiments of the first aspect of this disclosure, the Mamba U-shaped network employs a symmetrical encoder-decoder structure; the step of using the Mamba U-shaped network to obtain the second pixel-level organ category probability map of the photoacoustic image includes:

[0010] An encoder process is performed on the photoacoustic image to obtain context compression features. The encoder process includes: segmenting the photoacoustic image to obtain a photoacoustic image block sequence, linearly transforming the photoacoustic image block sequence to obtain embedding features, and the embedding features are sequentially processed by multiple levels of coding units to obtain context compression features. Each level of coding unit processing includes at least two levels of visual state space module processing in series.

[0011] Based on the context compression features and the output features obtained by the coding units at each level, decoder processing is performed to obtain the second pixel-level organ category probability map. The decoder processing includes: multi-level decoding unit processing, upsampling the sharpened reconstruction features obtained by the multi-level decoding unit processing and performing linear projection processing. Each level of decoding unit processing includes: upsampling the context compression features or the sharpened reconstruction features output by the previous level decoding unit to obtain upsampled features, fusing the upsampled features with the output features of the symmetric hierarchical coding unit processing to obtain fused features, and performing two-level cascaded visual state space module processing on the fused features to obtain the sharpened reconstruction features of the current level decoding unit.

[0012] In some embodiments of the first aspect of this disclosure, each stage of the cascaded two-stage visual state space module processing includes: main path processing, which includes sequentially executed depthwise convolution, SS2D operator operations, and SiLU activation function processing; gated branch path processing, which includes fully connected layer processing for generating gated signals; gated multiplication and linear projection of the feature map obtained from the main path processing and the feature map obtained from the gated branch path processing; and an addition operation of the output feature map of the gated multiplication and linear projection with the residual.

[0013] In some embodiments of the first aspect of this disclosure, the step of weightedly fusing the first pixel-level organ category probability map and the second pixel-level organ category probability map through the fusion module to obtain a third pixel-level organ category probability map includes:

[0014] The fusion module performs a weighted summation of the first pixel-level organ category probability map and the second pixel-level organ category probability map based on the following formula to obtain a third pixel-level organ category probability map.

[0015]

[0016] Where P represents the probability map of organ category at the third pixel level. This represents the probability map of organ categories at the first pixel level. This represents the probability map of organ categories at the second pixel level, with the weight α ranging from [0.7, 0.8].

[0017] In some embodiments of the first aspect of this disclosure, the hybrid dual-branch model is trained based on a synthetic dataset, which is constructed as follows: pre-selected abdominal MRI images from the CHAOS dataset are acquired, the abdominal MRI images being labeled with organ segmentation masks; the abdominal MRI images are converted into simulated photoacoustic computed tomography (PAT) images; the organ segmentation masks of the abdominal MRI images are subjected to erosion skeletonization processing to generate graffiti labels along the organ central axis; and the graffiti labels are used to label the simulated PAT images to form a synthetic dataset.

[0018] In some embodiments of the first aspect of this disclosure, the hybrid two-branch model is trained by: pre-training based on the synthetic dataset to determine the parameters of the hybrid two-branch model; and fine-tuning the parameters of the hybrid two-branch model based on real data, wherein the real data includes mouse abdominal PAT data obtained by real photoacoustic scanning and labeled with organ segmentation masks.

[0019] In some embodiments of the first aspect of this disclosure, the hybrid dual-branch model is trained based on the following loss function:

[0020]

[0021]

[0022]

[0023]

[0024] in, Indicates the total loss. This indicates Dice's loss. This represents the cross-entropy loss of the graffiti supervision part of CNN. This represents the cross-entropy loss of the graffiti-supervised portion of the Mamba U-shaped network. Represents the set of labeled pixels. This represents the probability value of pixel i in category c. This represents the pixel-level organ category probability value predicted by model k. express In the annotation pixel, c represents the category index, and k represents the model branch identifier. This represents the pixel-level organ category probability map predicted by the CNN. This represents the pixel-level organ category probability map predicted by the Mamba U-shaped network. The pseudo-label represents the hybrid two-branch model, and Dice() represents the soft Dice coefficient between the predicted probability map and the pseudo-label. This represents the fusion weights of the CNN.

[0025] According to a second aspect of this disclosure, a photoacoustic image enhancement apparatus combining Mamba and CNN is provided, the photoacoustic image enhancement apparatus combining Mamba and CNN comprising:

[0026] The first segmentation unit is used to obtain the first pixel-level organ category probability map of the photoacoustic image using CNN;

[0027] The second segmentation unit is used to obtain the second pixel-level organ category probability map of the photoacoustic image using a Mamba U-shaped network;

[0028] The weighted fusion unit is used to weightedly fuse the first pixel-level organ category probability map and the second pixel-level organ category probability map to obtain a third pixel-level organ category probability map, wherein the third pixel-level organ category probability map includes the probability value of each pixel in the photoacoustic image belonging to each type of organ.

[0029] According to a third aspect of this disclosure, an electronic device is provided, comprising: one or more processors and a memory storing a program, the program including instructions that, when executed by the processor, cause the processor to perform the methods described above.

[0030] According to a fourth aspect of this disclosure, a computer-readable storage medium storing a program, the program including instructions that, when executed by one or more processors of a computing device, cause the computing device to perform the method described above.

[0031] As can be seen from the above technical solution, the embodiments of this disclosure use a hybrid dual-branch model including CNN and Mamba U-shaped network to achieve high-precision organ segmentation of photoacoustic images under weak supervision, which has both high-precision organ segmentation function and low computational overhead. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1A flowchart of a photoacoustic image enhancement method combining Mamba and CNN provided in embodiments of this disclosure;

[0034] Figure 2 This is a schematic diagram of the structure of the hybrid dual-branch model involved in the embodiments of this disclosure;

[0035] Figure 3 This is a schematic diagram of an abdominal MRI image and its corresponding simulated PAT image, organ mask, and graffiti label involved in an embodiment of this disclosure;

[0036] Figure 4 This is a schematic diagram of the structure of the Mamba U-shaped network involved in the embodiments of this disclosure;

[0037] Figure 5 This is a schematic diagram of the structure of the VSS Block according to an embodiment of this disclosure;

[0038] Figure 6 This is a schematic diagram illustrating the parameter sensitivity analysis of the weight α involved in the embodiments of this disclosure;

[0039] Figure 7 This is a diagram illustrating the comparison of segmentation effects on synthetic data;

[0040] Figure 8 This is a schematic diagram comparing the segmentation results of mouse PAT images;

[0041] Figure 9 A schematic diagram of the structure of the photoacoustic image enhancement device combining Mamba and CNN provided in the embodiments of this disclosure;

[0042] Figure 10 A schematic structural block diagram of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0043] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0044] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of this disclosure. The singular forms “a,” “the,” and “the” as used in the embodiments of this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0045] Depending on the context, words such as "if," "when," etc., used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrases "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0046] As mentioned above, the relevant technologies mainly use models such as CNN and visual transformers to achieve organ segmentation of photoacoustic images, which has problems such as blurred organ boundary segmentation, low segmentation accuracy, high computational overhead, low computational efficiency and limited use.

[0047] In addition, the related technologies also have the following problems:

[0048] 1) High annotation cost: Pixel-level annotations for photoacoustic imaging usually require manual drawing by professional physicians, which is extremely costly.

[0049] 2) Insufficient Adaptability of Weakly Supervised Strategies: Among various weakly supervised (WSL) strategies, traditional WSL methods such as Conditional Random Fields (CRF) and pseudo-labels struggle to effectively integrate multi-scale features and lack the ability to collaboratively model global context and local details, making them unsuitable for the aforementioned photoacoustic image organ segmentation model. Although scribble-based supervision is effective, the sparsity of scribble annotations limits their ability to capture object boundaries and complex shapes, often leading to inaccurate segmentation near edges and in blurred regions, resulting in a sharp drop in edge region segmentation accuracy. Furthermore, the incompleteness of scribble labels poses a challenge to supervised loss functions.

[0050] In view of this, embodiments of this disclosure provide the following photoacoustic image enhancement method and apparatus combining Mamba and CNN. This method employs a hybrid two-branch model containing both CNN and Mamba U-shaped networks to achieve high-precision organ segmentation of photoacoustic images under weak supervision, combining high-precision organ segmentation with low computational overhead. Furthermore, this hybrid two-branch model can also be trained using a graffiti-based weak supervision method, achieving both high organ edge region segmentation accuracy and low annotation cost.

[0051] Figure 1 This diagram illustrates a flowchart of a photoacoustic image enhancement method combining Mamba and CNN provided in an embodiment of this disclosure. See also... Figure 1 The method of this disclosure embodiment may include the following steps:

[0052] Step 101: Use CNN to obtain the first pixel-level organ category probability map of the photoacoustic image;

[0053] Step 102: Use the Mamba U-shaped network to obtain the second pixel-level organ category probability map of the photoacoustic image;

[0054] Step 103: The first pixel-level organ category probability map and the second pixel-level organ category probability map are weighted and fused by the fusion module to obtain the third pixel-level organ category probability map. The third pixel-level organ category probability map includes the probability value of each pixel in the photoacoustic image belonging to each type of organ.

[0055] The pixel-level organ category probability maps involved in this disclosure, namely the first pixel-level organ category probability map, the second pixel-level organ category probability map, and the third pixel-level organ category probability map, can be single-channel probability maps or multi-channel probability maps. In the multi-channel probability map, each channel corresponds to a semantic category (such as background, liver, etc.). For example, if C organ categories need to be segmented, the output is a C-channel probability map, with each channel storing the probability value of an organ category. Each channel is a matrix of the same size as the photoacoustic image, where each element value ∈ [0,1] represents the probability that the pixel belongs to the organ category corresponding to the current channel. For example, in multi-organ segmentation, channel 1 can represent the probability that each pixel belongs to "liver" (the higher the value, the more likely it is to be liver), channel 2 can represent the probability that each pixel belongs to "background", and channel 3 can represent the probability that each pixel belongs to "kidney".

[0056] The photoacoustic image enhancement method combining Mamba and CNN provided in this disclosure is implemented through a hybrid dual-branch model, which includes the aforementioned CNN, Mamba U-shaped network, and a fusion module for weighted fusion of the first pixel-level organ category probability map and the second pixel-level organ category probability map.

[0057] Figure 2 A schematic diagram illustrating the structure and processing procedure of the hybrid dual-branch model provided in this embodiment is shown. Figure 2 In the example, "(a)Visual Mamba-CNN" represents a custom name for the hybrid two-branch model, F cnn (X;θ) represents the CNN processing procedure, F mamba (X;θ) represents the processing procedure of the Mamba U-shaped network, where X represents the input photoacoustic image and θ represents the parameters. Both CNN and Mamba U-shaped networks can use an encoder-decoder structure. See [link to documentation]. Figure 2 The hybrid dual-branch model consists of two branches: a CNN and a Mamba U-shaped network. The fusion module can fuse the organ segmentation results Pred1 predicted by the CNN and Pred2 predicted by the Mamba U-shaped network to obtain the final organ segmentation result.

[0058] See Figure 2 The hybrid two-branch model can be trained using scribble labels. During training, pseudo-labels S can be used. pseudo Pseudo-label S pseudo The organ segmentation result Pred1 predicted by CNN and the organ segmentation result Pred2 predicted by Mamba U-shaped network can be obtained by a fusion model.

[0059] In some implementations, the hybrid two-branch model can be trained on a synthetic dataset, which can be constructed as follows: obtain pre-selected abdominal magnetic resonance imaging (MRI) images from the CHAOS dataset, the abdominal MRI images are labeled with organ segmentation masks, the abdominal MRI images are converted into simulated photoacoustic computed tomography (PAT) images, the organ segmentation masks of the abdominal MRI images are subjected to erosion skeletonization processing to generate graffiti labels along the central axis of the organs, and the simulated PAT images are labeled with graffiti labels to form a synthetic dataset.

[0060] Figure 3 The image shows an abdominal MRI image and its corresponding simulated PAT image, organ mask, and corresponding graffiti labels. Figure 3 In the middle, from left to right, are: abdominal MRI image, simulated PAT image converted from abdominal MRI image, organ mask of abdominal MRI image, and graffiti label.

[0061] In some examples, abdominal MRI images can be converted into simulated photoacoustic tomography (PAT) images using the K-Wave toolbox. Specifically, a computational grid is built, i.e., a uniform grid is constructed, a perfectly matched layer with 20 grid points is added to suppress boundary reflections, acoustic property parameters such as sound velocity and attenuation coefficient are assigned based on tissue type, 512 acoustic sensors (radius 4.5 cm) are distributed in a ring array, sound wave propagation is simulated using the K-Wave toolbox, and the PAT image is reconstructed using the Delayed Summation (DAS) algorithm.

[0062] The hybrid two-branch model can be obtained through two-stage training. Specifically, the training process of the hybrid two-branch model can include: pre-training based on synthetic datasets to determine the parameters of the hybrid two-branch model; and fine-tuning the parameters of the hybrid two-branch model based on real data, including mouse abdominal PAT data obtained through real photoacoustic scanning and labeled with organ segmentation masks. Thus, the strategy of pre-training on synthetic data + fine-tuning on real data can solve the problem of scarcity of real data, enabling the hybrid two-branch model to better meet the needs of specific segmentation tasks.

[0063] In practical applications, the synthetic dataset can be constructed using 16 abdominal MRI images with a resolution of 224×224 from the CHAOS dataset. Mouse abdominal PAT data can be obtained through 1064nm laser scanning with a 512-channel array. During training, the hardware configuration can be an NVIDIA RTX 3090 GPU, a batch size of 24, using the SGD optimizer, a momentum setting of 0.9, and a learning rate of 0.01.

[0064] The hybrid dual-branch model can be trained based on the loss functions shown in equations (1) to (3) below, and equation (4) shows the method of generating pseudo-labels.

[0065] (1)

[0066] (2)

[0067] (3)

[0068] (4)

[0069] in, Indicates the total loss. This indicates Dice's loss. This represents the cross-entropy loss of the graffiti supervision part of CNN. This represents the cross-entropy loss of the graffiti-supervised portion of the Mamba U-shaped network. Represents the set of labeled pixels. This represents the probability value of pixel i in category c. This represents the pixel-level organ category probability value predicted by model k. express In the annotation pixel, c represents the category index, and k represents the model branch identifier. This represents the pixel-level organ category probability map predicted by the CNN. This represents the pixel-level organ category probability map predicted by the Mamba U-shaped network. The pseudo-label represents the hybrid two-branch model, and Dice() represents the soft Dice coefficient between the predicted probability map and the pseudo-label. This represents the fusion weights of the CNN.

[0070] Therefore, the training of the hybrid two-branch model can be achieved by combining the Dice loss with the cross-entropy loss of the graffiti supervision part of each branch, thereby further improving the organ segmentation accuracy of the hybrid two-branch model.

[0071] The specific implementation of each step of the photoacoustic image enhancement method combining Mamba and CNN provided in the embodiments of this disclosure will be described in detail below.

[0072] In a hybrid two-branch model, a CNN can include a decoder and an encoder. The encoder is used to progressively extract local features and compress spatial dimensions through convolution and downsampling, while the decoder is used to recover spatial details and fuse multi-level features through upsampling and skip connections. In some examples, the CNN decoder may include 3×3 convolutional layers, ReLU activation function layers, and max-pooling layers; in others, it may include bilinear upsampling layers and skip connections. CNNs can preserve key local information such as organ edges during downsampling and restore organ structure during reconstruction.

[0073] In step 101, the specific implementation process of obtaining the first pixel-level organ category probability map of the photoacoustic image using CNN may include the following steps a1 to a5:

[0074] Step a1: Use a 3×3 convolutional layer to process the photoacoustic image to obtain local neighborhood features, which include features such as the edges and textures of the photoacoustic image.

[0075] Step a2: Use the RelU activation function layer to enhance the effective features in the local neighborhood features to obtain enhanced local features;

[0076] The ReLU activation function layer can preserve effective features, suppress noise, introduce nonlinearity, suppress negative responses, highlight salient features, enhance activation values ​​in low-contrast regions, and prevent details such as organ edges from being buried by noise.

[0077] Step a3: The local enhancement features are processed by a max pooling layer to further enhance the effective features in the local enhancement features, thereby obtaining compressed local features. The spatial size of the compressed local features is smaller than that of the local enhancement features, but the number of channels remains unchanged.

[0078] Max pooling (i.e., downsampling) layers can preserve the strongest edges and compress background noise, reducing feature map size while retaining the most salient features of local regions and improving computational efficiency. For example, max pooling layers can use a 2×2 window to avoid excessive compression that could cause details such as organ edges to disappear.

[0079] Step a4: Use a bilinear upsampling layer to perform bilinear upsampling on the compressed local features to obtain semantic features, which contain semantic context information of organs in the photoacoustic image;

[0080] Bilinear upsampling can magnify low-resolution compressed local features to their original size and recover their coarse spatial structure through interpolation. Bilinear upsampling is not only computationally efficient but also avoids checkerboard artifacts, making it suitable for smooth transition regions in photoacoustic images.

[0081] Step a5: The local enhancement features and semantic features are fused through a skip connection to obtain a first pixel-level organ category probability map. The first pixel-level organ category probability map contains pixel-level positional features of organ edges in the photoacoustic image and semantic context information of the organ.

[0082] Here, the fusion method can be, but is not limited to, channel concatenation, element-by-element addition, etc.

[0083] Local enhancement features preserve details such as the edges and textures of organs in photoacoustic images and can provide spatial details and local structures. Semantic features contain high-level semantic information such as target category and global context, which can provide semantic guidance and context awareness. By fusing local enhancement features and semantic features through skip connections, the first pixel-level organ category probability map can be obtained, and local features can be restored with high accuracy.

[0084] The Mamba U-shaped network adopts a symmetrical encoder-decoder structure. The encoder and decoder are connected through the Visual State Space Block (VSS Block). The VSS Block has a stronger ability to model long-range dependencies. By replacing the traditional CNN or Transformer with the VSS Block, the limitations of local receptive field can be overcome, the computational complexity can be reduced, and the ability to capture semantic associations in photoacoustic images can be enhanced, while maintaining hardware-friendly linear efficiency.

[0085] The encoder of a Mamba U-shaped network is responsible for feature extraction and downsampling, while the decoder is responsible for feature reconstruction and upsampling. Both the encoder and decoder in a Mamba U-shaped network can include multiple symmetrical layers, with skip connections connecting corresponding layers. For example, the encoder of a Mamba U-shaped network can include a preprocessing layer performing segmentation and linear transformation, and multiple layers of encoding units. Each encoding unit includes a block merging layer and a VSS layer, and each VSS layer includes two concatenated visual state space modules. Symmetrically, the decoder of a Mamba U-shaped network can include multiple layers of decoding units and post-processing layers performing upsampling and linear projection. Each decoding unit includes a block expansion layer and a VSS layer, and each VSS layer includes two concatenated visual state space modules.

[0086] Specifically, step 102 may include the following steps b1 and b2:

[0087] Step b1: Perform encoder processing on the photoacoustic image to obtain contextual compression features;

[0088] The encoder processing includes: segmenting the photoacoustic image to obtain a photoacoustic image block sequence, linearly transforming the photoacoustic image block sequence to obtain embedding features, and processing the embedding features sequentially through multiple coding units to obtain context compression features. Each coding unit processing includes at least two levels of cascaded visual state space module processing.

[0089] Step b2: Based on the variable context compression features and the output features obtained from the processing of each level of coding units in the encoder, perform decoder processing to obtain the second pixel-level organ category probability map.

[0090] The decoder processing includes multi-level decoding unit processing, upsampling the sharpened reconstruction features obtained from the multi-level decoding unit processing, and performing linear projection processing. Each level of decoding unit processing includes: upsampling the context compression features or the sharpened reconstruction features output by the previous level decoding unit to obtain upsampled features, fusing the upsampled features with the output features of the same level coding unit to obtain fused features, and performing two-level cascaded visual state space module processing on the fused features to obtain the sharpened reconstruction features of the current level decoding unit.

[0091] Figure 4 An exemplary structural diagram of a Mamba U-Net is shown. Figure 4 In the example, "(b)Mamba U-Net" represents a custom name for a Mamba U-Net, which employs a 4-stage symmetrical encoder-decoder architecture. See also Figure 4 The encoder and decoder each contain four levels. Skip connections directly connect the output features of the VSS Block×2 module of each level of the encoder to the input of the corresponding VSSBlock×2 module of the decoder. This allows richly detailed features from the encoder to be directly passed to the decoder, helping the decoder recover these details during feature reconstruction and compensating for spatial details lost during downsampling by the encoder. This enables accurate segmentation of organ boundaries in photoacoustic image segmentation.

[0092] See Figure 4The encoder and decoder each have corresponding VSS layers, which employ two cascaded VSS Blocks (VSS Block × 2). In the VSS layer, the first VSS Block models local spatial dependencies to obtain a base feature map containing organ contours and primary textures. The second VSS Block, based on the base feature map output by the first VSS Block, dynamically adjusts parameters through the selective scanning mechanism of SS2D to focus on key regions, obtaining a refined feature map containing cross-organ associations and global semantic enhancement. Furthermore, the VSS layer uses a VSS Block × 2 structure, with the output of the first VSS Block serving as the input to the second, forming a residual learning path. This mitigates gradient vanishing in deep networks and effectively improves model convergence stability.

[0093] See Figure 4 The specific implementation process of obtaining the second pixel-level organ category probability map of the photoacoustic image using the Mamba U-shaped network in step 102 can include the following steps c1 to c2:

[0094] Step c1: The photoacoustic image is processed by the encoder of the Mamba U-shaped network to obtain context compression features;

[0095] Specifically, with Figure 4 For example, the encoder processing flow of a Mamba U-shaped network may include the following steps c11~c14:

[0096] Step c11: Input a photoacoustic image. The photoacoustic image is segmented into fixed-size non-overlapping image patches using a patch partitioning layer to obtain a photoacoustic image patch sequence. Each image patch in the photoacoustic image patch sequence is projected onto a high-dimensional feature space using a linear embedding layer to obtain embedding features. Finally, the first VSS layer (i.e., ...) is applied. Figure 4 The encoder part performs feature transformation on the embedded features from top to bottom (first VSS Block × 2) to obtain context-aware features.

[0097] Converting photoacoustic images into block sequences preserves the original spatial topology and prevents organ microstructures from being smoothed out in subsequent convolutions. Linear transformation converts each image block (i.e., the photoacoustic signal of each small region in the photoacoustic image) into a feature vector containing abstract information. This not only enhances the expressive power of the features and improves the signal-to-noise ratio of weakly absorbing regions on organs, avoiding detail loss caused by early pooling in CNNs, but also reduces the dimensionality of subsequent processing.

[0098] For each image block in the photoacoustic image block sequence corresponding to the embedded features, VSS Block processing is performed twice to extract the photoacoustic feature block sequence. Each feature block in the photoacoustic feature block sequence corresponds to an image block in the photoacoustic image block sequence. Each feature block contains the edge texture features and semantic features of the photoacoustic signal in the corresponding region of the photoacoustic image. This photoacoustic feature block sequence is the aforementioned context-aware feature.

[0099] The VSS layer effectively captures long-range dependencies and local features. By stacking two VSS Block processes, multi-scale features can be extracted from the high-dimensional feature representation of photoacoustic image block sequences. Specifically, the first VSS Block process can extract edge texture features, while the second VSS Block process can extract semantic features such as vascular structures and tissue boundaries. Thus, it is possible to capture the continuity of organs and tissues (e.g., vascular networks) and the differences between different tissues in photoacoustic images.

[0100] VSS Block utilizes a state-space model (SSM) to model the long-range dependencies of photoacoustic image block sequences. Its computational complexity is linear, making it highly efficient. The specific implementation process of a single VSS Block processing step will be detailed below.

[0101] Step c12, through the first patch merging layer (i.e., Figure 4 The encoder part (from top to bottom, the first Patch Merging) performs a downsampling operation on the context-aware features to obtain the first downsampled features, which are then passed through the second VSS layer (i.e., Figure 4 The second VSS Block (×2) from top to bottom in the encoder part performs VSS Block processing twice on the first downsampled feature to obtain the first semantic enhancement feature;

[0102] Here, the downsampling operation in Patch Merging can include performing 2×2 neighbor block merging and channel compression on context-aware features. Downsampling reduces spatial resolution and increases feature dimensionality, compressing spatial dimensions while increasing feature richness to integrate information over a larger receptive field, thereby capturing more macroscopic structural information such as organ contours. Simultaneously, computational efficiency is also improved.

[0103] The double VSS Block processing in this step can model organ spatial constraints and capture global contextual information in photoacoustic images, such as the global location and structure of different organs.

[0104] Step c13, through the second merge layer (i.e., Figure 4The encoder part (from top to bottom, the second PatchMerging) performs a downsampling operation on the first semantic enhancement feature to obtain the second downsampled feature, which is then passed through the third VSS layer (i.e., Figure 4 The third VSS Block (×2) from top to bottom in the encoder part performs VSS Block processing twice on the second downsampled feature to obtain the second semantic enhancement feature;

[0105] Step c14, through the third merging layer (i.e., Figure 4 The encoder part (from top to bottom, the third PatchMerging) performs a final downsampling on the second semantic enhancement features to obtain the third downsampled features, which are then passed through the fourth VSS layer (i.e., Figure 4 The VSS Block (×2) connecting the encoder and decoder performs two superimposed VSS Block processing on the third downsampled feature to obtain the context compression feature of the global representation.

[0106] Ultimate downsampling through Patch Merging can condense the core semantics of organs. The superposition of two VSS Block processes on the ultimate downsampling features can capture the contextual information of the entire photoacoustic image, such as the spatial relationship between different organs, and establish cross-regional organ associations to obtain contextual compression features.

[0107] Step c2: Based on the context compression features obtained by the encoder of the Mamba U-shaped network and the output features of each VSS layer of the encoder, the decoder of the Mamba U-shaped network is executed to obtain the second pixel-level organ category probability map.

[0108] Specifically, with Figure 4 For example, the decoder processing flow of a Mamba U-type network may include the following steps c21~c25:

[0109] Step c21, through the first expansion layer (Patch Expanding) (i.e., Figure 4 The decoder section (from bottom to top, the first Patch Expanding) upsamples the context compression features to obtain the first upsampled feature, and then compares the first upsampled feature with the symmetrical VSS layer in the encoder (i.e., Figure 4 The first fused feature is obtained by fusing the output features of the third VSSB lock (×2) from top to bottom in the encoder part (i.e., the second semantic enhancement feature), and then passing it through the first VSS layer of the decoder (i.e., Figure 4 In the middle decoder section, the first VSS Block (×2) from bottom to top is processed twice to obtain the first sharpened reconstruction feature by performing VSS Block processing on the first fusion feature;

[0110] Upsampling operations in the block expansion layer can progressively restore the spatial dimensions of the image, bringing the resolution of the feature map close to that of the photoacoustic image. In photoacoustic image segmentation tasks, this processing can help reconstruct the fine shape of organs. The fusion of upsampled features with the output features of symmetric layers in the encoder can integrate multi-scale information through methods such as channel stitching. Performing a double-layered VSS block processing on the first fused feature can repair fracture structures and enhance boundary contrast.

[0111] Step c22, through the second extension layer (i.e., Figure 4 The decoder section (from bottom to top, the second PatchExpanding) upsamples the first sharpened reconstructed features to obtain the second upsampled features. These second upsampled features are then compared with the VSS layer at a symmetrical position in the encoder (i.e., Figure 4 The output features of the second VSS Block (×2) from top to bottom in the encoder (i.e., the first semantic enhancement features mentioned above) are fused to obtain the second fused feature, which is then passed through the second VSS layer of the decoder (i.e., Figure 4 In the middle decoder section, the second VSS Block (×2) from bottom to top is used to perform VSS Block processing twice on the second fusion feature to obtain the second sharpened reconstruction feature;

[0112] Step c23, through the third extension layer (i.e., Figure 4 The middle decoder section performs an upsampling operation on the second sharpened reconstructed features (from bottom to top, the third PatchExpanding) to obtain the third upsampled features. These third upsampled features are then compared with the VSS layer at the symmetrical position of the encoder (i.e., Figure 4 The third fused feature is obtained by fusing the output features of the first VSS Block (×2) from top to bottom in the encoder part (i.e., the context-aware features mentioned above), and then passed through the third VSS layer of the decoder (i.e., Figure 4 In the middle decoder section, the third VSS Block (×2) from bottom to top performs VSSBlock processing twice on the third fusion feature to obtain the third sharpened reconstruction feature;

[0113] By fusing the third upsampled feature with the context-aware features obtained from the intermediate layers of the encoder, the highest resolution details can be incorporated into the upsampled features, achieving detail enhancement. Performing VSSBlock processing twice on the third fused feature can achieve final boundary sharpening and noise suppression.

[0114] Step c24, through the fourth extension layer (i.e., Figure 4The fourth patch expansion of the decoder (from bottom to top) upsamples the third sharpened reconstruction feature to obtain the fourth upsampled feature. Linear projection is then performed on the fourth upsampled feature to map the features output by the decoder back to the pixel space of the photoacoustic image, so as to obtain the second pixel-level organ category probability map.

[0115] In this embodiment of the disclosure, during the VSS Block ×2 processing of each VSS layer in the decoder, skip connections are used to provide detailed information from the VSS layer output at the symmetrical position of the encoder and upsampled features obtained from the previous block expansion, gradually reconstructing spatial details and refining features. The fusion between the upsampled features in the decoder and the encoder's same-layer output features can be, but is not limited to, channel splicing.

[0116] In some examples, Patch Merging operations may include: merging adjacent 2×2 blocks → increasing the number of channels by 4 → compressing linear layers to twice their size (e.g., D → 2D) to achieve spatial dimensionality reduction (H / 2, W / 2), similar to convolutional downsampling but preserving block structure. Patch Expanding operates in the opposite way to Patch Merging; it expands (upsamples) the feature map patches, reducing the number of channels and increasing spatial resolution.

[0117] The Visual State Space Block (VSS Block) combines a state space model with the characteristics of visual data, dynamically capturing global contextual relationships in an image through a selective scanning mechanism while maintaining linear computational complexity.

[0118] Furthermore, each stage of the two-stage visual state space module processing includes: main path processing, which includes sequentially executed depthwise convolution, SS2D operator operations, and SiLU activation function processing; gated branch path processing, which includes fully connected layer processing for generating gated signals; gated multiplication and linear projection of the feature maps obtained from the main path processing and the gated branch path processing; and addition of the output feature map of the gated multiplication and linear projection with the residual.

[0119] Figure 5 A schematic diagram of a VSS Block is shown. See also Figure 5The VSS Block includes a depthwise convolution (DWCNN), an SS2D operator, and a SiLU activation component (not shown in the figure). The depthwise convolution is located in the left branch path, the SS2D operator is located in the center of the main path, and although the SiLU activation component is not directly shown in the figure, the multiplication operation (×) of the gated branch needs to be dynamically weighted through the SiLU activation component.

[0120] See Figure 5 The processing of a single VSS Block may include the following steps d1 to d10.

[0121] Step d1, Input features Figure X The shape is [H, W, C], where H represents the height, W is the width, and C is the number of channels;

[0122] Step d2: Layer Normalization (LN) is performed using the second feature map, with the shape remaining unchanged;

[0123] Specifically, the input X is subjected to layer normalization to make its mean 0 and variance 1, thereby eliminating the attenuation differences of photoacoustic signals in deep tissues and reducing the internal covariate shift.

[0124] Step d3, Linear Projection (Linear) to output the third feature map, which becomes [H, W, 2C];

[0125] Specifically, the expressive power of features can be enhanced by mapping the number of channels from C to a higher-dimensional space through linear transformation (e.g., C→2C).

[0126] Step d4, main branch processing, that is, performing depthwise convolution (DW CNN) and SS2D operator operations on the third feature map in sequence to obtain the fifth feature map. The shape of the fifth feature map is still [H, W, 2C], but the features at each position are incorporated with global context information;

[0127] The third feature map is processed by a depthwise convolution (DW CNN) to output a fourth feature map, which is then processed by the SS2D operator to obtain a fifth feature map. The depthwise convolution (DW CNN) processing includes processing the third feature map using depthwise convolution, which performs spatial convolution independently on each input channel, is lightweight and can extract local features. The depthwise convolution does not change the number of channels, and the shape of the fourth feature map remains [H, W, 2C]. The SS2D operator operation includes performing a 2D selective scan on the fourth feature map, that is, an operation in the state-space model (SSM), unfolding the 2D feature map into a 1D sequence, and applying the following equation (5) to model long-range dependencies, which can capture the global contextual relationships in the image.

[0128] Step d5, Gated Branch Processing, Dynamic Gated Generation: The third feature map is processed through a fully connected layer to generate a gated signal, and the sixth feature map is output, with the shape still being [H, W, 2C].

[0129] Here, the gated branch connects directly to the subsequent gated multiplication operation (×), without going through depthwise convolution and SS2D.

[0130] Step d6: Perform layer normalization (LN) and linear projection on the fifth feature map to obtain the seventh feature map, which still has the shape [H, W, 2C].

[0131] Here, layer normalization normalizes the feature map output by SS2D, and linear projection applies a fully connected layer again to perform feature transformation.

[0132] Step d7: Apply the SiLU activation function to the seventh feature map to output the eighth feature map. The shape of the eighth feature map remains unchanged, still [H, W, 2C].

[0133] The SiLU activation function combines linear and nonlinear properties, which can enhance key features.

[0134] Step d8, gated multiplication (×), output the ninth feature map, with shape [H, W, 2C];

[0135] Specifically, the eighth feature map obtained after SiLU activation is multiplied element-wise with the sixth feature map output from the gated branch in step d5. Through the gating mechanism, the weights generated by the gated branch are dynamically adjusted to the features of the main branch, emphasizing important features and suppressing noise or irrelevant features.

[0136] Step d9, Linear projection, outputs the tenth feature map, which becomes [H, W, C].

[0137] Here, the result of the gated multiplication, i.e., the ninth feature map, undergoes a linear transformation to adjust the number of channels (e.g., mapping from 2C back to C) to output the tenth feature map.

[0138] Step d10: Add the residuals (+) and output the eleventh feature map.

[0139] Specifically, the tenth feature map output from step d9 is compared with the original input feature map. Figure X The summation yields the eleventh feature map, with a shape of [H, W, C]. This eleventh feature map is the output feature of the VSS Block. Residual connections preserve the original features, avoiding the gradient vanishing or degradation problems found in deep networks, while allowing the features learned by the network to be used as residuals for optimization.

[0140] VSS Block captures local features through deep convolution, models global dependencies using SS2D, dynamically adjusts feature importance through a gating mechanism, and ensures gradient stability through residual connections. In this way, it can effectively model local-global feature relationships while maintaining low computational complexity, making it particularly suitable for processing multi-scale targets (such as organs) in photoacoustic images.

[0141] The VSS Block's main path incorporates the SS2D operator and short-path residual connections. By modeling long-range dependencies with linear complexity through SS2D, it avoids the high complexity of the self-attention mechanism in traditional Transformers. The gating mechanism enhances feature expressiveness and noise resistance, while the residual connections ensure network training stability. Therefore, in optical image segmentation tasks, it can balance local details and global context, significantly improving segmentation accuracy (such as Dice coefficients).

[0142] The SS2D operator in VSS Block achieves recursive global modeling through discrete state recursive equations. Combined with selective mechanisms and discretization transformations, it can dynamically adapt to the physical characteristics of photoacoustic images, surpassing the segmentation performance of traditional CNNs and Transformers while maintaining linear complexity.

[0143] The discrete state recursive equations of the SS2D operator can be expressed as equations (5) to (6).

[0144] (5)

[0145] , (6)

[0146] in, This represents a discrete state vector at time t, which is a hidden state used to encode information from historical inputs ( arrive Information extracted from photoacoustic images. It can represent organ structure information along the scan path up to the current position. This represents the discrete state transition matrix, used to control the previous state. How is this propagated to the current state? In photoacoustic imaging, This determines the degree of decay and retention of historical information. It is a discrete input matrix that can be used to control the current input. How does this affect state updates? In photoacoustic imaging, The influence weight of the current pixel on the state can be adjusted. This represents the input data at time t. In SS2D, the input data is the pixel values ​​(or feature values) along the scan path (row or column). For example, in a horizontal scan, It may represent the pixel value in the i-th row and j-th column of the image. This represents the output projection matrix, used to project the state vector. Mapping to output . This represents the output at time t.

[0147] Equation (5) simulates a dynamic system where the current state is determined by both the previous state and the current input. In photoacoustic image processing, recursively updating the state along the scanning path (e.g., from left to right, from top to bottom) can capture long-range dependencies. For example:

[0148] , , For input condition mapping, both are learnable projection functions used to map input features to the parameter space. For activation functions, used to ensure It is positive.

[0149] This represents the continuous input matrix at time t. This represents the continuous output matrix at time t. Represents the discretization step size at time t, a scalar or vector. This determines the granularity of discretization for a continuous system, which can be used to control the frequency or rate of state updates. In Mamba, matrices... , , and step length From input Dynamic generation enables SS2D operators to adjust their behavior based on the input content.

[0150] In photoacoustic images The state update speed can be adjusted based on the importance of the input pixels. For example, at organ boundaries (where feature changes are large), a larger state update speed can be generated. To update the state rapidly; in a homogeneous organization, smaller values ​​are generated. In order to maintain a stable state.

[0151] The above , , The defined selectivity mechanism enables the model to adapt to the local characteristics of photoacoustic images, thereby improving segmentation accuracy.

[0152] In order to discretize the continuous-time dynamics of the underlying layer, the state transition matrix can be calculated by the following equations (7) to (8).

[0153] (7)

[0154] (8)

[0155] Where exp is the matrix exponentiation operation, and I is the identity matrix. Find the inverse of a matrix. This represents a continuous state transition matrix used to capture long-range dependencies.

[0156] The transformations in equations (7) and (8) convert the continuous-time state-space model into a discrete-time model (i.e., recursive equations), making it applicable to image pixel sequences. The discretization process is influenced by... The impact is significant. It will make Faster decay, and smaller make The decay is slower.

[0157] In the SS2D operator, the aforementioned formulas (5) to (8) work together to achieve the following processing: 1) Selective parameter generation, that is, dynamic generation based on input features. , and step length 2) Discretization transformation: Calculate discrete parameters using Δ and a fixed A. , 3) Discrete State Recursion: Recursively calculate the state and output along the scanning path. In organ segmentation tasks of photoacoustic images, this mechanism enables the SS2D operator to adapt to image content (such as organ vs. background), efficiently model long-distance dependencies (such as organs running through the image), maintain linear computational complexity while modeling the global context, and encode the biophysical characteristics of photoacoustic images into differentiable mathematical processes through discrete state equations, achieving efficient and accurate global modeling.

[0158] In step 103, the first pixel-level organ category probability map and the second pixel-level organ category probability map can be weighted and summed based on the following formula (9) to obtain the third pixel-level organ category probability map.

[0159] (9)

[0160] Where P represents the probability map of organ category at the third pixel level. This represents the probability map of organ categories at the first pixel level. This represents the probability map of organ categories at the second pixel level, with the weight α ranging from [0.7, 0.8].

[0161] Figure 6 A schematic diagram showing the results of the parameter sensitivity analysis for weight α is provided. See also... Figure 6 The Dice coefficient is optimal when α∈[0.7,0.8], with a peak value of 0.652.

[0162] Figure 7 A schematic diagram showing the comparison of synthetic data segmentation effects is presented. Figure 7 In this document, Image represents a photoacoustic image, Ours represents the segmentation result of this embodiment, Ground Label represents the ground truth label of the photoacoustic image, and pCE+UNet, USTM+Unet, Mumford+Unet, pCE+SwinUNet+UNet, USTM+SwinUNet, Mumford+SwinUNet, and Gated CRF+SwinUNet represent the segmentation results of the corresponding models. Figure 7 It can be seen that the method (Ours) of this disclosure embodiment is more accurate in segmenting the boundaries of the liver (yellow) and kidney (blue) in the photoacoustic image.

[0163] Table 1 below shows a comparison of the quantitative performance of the method of this disclosure embodiment with organ segmentation methods based on traditional SwinUNet and Gated CRF.

[0164] Table 1

[0165]

[0166] As can be seen from the above, the method of this disclosure embodiment can achieve at least the following technical effects:

[0167] 1) Segmentation accuracy is effectively improved: The embodiments of this disclosure improve the Dice coefficient by 8.5% and reduce HD95 by 36.7% through global dependency modeling (Mamba) + local detail preservation (CNN), which fully demonstrates that the organ boundary segmentation results obtained by the method of the embodiments of this disclosure are more accurate.

[0168] 2) High computational efficiency: Mamba's linear complexity (O(n)) is less than Transformer's complexity O(n²), and training time is only 4 hours per model.

[0169] 3) The cost of annotation is effectively reduced: the time required for graffiti annotation is only 10% of that for pixel-level annotation. Pre-training with synthetic data can effectively reduce the reliance on real annotations.

[0170] 4) Enhanced generalization: The training strategy of pre-training with synthetic data and fine-tuning with real data can solve the problem of scarce training data and reduce training costs.

[0171] Figure 8 A schematic diagram of the segmentation results of mouse PAT data is shown. Figure 8In this document, Image represents a mouse PAT image, Ours represents the segmentation result of the mouse PAT image according to the embodiments of this disclosure, Ground Label represents the ground truth label of the mouse PAT image, Scribble Label represents the scribbled label of the mouse PAT image, and pCE+UNet, USTM+Unet, Mumford+Unet, pCE+SwinUNet+UNet, USTM+SwinUNet, and Gated CRF+SwinUNet represent the segmentation results of the corresponding models. See also Figure 8 The method of this disclosure can accurately segment the outlines of the liver, kidney, and spleen in mouse PAT images (red box area).

[0172] Figure 9 A schematic diagram of a photoacoustic image enhancement device combining Mamba and CNN, provided in an embodiment of this disclosure, is shown. This photoacoustic image enhancement device combining Mamba and CNN is applied to an electronic device 1000. See also Figure 9 The photoacoustic image enhancement device 900 combining Mamba and CNN may include:

[0173] The first segmentation unit 901 is used to obtain the first pixel-level organ category probability map of the photoacoustic image using CNN;

[0174] The second segmentation unit 902 is used to obtain the second pixel-level organ category probability map of the photoacoustic image using a Mamba U-shaped network;

[0175] The weighted fusion unit 903 is used to weightedly fuse the first pixel-level organ category probability map and the second pixel-level organ category probability map to obtain a third pixel-level organ category probability map, wherein the third pixel-level organ category probability map includes the probability value of each pixel in the photoacoustic image belonging to each type of organ.

[0176] Further, the second segmentation unit 902 can be specifically used to: perform encoder processing on the photoacoustic image to obtain context compression features, the encoder processing including: segmenting the photoacoustic image to obtain a photoacoustic image block sequence, linearly transforming the photoacoustic image block sequence to obtain embedding features, the embedding features being processed sequentially by a multi-level coding unit to obtain context compression features, each coding unit processing including at least two levels of cascaded visual state space module processing; and performing decoder processing based on the context compression features and the output features obtained by each level of the coding unit processing to obtain the second pixel-level organ category probability map, the decoder processing including: multi-level decoding unit processing, upsampling the sharpened reconstruction features obtained by the multi-level decoding unit processing and performing linear projection processing, each decoding unit processing including: upsampling the context compression features or the sharpened reconstruction features output by the previous level decoding unit to obtain upsampled features, fusing the upsampled features with the output features of the symmetric hierarchical coding unit processing to obtain fused features, and performing two levels of cascaded visual state space module processing on the fused features to obtain the sharpened reconstruction features of the current level decoding unit.

[0177] Furthermore, the weighted fusion unit 903 can be used to perform a weighted summation of the first pixel-level organ category probability map and the second pixel-level organ category probability map based on the aforementioned equation (9) to obtain the third pixel-level organ category probability map.

[0178] Further technical details regarding the photoacoustic image enhancement device 900 combining Mamba and CNN can be found in the preceding section on photoacoustic image enhancement methods combining Mamba and CNN, and will not be repeated here. In specific applications, the photoacoustic image enhancement device 900 combining Mamba and CNN can be implemented by the electronic device 1000 described below, or it can be implemented as software within the electronic device 1000.

[0179] Additionally, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that includes instructions, which, when executed by one or more processors of a computing device, perform the steps of the aforementioned photoacoustic image enhancement method combining Mamba and CNN.

[0180] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. See also... Figure 10 The electronic device 1000 may include one or more processors 1001, and a memory 1002 storing one or more programs, which are executed by the one or more processors 1001 to implement the method flow and / or program units corresponding to each unit in the apparatus shown in the above embodiments of this disclosure.

[0181] The various components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. Processor 1001 can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a user interface on an external input / output device (such as a display device coupled to an interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storage devices, if desired.

[0182] Processor 1001 may include one or more single-core or multi-core processors. Processor 1001 may include any combination of general-purpose processors or special-purpose processors (such as graphics processors, application processors, baseband processors, etc.).

[0183] Memory 1002 is the computer-readable storage medium provided in this disclosure, which can be used to store non-transitory software programs, non-transitory computer-executable programs, and units, such as those in the embodiments of this disclosure. Figure 1 The program instructions / units corresponding to the photoacoustic image enhancement method combining Mamba and CNN are shown. Processor 1001 executes methods such as those described in the above embodiments by running non-transient software programs, instructions, and units stored in memory 1002. Figure 1 The program, instructions, and units corresponding to the photoacoustic image enhancement method combining Mamba and CNN are shown.

[0184] The electronic device 1000 may further include an input device 1003 and an output device 1004. The processor 1001, memory 1002, input device 1003, and output device 1004 can be connected via a bus or other means. Figure 10 Taking the example of a connection between China and Israel via a bus.

[0185] The aforementioned programs (also known as software, software applications, or code) include machine instructions for a programmable processor and can be implemented using object-oriented programming languages, assembly language, or machine language.

[0186] With the development of time and technology, the meaning of "medium" has become increasingly broad. The dissemination of computer programs is no longer limited to tangible media; they can also be downloaded directly from the network. Any combination of one or more computer-readable storage media can be used. Computer-readable storage media can be, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or apparatus.

[0187] The technical solutions provided in this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this disclosure. Furthermore, those skilled in the art will recognize that, based on the ideas of this disclosure, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

[0188] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications or equivalent substitutions made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A photoacoustic image enhancement method combining Mamba and CNN, characterized in that, The method is implemented by a pre-trained hybrid dual-branch model, the hybrid dual-branch model comprising a CNN, a Mamba U-shaped network and a fusion module; the method comprises: obtaining a first pixel-level organ class probability map of the photoacoustic image by using the CNN; obtaining a second pixel-level organ class probability map of the photoacoustic image by using the Mamba U-shaped network; obtaining a third pixel-level organ class probability map by weightedly fusing the first pixel-level organ class probability map and the second pixel-level organ class probability map through the fusion module, the third pixel-level organ class probability map comprising probability values of each pixel in the photoacoustic image belonging to each organ class; wherein the Mamba U-shaped network adopts a symmetrical encoder-decoder structure, and the encoder and the decoder are connected through a visual state space module; the encoder of the Mamba U-shaped network comprises a preprocessing level performing segmentation and linear transformation and a plurality of levels of encoding units, each encoding unit comprising a block merging layer and a VSS level, and each VSS level comprising two levels of visual state space modules connected in series; symmetrically, the decoder of the Mamba U-shaped network comprises a plurality of levels of decoding units and a post-processing level performing up-sampling and linear projection, each decoding unit comprising a block expansion layer and a VSS level, and each VSS level comprising two levels of visual state space modules connected in series; in each VSS level, the first visual state space module is used for modeling local spatial dependence to obtain a basic feature map containing organ contours and primary textures, and the second visual state space module is used for dynamically adjusting parameter focusing key areas based on the basic feature map output by the first visual state space module through a selective scanning mechanism of SS2D to obtain a refined feature map containing cross-organ correlation and global semantic enhancement; wherein the obtaining of the second pixel-level organ class probability map of the photoacoustic image by using the Mamba U-shaped network comprises: performing encoder processing on the photoacoustic image to obtain context compressed features, the encoder processing comprising: segmenting the photoacoustic image to obtain a photoacoustic image block sequence, linearly transforming the photoacoustic image block sequence to obtain embedded features, and sequentially processing the embedded features through a plurality of levels of encoding units to obtain the context compressed features, each level of encoding unit processing comprising at least two levels of visual state space module processing connected in series; performing decoder processing based on the context compressed features and output features obtained by each level of the encoding unit processing to obtain the second pixel-level organ class probability map, the decoder processing comprising: a plurality of levels of decoding unit processing, up-sampling sharpened reconstruction features obtained by the plurality of levels of decoding unit processing and performing linear projection processing, each level of decoding unit processing comprising: up-sampling the context compressed features or the sharpened reconstruction features output by the previous level of decoding unit to obtain up-sampled features, fusing the up-sampled features with output features of a symmetric level of encoding unit processing to obtain fusion features, and performing two levels of visual state space module processing on the fusion features to obtain the sharpened reconstruction features of the current level of decoding unit.

2. The method of claim 1, wherein, Each of the two-stage visual state space module processes in the series includes: a main path process including sequentially executed deep convolution operations, SS2D operator operations, and SiLU activation function processes; a gated branch path process including a fully connected layer process for generating a gating signal; gating multiplication of a feature map obtained for the main path process and a feature map obtained for the gated branch path process, and linear projection; and an output feature map of the gating multiplication and linear projection and a residual addition operation.

3. The method of claim 1, wherein, The weighting and fusing of the first pixel-level organ class probability map and the second pixel-level organ class probability map by the fusion module to obtain a third pixel-level organ class probability map includes: The fusion module performs weighted summation on the first pixel-level organ class probability map and the second pixel-level organ class probability map based on the following formula to obtain a third pixel-level organ class probability map: wherein P represents a third pixel-level organ class probability map, represents a first pixel-level organ class probability map, represents a second pixel-level organ class probability map, and the weight a has a value range of [0.7, 0.8].

4. The method of claim 1, wherein, The hybrid dual-branch model is trained based on a synthetic data set, and the synthetic data set is constructed in the following manner: Obtain pre-selected abdominal MRI images in the CHAOS data set, and the abdominal MRI images are labeled with organ segmentation masks; Convert the abdominal MRI images into simulated photoacoustic tomography (PAT) images, perform skeletonization processing on the organ segmentation masks of the abdominal MRI images to generate scribble labels along the organ center axis, and label the simulated PAT images using the scribble labels to form a synthetic data set.

5. The method of claim 4, wherein, The hybrid dual-branch model is trained in the following manner: Pre-train based on the synthetic data set to determine the parameters of the hybrid dual-branch model; Fine-tune the parameters of the hybrid dual-branch model based on real data, and the real data includes mouse abdominal PAT data obtained by real photoacoustic scanning and labeled with organ segmentation masks.

6. A photoacoustic image enhancement device combining Mamba with CNN, characterized by, The photoacoustic image enhancement device combining Mamba and CNN includes: A first segmentation unit for obtaining a first pixel-level organ class probability map of a photoacoustic image using a CNN; A second segmentation unit for obtaining a second pixel-level organ class probability map of the photoacoustic image using a Mamba U-shaped network; The Mamba U-shaped network adopts a symmetrical encoder-decoder structure, and the encoder and the decoder are connected through a visual state space module; the encoder of the Mamba U-shaped network comprises a preprocessing level performing segmentation and linear transformation and a plurality of encoding units, each encoding unit comprises a block merging layer and a VSS level, and each VSS level comprises two visual state space modules connected in series; symmetrically, the decoder of the Mamba U-shaped network comprises a plurality of decoding units and a post-processing level performing up-sampling and linear projection, each decoding unit comprises a block expansion layer and a VSS level, and each VSS level comprises two visual state space modules connected in series; in each VSS level, the first visual state space module is used for modeling local spatial dependence to obtain a basic feature map containing organ contour and primary texture, and the second visual state space module is used for dynamically adjusting parameter focusing key areas based on the basic feature map output by the first visual state space module through a selective scanning mechanism of SS2D to obtain a refined feature map containing cross-organ correlation and global semantic enhancement; The Mamba U-shaped network adopts a symmetrical encoder-decoder structure, and the encoder and the decoder are connected through a visual state space module; the encoder of the Mamba U-shaped network comprises a preprocessing level performing segmentation and linear transformation and a plurality of encoding units, each encoding unit comprises a block merging layer and a VSS level, and each VSS level comprises two visual state space modules connected in series; symmetrically, the decoder of the Mamba U-shaped network comprises a plurality of decoding units and a post-processing level performing up-sampling and linear projection, each decoding unit comprises a block expansion layer and a VSS level, and each VSS level comprises two visual state space modules connected in series; in each VSS level, the first visual state space module is used for modeling local spatial dependence to obtain a basic feature map containing organ contour and primary texture, and the second visual state space module is used for dynamically adjusting parameter focusing key areas based on the basic feature map output by the first visual state space module through a selective scanning mechanism of SS2D to obtain a refined feature map containing cross-organ correlation and global semantic enhancement; The weighting fusion unit is configured to weight and fuse the first pixel-level organ class probability map and the second pixel-level organ class probability map to obtain a third pixel-level organ class probability map, and the third pixel-level organ class probability map comprises probability values of each pixel in the photoacoustic image belonging to each organ class.

7. An electronic device, comprising: The program comprises instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 1-5. The program comprises instructions which, when executed by the processor, cause the processor to perform the method of any one of claims 1-5.

8. A computer readable storage medium having stored thereon a program, the program comprising instructions, which when executed by one or more processors of a computing device, cause the computing device to perform the method of any one of claims 1-5.

Images