Interactive medical image segmentation method and system based on robust sequence prompt optimization

Abstract

The application discloses an interactive medical image segmentation method and system based on robust sequence prompt optimization, and belongs to the technical field of medical image segmentation and artificial intelligence interaction. In view of the problems that the prior art is sensitive to prompt selection and poor in adaptability to expert annotation uncertainty, a segmentation interaction process is modeled as a Markov decision process, an optimal prompt action is selected by analyzing a current image, a historical mask and a prompt sequence through a double-branch prompt fusion strategy network, a robust reward function is introduced, the function fuses a certainty reward based on a gold standard annotation and an uncertainty reward generated based on a PCA boundary disturbance technology, anisotropic noise is injected in a principal component space to simulate expert annotation differences, and thus the model is trained to maintain stable performance in a labeled uncertain environment. The application significantly improves the precision, robustness and interaction efficiency of medical image segmentation, has excellent zero-shot generalization capability, and can be widely applied to auxiliary diagnosis and segmentation tasks of multi-modal medical images.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image segmentation and artificial intelligence interaction, in particular to an interactive medical image segmentation method and system based on robust sequence prompt optimization, which is suitable for medical image analysis, clinical auxiliary diagnosis and the like. BACKGROUND

[0002] Interactive medical image segmentation (IMIS) requires human (usually physicians) to provide interactive prompts (such as clicks, box selections) to guide image segmentation models to complete accurate segmentation, aiming to overcome the poor performance of fully automatic segmentation models in complex and ambiguous lesion areas, while avoiding the low efficiency, strong subjectivity and fatigue of pure manual segmentation. It is widely used in key clinical scenarios such as tumor delineation, surgical planning, and radiotherapy target area development. Existing medical image segmentation technology faces two major challenges:

[0003] Prompt dependency: The performance of Segment Anything Model (SAM) is highly dependent on the choice of prompt form. Existing methods (such as the enhancement strategy of SAMAug and the reinforcement learning strategy of TEPO) do not fully consider the dependency of historical prompts and the uncertainty of expert annotations.

[0004] Annotation uncertainty: The boundaries of lesions in medical images are ambiguous, and expert annotations have subjective differences, resulting in insufficient stability of existing methods (such as TEPO) in real clinical scenarios.

[0005] Currently, interactive medical image segmentation methods based on SAM can be mainly divided into two categories: prompt enhancement methods and strategy learning methods.

[0006] Prompt enhancement methods (such as SAMAug): additional prompts or prompt positions are generated through handcrafted rules (such as random selection, maximum entropy). The former often relies on human intuition or pre-trained models, and the latter method often requires multiple gradient calculations during inference, which is inefficient and heavily dependent on pre-set rules, making it difficult to adapt to complex clinical scenarios.

[0007] Strategy learning methods (such as TEPO): a reinforcement learning framework is used to learn the prompt selection strategy. However, existing strategy methods have significant drawbacks: 1. They fail to fully utilize the information of historical prompt sequences, leading to interactive redundancy and short-sighted strategies; 2. In medical images, the boundaries of many lesions (such as gliomas and ground glass nodules) are themselves ambiguous, diffuse, and unclear. Different expert physicians may have different annotations of the same lesion based on their experience, judgment criteria, and visual perception. The training process relies entirely on a single "gold standard" annotation, ignoring the annotation differences (i.e., annotation uncertainty) between different experts, resulting in poor generalization ability and insufficient stability of the learned strategy in real clinical environments.

[0008] Therefore, there is an urgent need for a new generation of IMIS framework that can deeply understand the interaction history and simulate the adaptation to the differences of expert annotations, in order to promote its transition from laboratory demonstration to real clinical application. SUMMARY

[0009] The present application aims to overcome the technical problems existing in the prior art, and provides an interactive medical image segmentation method and system based on robust sequence prompt optimization, which improves the zero-shot performance and robustness of a basic segmentation model (such as SAM) in medical image segmentation through dynamic prompt selection, uncertain reward modeling and boundary disturbance technology, significantly reduces the annotation cost and improves the reliability of complex lesion segmentation.

[0010] The purpose of the present application is achieved by the following technical solutions:

[0011] In a first aspect, an interactive medical image segmentation method based on robust sequence prompt optimization is provided, comprising the following steps:

[0012] S1. Model the interactive medical image segmentation process as a Markov process, and construct an interaction state space, wherein the interaction state space includes a medical image, a historical prompt sequence and a historical image segmentation result;

[0013] S2. Use an image segmentation model to generate a current image segmentation result based on the current input medical image and the prompt;

[0014] S3. Output a prompt selection action based on a double-branch prompt fusion strategy network; wherein a robust reward function is introduced, including a deterministic reward and an uncertain reward generated based on a PCA boundary disturbance technology;

[0015] S4. Perform the selected prompt action, input it into the image segmentation model, and obtain a new round of image segmentation result;

[0016] S5. Repeat steps S1 to S4 until the segmentation result meets the preset accuracy or the upper limit of the number of interactions is reached;

[0017] S6. Output the final image segmentation mask and generate a visual medical image boundary overlay map.

[0018] Preferably, the step S1 specifically comprises:

[0019] The state at each interaction is defined as a triple:

[0020]

[0021] wherein, denotes the state at the tth interaction, denotes a preprocessed medical image, an image segmentation mask after the t-th interaction, a history prompt sequence.

[0022] Preferably, the robust reward function is defined as:

[0023]

[0024] wherein, a robust reward function of the t+1-th interaction, a deterministic reward of the t+1-th interaction, an uncertainty reward of the t+1-th interaction, and λ represents a penalty weight.

[0025] Preferably, the dual-branch prompt fusion strategy network comprises:

[0026] an image encoding branch for extracting medical image features through a convolutional neural network;

[0027] a mask encoding branch for extracting mask features of historical image segmentation through a convolutional neural network;

[0028] a prompt sequence encoding branch for extracting a history prompt sequence through a self-attention mechanism;

[0029] a multi-modal fusion module for fusing the medical image features, the mask features of historical image segmentation, and the history prompt sequence.

[0030] Preferably, the fusing of the medical image features, the mask features of historical image segmentation, and the history prompt sequence comprises:

[0031] linear processing after concatenating the medical image features and the mask features of historical image segmentation;

[0032] linear processing after concatenating the output of the linear processing and the history prompt sequence, to obtain a final output.

[0033] In a second aspect, an interactive medical image segmentation system based on robust sequence prompt optimization is provided, comprising:

[0034] an interaction state representation module for modeling an interactive medical image segmentation process as a Markov process and constructing an interaction state space, the interaction state space comprising a medical image, a history prompt sequence, and a historical image segmentation result;

[0035] an image segmentation module for generating a current image segmentation result based on a current input medical image and a prompt using an image segmentation model;

[0036] The prompt selection module is configured to select a prompt based on a double-branch prompt fusion strategy network output prompt selection action.

[0037] The policy update module is configured to perform the selected prompt action, input into the image segmentation model, and obtain a new round of image segmentation result until the segmentation result meets the preset accuracy or the upper limit of the number of interactions.

[0038] The visualization output module is configured to output a final image segmentation mask and generate a visual medical image boundary overlay.

[0039] It should be further pointed out that the technical features corresponding to the above options can be combined or replaced with each other to form new technical solutions without conflict.

[0040] Compared with the prior art, the present application has the following advantages:

[0041] 1. The segmentation accuracy and stability are significantly improved

[0042] Quantitative accuracy advantage: In the BraTS2021 brain tumor segmentation task, the method achieves a Dice score of 0.834±0.134, which is more than 4.1% higher than the best baseline method (TEPO, 0.801±0.162), and the standard deviation is reduced by 17.3%, proving that the output is more stable and reliable.

[0043] Boundary fitting optimization: The Hausdorff distance is reduced to 19.79±16.17mm, which is optimized by 22.5% compared with the random strategy (25.54±16.55mm), significantly improving the ability to outline complex and irregular lesions.

[0044] 2. Robustness breakthrough to annotation differences

[0045] Clinical uncertainty immunity: Through the introduction of PCA boundary perturbation technology and uncertainty reward mechanism, in the test of simulating expert annotation differences, the performance fluctuation range (Dice standard deviation ±0.038) of the method is reduced by 76.5% compared with the traditional reinforcement learning method (TEPO, ±0.162), showing strong clinical adaptability.

[0046] Cross-center generalization ability: On the medical center data not seen before, the performance decay of the method is less than 3%, while the decay of the baseline method is more than 15%, greatly improving the feasibility of the model in actual multi-center clinical application.

[0047] 3. Interaction efficiency and computational performance optimization

[0048] Greatly reduced interaction times: The method only needs an average of 5 interactions to achieve convergence, which is more than 40% more efficient than the rule-based random strategy (usually 8-10 times), greatly reducing the interaction burden of physicians.

[0049] Real-time inference capability: The optimized DPFPN network model is lightweight, and can achieve real-time decision-making speed of 9.4 milliseconds per step on a single NVIDIA RTX 3080 GPU, supporting intraoperative real-time segmentation applications.

[0050] 4. Excellent cross-modal zero-shot generalization capability

[0051] Applicable to multiple modalities without fine-tuning: The method achieves leading performance (Dice scores of 0.834, 0.817, and 0.809, respectively) on three vastly different datasets: BraTS2021 (MRI), AutoPET22 (PET-CT), and AMOS2022 (CT), without any parameter adjustment or fine-tuning, demonstrating its strong zero-shot generalization capability.

[0052] Unified architecture solves multiple tasks: One set of model parameters can handle brain tumor, whole body tumor, and abdominal organ segmentation tasks, reducing the complexity and maintenance cost of clinical deployment.

[0053] 5. Strong technology fusion and innovation

[0054] Multi-disciplinary deep cross: This invention first integrates reinforcement learning sequential decision-making, principal component analysis (PCA) for medical image labeling uncertainty modeling, and a dual-branch prompt fusion strategy network (DPFPN) into a complete "perception-decision-robust optimization" technical closed loop, providing an innovative solution for the field of interactive medical image segmentation.

[0055] Opens up new research directions: The proposed PCA boundary perturbation method provides a new technical tool for uncertainty quantification research in medical image analysis, with important academic value.

[0056] In summary, through systematic innovation in the four core dimensions of precision, robustness, efficiency, and generalization, the invention effectively solves the key bottleneck problems of interactive medical image segmentation in clinical deployment, laying a solid technical foundation for the development of the next generation of intelligent, reliable, and efficient auxiliary diagnostic tools. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 Flowchart of the interactive medical image segmentation method based on robust sequential prompt optimization of the invention;

[0058] Figure 2 Uncertainty reward function calculation process of the invention;

[0059] Figure 3 Schematic diagram for generating different boundary noises of the present application;

[0060] Figure 4 Schematic diagram of the dual-branch prompt fusion strategy network structure of the present application;

[0061] Figure 5 Comparison chart of the present application with different interaction strategies. DETAILED DESCRIPTION

[0062] The technical solutions of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application.

[0063] It should be noted that the defects in the above prior art solutions are the results obtained by the inventors after practice and careful study. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present application to the above problems should be the contributions made by the inventors to the present application during the process of invention and creation, and should not be understood as technical content known to those skilled in the art.

[0064] Based on the above related statements, since the present embodiments involve medical image segmentation related technologies, in order to make the purposes, technical solutions and advantages of the embodiments of the present application more clear, the related professional terms that may be involved in the present application are explained as follows:

[0065] 1. IMIS (Interactive Medical Image Segmentation) is an interactive medical image segmentation method that requires human (usually a doctor) to provide interactive prompts (such as clicking, framing) to guide the image segmentation model to complete accurate segmentation. It is especially suitable for medical imaging scenarios with high labeling cost and strict accuracy requirements.

[0066] 2. SAM (Segment Anything Model) is a basic segmentation model. It has strong zero-shot segmentation capability, that is, it can segment objects in images according to prompts without training for specific tasks. In the present application, SAM is used as the core basic segmentation model, and the work of the present application is to optimize the prompts provided to SAM to improve its performance on medical images.

[0067] 3、 PCA (Principal Component Analysis) is a classical statistical method for reducing data dimensionality and finding the most important change direction (principal component) in the data. In the present invention, PCA is innovatively used to analyze the boundary shape of the gold standard annotation and perform anisotropic noise disturbance in its principal direction, thereby simulating the annotation differences (uncertainty) between different experts. This is one of the core technologies for realizing the robustness of the present invention.

[0068] To solve the technical problems pointed out in the background art, the embodiments provided by the present invention are as follows:

[0069] Reference Figure 1 In an exemplary embodiment, an interactive medical image segmentation method based on robust sequence prompt optimization is provided, comprising the following steps:

[0070] S1. Model the interactive medical image segmentation process as a Markov process and construct an interactive state space, which includes a medical image, a historical prompt sequence and a historical image segmentation result;

[0071] S2. Use an image segmentation model to generate a current image segmentation result based on the current input medical image and the prompt;

[0072] S3. Output a prompt selection action based on a double-branch prompt fusion strategy network; wherein a robust reward function is introduced, including a deterministic reward and an uncertainty reward generated based on PCA boundary disturbance technology;

[0073] S4. Perform the selected prompt action and input it into the image segmentation model to obtain a new round of image segmentation result;

[0074] S5. Repeat steps S1 to S4 until the segmentation result meets the preset accuracy or the upper limit of the number of interactions;

[0075] S6. Output the final image segmentation mask and generate a visualized medical image boundary overlay map.

[0076] The present invention proposes a robust sequence prompt optimization framework, which is based on three major technical breakthroughs: multi-modal state perception, clinical uncertainty modeling and reinforcement decision optimization, and systematically solves the problems of prompt sensitivity and annotation difference in medical image segmentation. The specific technical solutions are as follows:

[0077] In step S1, the IMIS process is dynamically modeled as a Markov process, and a reinforcement learning method is used to find the optimal policy. To implement reinforcement learning, the interactive state space of the process needs to be modeled first. The interactive state space used by the present application includes multi-modal data composed of medical images, historical prompt sequences, and historical image segmentation results. The specific modeling process is as follows:

[0078] The system state at each interaction is defined as a triple:

[0079]

[0080] wherein, denotes the state at the tth interaction, denotes the pre-processed medical image (grayscale normalized to [0, 1]), denotes the image segmentation mask after the tth interaction, denotes the historical prompt sequence.

[0081] In step S3, in order to adapt to the labeling uncertainty in the interaction process, the present application introduces a robust reward function, so that the reinforcement learning model can stably implement IMIS in an environment with boundary noise. The reward function essentially guides the model to obtain the desired segmentation result, so the present application uses the DICE function as the reward function. The DICE function is a benchmark reward for measuring the improvement of segmentation accuracy:

[0082]

[0083] wherein, denotes the gold standard, the numerator represents the multiplication of true positive pixels (emphasizing the recall of lesion area), and the denominator represents the sum of true and predicted pixels (punishing over-segmentation and under-segmentation). The DICE function is a quantitative indicator for deterministic segmentation results and can be used as a deterministic reward function. The increment of the quantitative indicator before and after the interaction is defined as the reward function, and the formula is as follows.

[0084]

[0085] wherein, denotes the state variable after the tth interaction, denotes the action variable of the tth interaction.

[0086] On this basis, an uncertainty reward is developed. By using PCA technology to perturb the gold standard boundary to simulate the perturbation of expert labeling differences, an uncertain segmentation reward is constructed, and the formula is as follows:

[0087]

[0088] wherein The segmentation result after the interaction t+1 is based on the perturbed label The generated noisy segmentation.

[0089] Finally, the present application uses a composite reward function containing deterministic reward and uncertainty reward:

[0090]

[0091] wherein, represents the robust reward function of the t+1th interaction, represents the deterministic reward of the t+1th interaction, represents the uncertainty reward of the t+1th interaction, and λ represents the penalty weight. Since the perturbed label is a random variable with uncertainty, the present application uses the reward in the expected sense as the uncertainty reward in the actual training in order to stabilize the training. Figure 2 The calculation process of the uncertainty reward function is given as follows:

[0092] Noise sampling is added according to the selected action of decision, and a noisy segmentation label is generated:

[0093] A new hint is created based on the noisy label:

[0094] A new segmentation result is generated using the new hint, and the sampling process is repeated multiple times to generate multiple noisy segmentation results;

[0095] The reward value of each sampling is calculated, a plurality of perturbed label samples are obtained by Monte Carlo simulation of the perturbed label, and the corresponding segmentation result and quantitative index are calculated by sampling, and the average is taken as the uncertainty reward in the expected sense.

[0096] Further, in combination with Figure 3 , the PCA boundary perturbation technology is specifically as follows:

[0097] Based on the characteristics of the mask being a 0-1 binary matrix, the present application uses a more refined design to quickly extract the lesion boundary, that is, an 8-connected region detection and morphological thinning algorithm is used:

[0098]

[0099] wherein, B represents the boundary matrix, represents the boundary extraction operation convolution kernel, .

[0100] After extracting the lesion boundary, the present application uses principal component analysis dimension reduction to extract the contour main direction. First, the boundary coordinates are re-parameterized as

[0101]

[0102] Secondly, the boundary point rearrangement matrix X is centralized: , represents the mean value of the point coordinates in each dimension.

[0103] Then, the covariance matrix is decomposed:

[0104]

[0105]

[0106] wherein, represents the covariance matrix, represents the transformation matrix obtained by singular value decomposition, represents the eigenvalue matrix obtained by singular value decomposition.

[0107] Finally, the boundary point rearrangement matrix is projected into the principal component space: , represents the new coordinate matrix of the boundary points projected onto the feature space, and the application injects anisotropic noise into the real boundary to simulate expert uncertainty labeling. The specific implementation is to add scale adaptive noise in the PCA space:

[0108]

[0109] wherein, represents the projection point matrix after adding noise, is anisotropic Gaussian noise, that is,

[0110]

[0111]

[0112] wherein, represents the coordinate range of the kth principal direction. represents the perturbation range (noise intensity), which is usually taken as 0.05. This anisotropic noise addition maintains the importance difference of different directions in morphology.

[0113] After adding noise in each principal direction, the application reconstructs the boundary point coordinates back to the original space, and this reconstruction maintains the anatomical topological consistency in the original space. Specifically, the boundary point matrix after inverse transformation is obtained by inverse transformation

[0114]

[0115] wherein, represents the boundary point matrix after inverse transformation.

[0116] The boundary points are approximately valued

[0117]

[0118] Final binary mask generation

[0119]

[0120] Exemplarily, the present application proposes a DPFPN network, comprising:

[0121] An image encoding branch for extracting medical image features through a convolutional neural network;

[0122] A mask encoding branch for extracting mask features of historical image segmentation through a convolutional neural network;

[0123] A prompt sequence encoding branch for extracting historical prompt sequences through a self-attention mechanism;

[0124] A multi-modal fusion module for fusing the medical image features, the mask features of historical image segmentation and the historical prompt sequences.

[0125] In combination Figure 4 The specific details of each branch are as follows:

[0126] Image encoding branch:

[0127] - Input: preprocessed medical image

[0128] - Architecture: 4 layers of convolutional layers (channel number 16-32-64-128) + max pooling layer

[0129] - Output: medical image features

[0130] Mask encoding branch:

[0131] - Input: binary existing segmentation

[0132] - Architecture: 4 layers of convolutional layers (channel number 16-32-64-128) + max pooling layer

[0133] - Output: mask features

[0134] Prompt sequence encoding branch:

[0135] - Vectorization: hidden unit 256 dimensions, output 1024-dimensional linear layer

[0136] - Attention mechanism: self-attention mechanism

[0137] - Output: historical prompt sequences

[0138] Multi-modal fusion module:

[0139] First, the medical image features and the mask features of the historical image segmentation are connected and then processed by a linear layer, then the fused features and the historical prompt sequence are connected and processed by a linear layer, and finally the final output is obtained by processing through a linear layer. The specific details are as shown below:

[0140]

[0141] Wherein represents the fusion features of the image and the mask, represents an activation function, represents a matrix flattening operation into a vector, represents image features, represents mask features, represents a vector splicing operation, represents a 1000-dimensional real vector space. represents policy features, represents an activation function, represents prompt sequence features, represents the output result, i.e. action prediction, represents a linear connection layer, is a parameter of the linear connection layer, represents a 4-dimensional vector space.

[0142] Further, the present application also performs robust reinforcement learning optimization design, and the prior art performs reinforcement learning based on Bellman equation, but since the present application introduces an uncertainty reward function, the Bellman equation needs to be improved, and the specific details are as follows:

[0143]

[0144] Wherein, represents the value function of the tth interaction, is the state before the tth interaction and the action taken, represents the immediate return, represents the decay coefficient of the return, represents the expectation of the sample generated by SAM interaction, represents the maximum Q value operation in the action space corresponding to the greedy selection, represents the value function of the t+1th interaction.

[0145] The MSE loss function of Bellman iteration is used to optimize the network parameters

[0146]

[0147] Wherein For the fragment of the interaction sequence, called a segment trajectory, as a sample of the training sample, For the Q value of the model fitting, For the target Q value, the calculation method is

[0148]

[0149] In an example, for the demand for precise delineation of brain glioma in neurosurgical operation planning, brain glioma MRI image segmentation is carried out, aiming to quickly and accurately segment the tumor area in the MRI image through a small amount of interaction. The specific content is as follows:

[0150] 1. Data preprocessing:

[0151] Data source: FLAIR sequence MRI image in the BraTS2021 training set.

[0152] Slice screening: Extract axial slices from the 3D volume, and automatically screen slices with tumor area ≥256 pixels (about 35% of the total number of slices).

[0153] Preprocessing flow:

[0154] Pixel value normalization: Linearly transform the pixel value of the original DICOM image to the interval [0, 1].

[0155] Spatial standardization: All slices are uniformly cropped by center or zero-padded to 256x256 resolution.

[0156] Data augmentation: Horizontal flip (probability 0.5) and random rotation (±15°) are used in the training stage to improve the generalization of the model.

[0157] 3. System configuration and parameters:

[0158] Basic model: Use SAM (ViT-H) as the segmentation basic model, and the weight is fixed.

[0159] DPFPN network parameters:

[0160] Image / mask encoder: 4 layers of convolution, channel number [16, 32, 64, 128], kernel size 3x3, step 1.

[0161] Prompt sequence encoder: LSTM hidden unit number 256, self-attention head number 8.

[0162] Fusion layer: Fully connected layer dimension [1024, 512, 4], activation function GeLU.

[0163] Reinforcement learning parameters:

[0164] Discount factor γ = 0.95, reward balancing coefficient λ = 0.5.

[0165] Experience replay buffer capacity 10000, minimum batch size 32.

[0166] Optimizer: Adam (lr = le-4), training period 100.

[0167] 4. Implementation steps:

[0168] Step 1: Initialization

[0169] Input an MRI slice, initial state = (slice, all-zero mask, empty hint sequence).

[0170] Step 2: First interaction

[0171] DPFPN network analyzes the initial state , outputs the action probability distribution [0.1, 0.1, 0.2, 0.6] → selects the action (bounding box). Simulate the expert drawing the minimum bounding rectangle around the tumor area according to the gold standard . SAM receives , outputs the initial segmentation mask . Calculate the reward: , , 0.02 by PCA perturbation sampling calculation → 0.68 + 0.5 × 0.02 = 0.69. The state is updated to .

[0172] Steps 3-5: Iterative optimization

[0173] Repeat the above process:

[0174] Second step: DPFPN selects (frontier point), the expert clicks in the under-segmented area, and the DICE improves to 0.76.

[0175] Third step: Select (center point), place in the tumor core, DICE improves to 0.81

[0176] Fourth step: Select (background point), mark the over-segmented area, DICE improves to 0.83.

[0177] Fifth step: Select (frontier point), fine-tune the edge, and the final DICE = 0.87.

[0178] Step 6: Output results

[0179] After 5-step interaction, output the final segmentation mask And generate a visual tumor boundary overlay for physician review.

[0180] 5. Implementation effects:

[0181] On 50 test cases, the average DICE score reached 0.834±0.134, which was 5 times more efficient than manual full-time annotation.

[0182] The average interaction time of a single case was 47 seconds (including SAM inference and physician operation), which met the timeliness requirements of preoperative planning.

[0183] The segmentation result can be directly imported into the surgical navigation system to guide tumor resection.

[0184] Further, the interactive segmentation strategy based on reinforcement learning of the present application is compared with the existing alternating strategy and random strategy, as shown in Figure 5 The interactive segmentation strategy based on reinforcement learning (RSPO strategy) of the present application optimizes the segmentation result by selecting the optimal action. Through the learned intelligent strategy, it tends to continuously select interaction in the central region of the image. From the results, its DICE coefficient quickly improves and reaches a high level (0.91) with the interaction steps, indicating that the strategy can efficiently guide the model to focus on the key region and obtain high-quality segmentation results with the least interaction steps. The interaction action of the alternating strategy alternates between the foreground and the background. Although the final result is good (DICE 0.87), the first step interaction effect is poor (DICE 0.36), which shows that this mechanical alternating method may not provide the most effective correction information at the initial stage. The interaction action (such as box, center, and fore) of the random strategy is randomly selected. Its DICE coefficient fluctuates greatly throughout the process and the final result is relatively the worst (0.77), which proves that the random interaction without guidance is inefficient and difficult to stably improve the segmentation quality.

[0185] Figure 5 The differences in segmentation accuracy (DICE coefficient) and efficiency (improvement speed) of different interaction strategies are intuitively compared, highlighting the superiority of the intelligent strategy of the present application over fixed rules or random strategies.

[0186] In another exemplary embodiment, an interactive medical image segmentation system based on robust sequence prompt optimization is provided, comprising:

[0187] An interaction state representation module for modeling the interactive medical image segmentation process as a Markov process and constructing an interaction state space, the interaction state space including a medical image, a historical prompt sequence, and a historical image segmentation result;

[0188] an image segmentation module configured to generate a current image segmentation result based on a current input medical image and a prompt by using an image segmentation model;

[0189] a prompt selection module configured to select a prompt action based on a double-branch prompt fusion strategy network;

[0190] a policy update module configured to execute the selected prompt action, input into the image segmentation model, and obtain a new round of image segmentation result until the segmentation result meets a preset accuracy or an upper limit of an interaction number;

[0191] a visualization output module configured to output a final image segmentation mask and generate a visual medical image boundary overlay.

[0192] The above detailed description is a detailed description of the present application, which cannot be considered as limiting the specific embodiments of the present application to these descriptions. For those skilled in the art, without departing from the concept of the present application, a number of simple deductions and substitutions can be made, which should be considered as falling within the scope of protection of the present application.

Claims

1. An interactive medical image segmentation method based on robust sequence hint optimization, characterized in that, The method comprises the following steps: S1. modeling the interactive medical image segmentation process as a Markov process, and constructing an interactive state space comprising a medical image, a historical prompt sequence, and a historical image segmentation result; S2. generating a current image segmentation result based on a current input medical image and a prompt using an image segmentation model; S3. outputting a prompt selection action based on a double-branch prompt fusion strategy network; wherein a robust reward function comprising a deterministic reward and an uncertainty reward generated based on a PCA boundary perturbation technique is introduced; the PCA boundary perturbation technique comprises: After extracting the lesion boundary, the main direction of the contour is extracted using principal component analysis dimension reduction, first reparameterizing the boundary coordinates as ； Secondly, the boundary points rearrangement matrix X is centralized: , denotes the mean value of the point coordinates in each dimension. Then, perform eigenvalue decomposition on the covariance matrix: ， wherein denotes a covariance matrix, denotes a transformation matrix resulting from singular value decomposition, denotes an eigenvalue matrix resulting from singular value decomposition; Finally, the boundary points rearrangement matrix is projected to the principal component space: , denotes the new coordinate matrix of the boundary points projected onto the feature space; The robust reward function is defined as: wherein, represents a robust reward function for the t+1th interaction, represents a deterministic reward for the t+1th interaction, represents an uncertainty reward for the t+1th interaction, and λ represents a penalty weight; wherein, denotes the state variable after the tth interaction, denotes the action variable of the tth interaction; wherein, denotes the segmentation result after the t+1th interaction under the premise of introducing boundary uncertainty; S4. performing the selected prompt action, inputting into the image segmentation model, and obtaining a new round of image segmentation result; S5. repeating steps S1 to S4 until the segmentation result meets the preset accuracy or the upper limit of the number of interactions; S6. outputting the final image segmentation mask and generating a visual medical image boundary overlay map.

2. The interactive medical image segmentation method based on robust sequence hint optimization of claim 1, wherein, The step S1 specifically comprises: The state at each interaction is defined as a triple: wherein, denotes the state at the t-th interaction, denotes the pre-processed medical image, denotes the image segmentation mask after the t-th interaction, denotes the history of the hint sequence.

3. The robust sequence prompt optimization-based interactive medical image segmentation method of claim 1, wherein, The double-branch prompt fusion strategy network comprises: An image encoding branch for extracting medical image features through a convolutional neural network; A mask encoding branch for extracting mask features of historical image segmentation through a convolutional neural network; A prompt sequence encoding branch for extracting historical prompt sequences through a self-attention mechanism; A multi-modal fusion module for fusing medical image features, historical image segmentation mask features, and historical prompt sequences.

4. The robust sequence prompt optimization-based interactive medical image segmentation method of claim 1, wherein, The fusion of medical image features, historical image segmentation mask features, and historical prompt sequences comprises: Connecting the medical image features and the historical image segmentation mask features and performing linear processing; Connecting the output after linear processing with the historical prompt sequence and performing linear processing again to obtain the final output.

5. An interactive medical image segmentation system based on robust sequence hint optimization, characterized in that, Comprise: An interactive state representation module for modeling the interactive medical image segmentation process as a Markov process and constructing an interactive state space comprising a medical image, a historical prompt sequence, and a historical image segmentation result; An image segmentation module for generating a current image segmentation result based on a current input medical image and a prompt using an image segmentation model; A prompt selection module for outputting a prompt selection action based on a double-branch prompt fusion strategy network; wherein a robust reward function comprising a deterministic reward and an uncertainty reward generated based on a PCA boundary perturbation technique is introduced; the PCA boundary perturbation technique comprises: After extracting the lesion boundary, the main direction of the contour is extracted using principal component analysis dimension reduction, first reparameterizing the boundary coordinates as ； Secondly, the boundary points rearrangement matrix X is centralized: , denotes the mean value of the point coordinates in each dimension. Then, perform eigenvalue decomposition on the covariance matrix: ， wherein denotes a covariance matrix, denotes a transformation matrix resulting from singular value decomposition, denotes an eigenvalue matrix resulting from singular value decomposition; Finally, the boundary points rearrangement matrix is projected to the principal component space: , denotes the new coordinate matrix of the boundary points projected onto the feature space; The robust reward function is defined as: wherein, represents the robust reward function of the t+1th interaction, represents the deterministic reward of the t+1th interaction, represents the uncertainty reward of the t+1th interaction, and λ represents a penalty weight; wherein, denotes the state variable after the tth interaction, denotes the action variable of the tth interaction; wherein, denotes the segmentation result after the t+1th interaction under the premise of introducing boundary uncertainty; A strategy update module for performing the selected prompt action, inputting into the image segmentation model, and obtaining a new round of image segmentation result until the segmentation result meets the preset accuracy or the upper limit of the number of interactions; A visualization output module for outputting the final image segmentation mask and generating a visual medical image boundary overlay map.

Images