Point prompt segmentation method and device based on attack defense mechanism and readable storage medium

Abstract

The invention discloses a point prompt segmentation method and device based on an attack defense mechanism and a readable storage medium, and belongs to the field of computer vision. The problems that an existing prompt-driven segmentation technology is sensitive to prompt noise, lacks an automatic prompt quality evaluation and optimization mechanism and is insufficient in cross-scene generalization ability are solved. According to the technical scheme for solving the technical problem, the method comprises the following steps: constructing an initial prompt point heterogeneous graph fusing space and semantic information; inputting the prompt point heterogeneous graph as an environment state into an attack agent to complete noise prompt point injection and instant reward calculation; the defense agent performs prompt point screening, restoration and reward calculation; performing double-agent confrontation game and cooperative training; performing iterative prompt point evaluation and deletion on the new input image and the initial prompt point set by adopting the defense agent subjected to solidification training, and outputting an optimized prompt point set; the method is applied to image segmentation.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image segmentation, specifically to a point cue segmentation method, apparatus, and readable storage medium based on an attack defense mechanism. Background Technology

[0002] In existing image segmentation techniques, cue-driven segmentation models, such as the Segment Anything Model (SAM), rely heavily on the quality of the input cue points for their segmentation performance. Incorrect or redundant cue points can significantly reduce the model's segmentation accuracy, especially in cross-domain applications or when the cue information is noisy, where existing models often exhibit unstable segmentation results.

[0003] To improve the quality of cue points, various methods for statically generating cue points have been proposed, including feature matching-based strategies, geometric heuristics, and methods using auxiliary models. However, these methods are generally unidirectional generation mechanisms and lack the ability to adaptively adjust to the segmentation results of the SAM model. Therefore, when the initial cue points are of poor quality, unevenly distributed, or contain misleading content, problems such as segmentation boundary drift and missed target segmentation can easily occur.

[0004] In recent years, some learnable prompting optimization methods, such as VRP-SAM and MaskSAM, have achieved certain results on specific tasks by training dedicated prompting encoders or introducing additional networks to optimize prompt generation. However, these methods typically rely on task-specific supervised data or require model fine-tuning, making them difficult to apply directly to unlabeled or cross-domain scenarios. Furthermore, because the prompt generation module and the SAM decoding process are independent, they cannot dynamically adjust the prompt points in real time based on the actual segmentation error, resulting in insufficient robustness to prompt noise and limiting their practical applicability.

[0005] Other studies have attempted to optimize cue layout using reinforcement learning methods, such as PPO-based methods that adjust cue distribution by learning structural relationships between points. However, these methods typically employ relatively simple training environments, and the reward function is not directly correlated with the SAM segmentation results, leading to weak adaptability of the optimization strategy to initial cue quality and scene changes. More critically, existing methods generally lack effective mechanisms for identifying and suppressing "harmful cue points." When cue points originate from feature matching, weakly supervised segmentation, or manual annotation, they often introduce a large number of misleading points, which current technologies struggle to effectively filter out.

[0006] In summary, current technology has not yet provided a cue optimization mechanism that is task-agnostic, can adaptively adjust using segmentation feedback, and has the ability to clean up noise cues. Summary of the Invention

[0007] To address the technical problems of existing prompt-driven segmentation technologies, such as sensitivity to prompt noise, lack of automated prompt quality assessment and optimization mechanisms, and insufficient cross-scenario generalization ability, this invention proposes a point prompt segmentation method, device, and readable storage medium based on an attack defense mechanism.

[0008] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a point-based prompt segmentation method based on an attack defense mechanism, comprising the following steps:

[0009] Step S1: Construct an initial heterogeneous graph of cue points that integrates spatial and semantic information;

[0010] Step S2: Input the heterogeneous cue point graph as the environmental state into the attacking agent to complete the noise cue point injection and immediate reward calculation;

[0011] Step S3: Input the updated cue point heterogeneous graph and segmentation performance feedback information into the defense agent. The defense agent evaluates the harmfulness of each cue point in the updated cue point heterogeneous graph, selects the cue point with the highest harmfulness and performs a deletion action to optimize the cue set. Based on the segmentation performance improvement brought about by the deletion action, calculate the immediate reward of the defense agent.

[0012] Step S4: Based on the immediate rewards of the attacking agent and the defending agent, the policy network parameters of the attacking agent and the defending agent are updated using a deep reinforcement learning algorithm, so that the two agents can cooperate to optimize in the adversarial game and complete the dual-agent adversarial game and cooperative training; wherein, the goal of the attacking agent is to minimize the segmentation performance, and the goal of the defending agent is to maximize the segmentation performance.

[0013] Step S5: Using the defense agent after solidified training, iteratively evaluate and delete cue points on the new input image and the initial cue point set, and output the optimized cue point set.

[0014] Furthermore, in step S1, an initial cue point heterogeneous graph is constructed that integrates spatial physical edges and semantic feature edges, including the following steps:

[0015] Step S11, Offline Feature Extraction: Use the pre-trained visual base model to extract features from the input image, obtain the global feature map of the input image, and extract the visual feature vector of each prompt point according to the initial prompt point position of the input image; at the same time, pre-calculate the spatial distance matrix and semantic similarity matrix between all prompt points and persist them.

[0016] Step S12, Online Environment Construction: Load pre-stored features in the reinforcement learning environment; initialize all cue point features by fusing the location coordinate encoding, positive / negative label embedding, and corresponding visual features of each cue point; based on the pre-computed spatial distance matrix and semantic similarity matrix, set thresholds to construct spatially adjacent physical edges and semantically similar feature edges, thereby forming an initial cue point heterogeneous graph that simultaneously represents the spatial layout and semantic association of cue points.

[0017] Furthermore, in step S2, completing the noise cue point injection and immediate reward calculation includes the following steps:

[0018] Step S21: Input the initial cue point heterogeneous graph as the environment state into the attack agent; The attack agent evaluates the value of attack actions such as adding positive or negative cue points on the preset candidate position network based on its built-in deep Q network, and selects target attack actions according to the ε-greedy policy and executes them, and finally generates noisy cue points in the current cue point heterogeneous graph.

[0019] Step S22: Add the noise cue points to the initial cue point set to complete the update of the cue point heterogeneous graph;

[0020] Step S23: Input the set of cue points after the attack action perturbation into the downstream segmentation model, and output the perturbation segmentation mask from the downstream segmentation model; calculate the second similarity index based on the perturbation segmentation mask and the real segmentation mask. The second similarity index characterizes the segmentation performance of the downstream segmentation model after the attack action perturbation.

[0021] Step S24: Retrieve the preset first similarity index and compare it with the second similarity index; wherein, the first similarity index is the quantification result of the similarity between the initial segmentation mask output by the model and the real segmentation mask after the initial cue point set before the attack is input into the downstream segmentation model, and the immediate reward of the attacking agent is obtained.

[0022] Furthermore, in step S3, calculating the immediate reward for the defensive agent includes the following steps:

[0023] Step S31: Input the heterogeneous graph of the current cue points and the segmentation performance feedback information into the defense agent.

[0024] Step S32: The defensive agent analyzes the heterogeneous graph of prompt points in the current state through its built-in deep Q-network, and encodes each prompt point and the relationship between the edges in the heterogeneous graph using a graph neural network.

[0025] Step S33: Based on the encoding results, the defense agent evaluates the "harmfulness" of each existing cue point in the current state's cue point heterogeneous graph and outputs a Q value for each existing cue point. This Q value is used to characterize the expected value of the segmentation performance improvement after deleting the corresponding existing cue point.

[0026] Step S34: The defensive agent selects the cue point with the highest Q value and performs the deletion action, removing the cue point and all its associated edges from the cue point heterogeneous graph in the current state, thus completing the purification and repair of the cue point set;

[0027] Step S35: Calculate the improvement in segmentation performance after the execution of the defense action and use it as an immediate reward for the defense agent.

[0028] Furthermore, in step S31, the segmentation performance feedback information specifically includes the second similarity index corresponding to the attack action, the first similarity index corresponding to the attack action, and the immediate reward of the attacking agent. The core purpose is to provide feedback on the changes in segmentation performance caused by the current set of prompt points.

[0029] Furthermore, the improvement in segmentation performance is obtained by comparing the third similarity index between the repaired segmentation mask and the real segmentation mask obtained by inputting the set of post-defense hint points into the downstream segmentation model, and the second similarity index before defense.

[0030] Further, step S5, outputting the optimized set of prompts includes the following steps:

[0031] Step S51: After the dual-agent adversarial game and collaborative training are completed, only the defensive agent is retained and the parameters of the defensive agent are solidified. Based on the solidified defensive agent, the final cue point optimization model is formed and then the inference stage is entered.

[0032] Step S52: After entering the inference stage, for the new input image and the initial cue point set, repeat the initial cue point heterogeneous graph constructed in step S1, call the solidified defense agent to perform multiple rounds of iterative cue point evaluation and deletion operations, delete the most harmful cue point in each round, until the preset stopping condition is reached.

[0033] Step S53: Output the final set of remaining cue points as the optimization result and input it into the downstream segmentation model.

[0034] Furthermore, in step S52, the preset stopping condition is that the maximum number of deletion rounds or the highest Q value of the remaining indicator points is lower than a threshold.

[0035] A dot-mapping segmentation device based on an attack defense mechanism for implementing the above-described method includes:

[0036] The composition module is used to integrate spatial and semantic information into a heterogeneous cue point graph;

[0037] The adversarial training module is used to build and train attack and defense agents;

[0038] The model solidification module is used to solidify the policy network of the defensive agent after adversarial training is completed.

[0039] The inference optimization module is used to load the defense agent after it has been trained and perform iterative deletion optimization on new input images and cue points.

[0040] A readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0041] The advantages of this invention over the prior art are as follows:

[0042] 1. Structured Representation Enhances the Accuracy of Cue Analysis: This invention constructs an adversarial learning framework where attacking and defending agents engage in a game-like interaction. Simulating real-world noise injection and adaptive cleanup processes, the system can automatically learn to filter effective information from noisy or low-quality interactive cue sets, thereby improving the stability and robustness of the segmentation model in diverse scenarios. It fundamentally solves the problem of traditional methods relying solely on cue coordinates and failing to perceive semantic differences in cue points, fully utilizing the correlation information between cue points and significantly improving the accuracy of cue analysis.

[0043] 2. Adversarial Training Enhances Model Robustness: After completing the adversarial game and collaborative training between two agents, the method of this invention retains only the defensive agent and solidifies its parameters. Based on the solidified defensive agent, a final cue point optimization model is formed and then enters the inference stage. In the inference stage, the cue point optimization model is invoked, and it performs multiple rounds of iterative cue point evaluation and filtering operations on the initial cue point set. Finally, a purified and optimized cue point set is output. This optimized cue point set is input into the downstream segmentation model, which can stably improve the segmentation performance and robustness of the downstream segmentation model in various application scenarios.

[0044] 3. Segmentation Performance-Driven Accurate Prompt Diagnosis: This invention establishes a direct correlation between prompting behavior and segmentation performance by using changes in the Dice or IoU output of the segmentation model as the sole reinforcement learning reward. This enables the agent to accurately judge the harmfulness or effectiveness of prompts based on the segmentation results themselves, truly possessing prompt quality diagnostic capabilities. Compared to existing technologies that lack prompt correction mechanisms based on segmentation feedback and cannot accurately respond to erroneous prompts, this invention can automatically filter out misleading prompts through reward-driven mechanisms, significantly reducing segmentation bias caused by prompt errors.

[0045] 4. Enhanced scalability and versatility without additional dependencies: This invention uses a defensive agent to filter and remove prompts from the prompt set, performing quality filtering before the prompt set is input into the segmentation model. This effectively improves the applicability of the segmentation model to complex boundaries, noisy regions, and cross-domain images. Furthermore, this reinforcement learning-driven prompt correction method requires no additional supervision and does not rely on additional training of the segmentation model, exhibiting strong scalability and versatility.

[0046] 5. Adversarial Training + Streamlined Inference: Balancing Performance and Practicality: Employing an adversarial training mode with attacking and defending agents, the training phase automatically generates a distribution of difficult examples, expanding the model's learning coverage. During the inference phase, only the defending agent is retained to operate independently, enabling efficient system operation and simple deployment. Furthermore, it maintains stable and optimized performance under different tasks, image types, and cue sources. This advantage stems from its inherent structured design, rather than manual rules or task-specific data, and possesses significant practical application value. Attached Figure Description

[0047] The present invention will be further described below with reference to the accompanying drawings:

[0048] Figure 1 A flowchart illustrating an embodiment of the method of the present invention. Figure 1 ;

[0049] Figure 2 This is a flowchart illustrating the process of injecting noise cue points and calculating immediate rewards according to an embodiment of the present invention;

[0050] Figure 3 A flowchart illustrating an embodiment of the method of the present invention is provided. Figure 2 ;

[0051] Figure 4 This is a schematic diagram of the segmentation structure comparison results of the method of the present invention;

[0052] Figure 5 A schematic diagram of the device of the present invention. Detailed Implementation

[0053] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate relative orientations or positional relationships and are used only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0054] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0055] like Figures 1 to 5 As shown, this invention provides a point-based cue segmentation method based on an attack defense mechanism, comprising the following steps:

[0056] Step S1: Construct an initial cue point heterogeneous graph that integrates spatial physical edges and semantic feature edges. The cue point heterogeneous graph integrates the position, label and visual features of the cue points in the input image.

[0057] Specifically, the process involves constructing an initial cue point heterogeneous graph that integrates spatial physical edges and semantic feature edges, including the following steps:

[0058] Step S11, Offline Feature Extraction: Use a pre-trained visual base model (such as DINO) to extract features from the input image, obtain the global feature map of the input image, and extract the visual feature vector of each prompt point according to the initial prompt point position of the input image; at the same time, pre-calculate the spatial distance matrix and semantic similarity matrix between all prompt points and persist them.

[0059] Step S12, Online Environment Construction: Load pre-stored features in the reinforcement learning environment; initialize all cue point features by fusing the location coordinate encoding, positive / negative label embedding, and corresponding visual features of each cue point; based on the pre-computed spatial distance matrix and semantic similarity matrix, set thresholds to construct spatially adjacent physical edges and semantically similar feature edges, thereby forming an initial cue point heterogeneous graph that simultaneously represents the spatial layout and semantic association of cue points.

[0060] Step S2: Input the heterogeneous cue point graph as the environmental state into the attacking agent to complete the noise cue point injection and immediate reward calculation.

[0061] Specifically, it includes the following sub-steps:

[0062] Step S21: The initial cue point heterogeneous graph constructed in step S1 is used as the environmental state input to the attack agent. The attack agent evaluates the value of attack actions such as adding positive or negative cue points on the preset candidate position network based on its built-in deep Q network (DQN), and selects target attack actions according to the ε-greedy policy and executes them. Finally, noisy cue points are generated in the current cue point heterogeneous graph. Noisy cue points are misleading cue points.

[0063] Step S22: Add the noise cue points to the initial cue point set to complete the update of the cue point heterogeneous graph.

[0064] Step S23: Input the set of cue points after the attack action perturbation into the downstream segmentation model (e.g., the segmentation model SAM), and the downstream segmentation model outputs the perturbation segmentation mask; based on the perturbation segmentation mask and the real segmentation mask, calculate the second similarity index (e.g., the Dice coefficient), which characterizes the segmentation performance of the downstream segmentation model after the attack action perturbation.

[0065] Step S24: Retrieve the preset first similarity index (characterizing the segmentation performance of the downstream segmentation model before the attack action perturbation) and compare it with the second similarity index; wherein, the first similarity index is the similarity measurement result between the initial segmentation mask output by the downstream segmentation model after the initial cue point set before the attack is input into the downstream segmentation model and the real segmentation mask; the difference between the first similarity index and the second similarity index obtained by comparison (performance degradation) is the immediate reward of the attacking agent.

[0066] Step S2 aims to actively simulate various inaccurate or random user prompts (noise prompts) that may occur in real-world scenarios, thereby constructing a training environment that covers a wide range of defect patterns.

[0067] Step S3: The defensive agent performs cue point filtering, repair, and reward calculation: The updated cue point heterogeneity graph and segmentation performance feedback information are input into the defensive agent. The agent evaluates the harmfulness of each cue point in the updated cue point heterogeneity graph, selects the cue point with the highest harmfulness, and performs a deletion action to optimize the cue set. Based on the segmentation performance improvement brought about by the deletion action, the immediate reward of the defensive agent is calculated. Specifically, this includes the following steps:

[0068] Step S31: Input the heterogeneous graph of the current cue points and the segmentation performance feedback information into the defense agent.

[0069] The segmentation performance feedback information specifically includes the second similarity index corresponding to the attack action, the first similarity index corresponding to the attack action, and the immediate reward of the attacking agent (i.e., the decrease in segmentation performance). The core purpose is to provide feedback on the changes in segmentation performance caused by the current set of cue points.

[0070] Step S32: The defensive agent analyzes the heterogeneous graph of the prompt points in the current state through its built-in deep Q-network, and at the same time uses a graph neural network to encode the relationship between each prompt point and the edges (including physical edges and feature edges) in the heterogeneous graph.

[0071] Step S33: Based on the encoding results, the defensive agent evaluates the "harmfulness" of each existing cue point (the cue point added by the attacking agent or the initial cue point) in the cue point heterogeneous graph in the current state, and outputs a Q value (attack action value function value) for each existing cue point. The Q value is used to characterize the expected value of the segmentation performance improvement after deleting the corresponding existing cue point.

[0072] Step S34: The defensive agent selects the cue point with the highest Q value and performs the deletion action, removing the cue point and all its associated edges (including physical edges and feature edges) from the cue point heterogeneous graph in the current state, thus completing the purification and repair of the cue point set;

[0073] Step S35: Calculate the segmentation performance improvement after the defense action is executed, and use it as the immediate reward for the defense agent. The segmentation performance improvement is obtained by comparing the third similarity index between the repaired segmentation mask and the true segmentation mask (obtained by inputting the set of cue points after defense into the downstream segmentation model) with the second similarity index before defense.

[0074] Step S4, Dual-Agent Adversarial Game and Collaborative Training: Based on the immediate rewards of the attacking agent and the defending agent, a deep reinforcement learning algorithm is used to update the policy network parameters of the attacking agent and the defending agent in parallel or alternately. The attacking agent and the defending agent form a dynamic adversarial game relationship, enabling them to collaboratively optimize in the adversarial game. This game process can enable the defending agent to learn and master the ability to handle various complex noise cues in continuous adversarial play, thereby improving its defense generalization.

[0075] The attacking agent aims to minimize segmentation performance (i.e., reduce segmentation accuracy), while the defending agent aims to maximize segmentation performance (i.e., improve segmentation accuracy).

[0076] Step S5: Using the defense agent after solidified training, iteratively evaluate and delete cue points on the new input image and the initial cue point set, and output the optimized cue point set.

[0077] Specifically, the following steps are included:

[0078] Step S51: After the dual-agent adversarial game and collaborative training are completed, only the defensive agent is retained and the parameters of the defensive agent are solidified. Based on the solidified defensive agent, the final cue point optimization model (i.e. the optimized cue point set) is formed and then the inference stage is entered.

[0079] Step S52: After entering the inference stage, for the new input image and the initial set of prompt points, repeat step S1 to construct the initial prompt point heterogeneous graph, call the solidified defense agent to perform multi-round iterative prompt point evaluation and deletion operations, delete the most harmful prompt point in each round, until the preset stopping condition is reached (such as the maximum number of deletion rounds, or the highest Q value of the remaining indicator points is lower than the threshold).

[0080] Step S53: Output the final set of remaining cue points as the optimization result, and input it into the downstream segmentation model to obtain a more stable and accurate segmentation mask.

[0081] This invention provides a specific embodiment:

[0082] The method of the present invention includes the following steps:

[0083] Step S1: Construct an initial cue point heterogeneous graph that integrates spatial physical edges and semantic feature edges, including the following steps:

[0084] Step S11, Offline Feature Extraction: For the input image (Where H is the image height, W is the image width, and 3 is the number of image channels), firstly, the global feature map of the image is extracted using the DINO model. (in, These represent the height and width of the global feature map, respectively, and C is the number of feature channels. For the set of time-based cue points... (where N is the number of prompts, (where is the coordinate of the i-th prompt point), where These represent positive and negative cue points, respectively. The visual feature vector corresponding to the i-th cue point is obtained from the global feature map F using the bilinear sampling method. Simultaneously, a semantic feature vector based on DINO features is calculated for each cue point, and a semantic relationship matrix between the cue points is constructed based on cosine similarity. Finally, the global feature map F and the visual feature vectors of all cue points will be combined. Semantic relation matrix Spatial distance matrix The initial cue point set P is packaged into a weight file and stored, i.e. This weight file serves as input data for the reinforcement learning environment.

[0085] Step S12, Online Environment Construction: When starting the reinforcement learning environment, load the stored weight file to obtain the dataset. Subsequently, the multilayer perceptron was initialized. Encode the position coordinates of the i-th prompt point Tag embedding and visual feature vectors By performing fusion, the initial features of the prompt point are obtained. .

[0086] Constructing the physical edges and feature edges of the heterogeneous graph of the initial cue points: using a pre-computed spatial distance matrix Set spatial distance threshold ,when When determining that a physical edge exists between the i-th and j-th prompt points, this physical edge is used to characterize the spatial neighborhood relationship between the prompt points; a pre-computed cosine similarity matrix is ​​then used. Set a semantic similarity threshold ,when In this case, the feature edge is used to characterize the semantic feature similarity between the prompt points. Even if the spatial distance between two prompt points is far, a connection can still be established through the feature edge.

[0087] The resulting initial cue point heterogeneous graph is represented as follows: , where V is the set of cue point nodes (corresponding to the initial cue point set P). For the set of physical edges, Let H be the set of feature edges, and H be the initial feature set of all nodes.

[0088] Step S2: Input the heterogeneous cue point graph as the environmental state into the attacking agent to complete the noise cue point injection and immediate reward calculation. This includes the following steps:

[0089] Step S21: Input the current environmental state (i.e., the current cue point heterogeneous graph G) into the attacking agent;

[0090] The attacking agent uses its built-in deep Q-network. Each candidate action is evaluated; the value of each candidate attack action is assessed; and the action space of the attacking agent is defined. From a predefined candidate location grid Together with tag types, it constitutes, that is (+1 indicates a positive warning label, -1 indicates a negative warning label).

[0091] Attacking agents based on Strategy selection of target attack actions The specific selection logic is as follows:

[0092] ;

[0093] Step S22: Execute the selected attack action. That is, in the candidate location grid Add a new tag here. The newly added cue point (which is also a noise cue point) is added as a new node to the current cue point heterogeneous graph. The updated graph structure is obtained. .

[0094] Step S23: Set the core mask variable. Before the attack is executed, the initial cue point set is input into the downstream segmentation model to obtain the segmentation mask. After the attack is executed, the segmentation mask obtained by inputting the set of perturbation cue points into the model is: At the same time, the true segmentation mask is determined as .

[0095] Step S24: Calculate the pre-attack segmentation performance metrics: Calculate the pre-attack segmentation mask. Compared with the real segmentation mask The Dice coefficient is obtained. ,Should This is the primary similarity metric.

[0096] Calculate post-attack segmentation performance metrics: Calculate the post-attack segmentation mask. Compared with the real segmentation mask The Dice coefficient is obtained. ,Should This is the second similarity index.

[0097] Calculate the segmentation performance degradation: The degradation is obtained by comparing the similarity metrics before and after the attack. That is, the decrease in segmentation performance. The larger the reward value, the more destructive the attack is to the segmentation results of the downstream segmentation model.

[0098] Step S3: The defensive agent performs cue point filtering, repair, and reward calculation: The updated cue point heterogeneity graph and segmentation performance feedback information are input into the defensive agent. The agent evaluates the harmfulness of each cue point in the updated cue point heterogeneity graph, selects the cue point with the highest harmfulness, and performs a deletion action to optimize the cue set. Based on the segmentation performance improvement brought about by the deletion action, the immediate reward of the defensive agent is calculated. Specifically, this includes the following steps:

[0099] Step S31, Input Information Preparation: Prepare the heterogeneous graph of the prompt points in the current state. The segmentation performance feedback information, along with other data, is input into the defensive agent. Among these:

[0100] The segmentation performance feedback information specifically includes: the first similarity index before the attack action, the second similarity index after the attack action, and the immediate reward for the attacking agent (i.e., the decrease in segmentation performance); its core function is to provide feedback on the changes in segmentation performance caused by the current set of cue points.

[0101] Step S32, Heterogeneous Graph Analysis and Encoding: The defensive agent utilizes a built-in deep Q-network The current heterogeneous graph of prompt points is analyzed, and graph neural networks are used to encode the relationships between each prompt point and the edges (including physical edges and feature edges) in the heterogeneous graph, thus completing the deep analysis of the features of the prompt points and the relationship between the edges.

[0102] Step S33, Cue Point Harmfulness Assessment and Q-Score Output: Based on the above encoding results, the defensive agent assesses the "harmfulness" of each existing cue point (including initial cue points or cue points added by the attacking agent) in the current state's cue point heterogeneity graph, and outputs a Q-score (attack action value function value) for each existing cue point. The core significance of this Q-score is: to characterize the expected value of improving the performance of the downstream segmentation model after deleting the corresponding existing cue point.

[0103] Step S34, Optimal Cue Point Deletion and Set Repair: The defensive agent selects the cue point with the highest Q value and performs the deletion action: from the current state cue point heterogeneous graph Remove the cue point and all its associated edges (including physical edges and feature edges) to obtain the optimized cue point heterogeneous graph. This completes the purification and repair of the set of prompt points;

[0104] Core action formula: ,in, As an indicator point, This is the set of all indicator points in the current heterogeneous graph. Indicator point The Q value.

[0105] Step S35, Instant Reward Calculation: Calculate the improvement in segmentation performance after the defensive action is executed, and use it as the instant reward for the defensive agent. The calculation logic for the segmentation performance improvement is as follows: The third similarity index between the repaired segmentation mask (obtained by inputting the set of post-defense hints into the downstream segmentation model) and the actual segmentation mask is compared with the second similarity index before defense. The specific calculation logic is as follows:

[0106] Obtain key metrics:

[0107] Segmentation performance metrics before defense (i.e. after attack): Second similarity metric (segmentation mask obtained by inputting the set of cue points into the downstream segmentation model after the attack action and the real segmentation mask).

[0108] Segmentation performance metrics after defense: Third similarity metric (the similarity between the repaired segmentation mask obtained by inputting the repaired cue point set into the downstream segmentation model after defense and the real segmentation mask).

[0109] The improvement in segmentation performance = the third similarity index - the second similarity index. This improvement in segmentation performance is the immediate reward for the defensive agent. .

[0110] Step S4: Based on the immediate rewards of the attacking agent and the defending agent, a deep reinforcement learning algorithm is used to update the policy network parameters of the attacking and defending agents in parallel or alternately. The attacking and defending agents form a dynamic adversarial game relationship, enabling them to collaboratively optimize in the adversarial game. This game process allows the defending agent to learn and master the ability to handle various complex noise cues in continuous adversarial play, thereby improving its defense generalization. The attacking agent's goal is to minimize segmentation performance (i.e., reduce segmentation accuracy), while the defending agent's goal is to maximize segmentation performance (i.e., improve segmentation accuracy). Specifically, this includes the following steps:

[0111] Step S41: Define the adversarial training objective: attack agent parameters With defensive agent parameters The optimization process constitutes a mini-maximum game problem, the objective of which is to maximize the final segmentation accuracy of the defensive agent under the worst cue perturbation, formally represented as: .

[0112] Step S42: Establish the training framework and experience storage: A deep Q-learning algorithm based on experience replay is used to train the two agents alternately. Experience replay buffers are provided for the attacking agent and the defending agent, respectively. Used to store their respective state transition experiences This provides data support for subsequent parameter updates.

[0113] Step S43: Attacking agent parameter update: In each training iteration, update the parameters from the attacking agent's experience replay buffer. The empirical data of a batch is sampled, and its Q-network parameters are updated by minimizing the temporal difference error. The loss function is defined as follows: .

[0114] Step S44, Defense Agent Parameter Update: Update defense agent parameters from the defense agent experience replay buffer. The system collects sampling experience data and uses the same update logic as the attacking agent to update the Q-network parameters by minimizing its own temporal difference error. Loss function .

[0115] Step S45, Target Network Soft Update Synchronization: After completing the main network parameter update, synchronize the target network parameters of the attacking agent and the defending agent according to the soft update rules: The update formula is as follows: ,in This is the soft update coefficient, which is usually close to 1.

[0116] Step S5: Using the defense agent after solidified training, iteratively evaluate and delete cue points on the new input image and the initial cue point set, and output the optimized cue point set.

[0117] Specifically, the following steps are included:

[0118] Step S51: Model Solidification and Loading. After the two-agent adversarial game and collaborative training are completed, the attacking agent and all its related components are discarded, and only the defending agent is retained, along with its optimal policy network parameters. The process involves solidifying the defensive agent. Based on this solidified model, the final cue point optimization model (i.e., the cue point optimizer) is constructed, and then the inference phase begins.

[0119] Step S52: Initialize the inference environment. For the new input image... and its initial prompt point set Following the process in step S1, load or calculate the corresponding DINO features and construct the initial cue point heterogeneous graph. Set it as the current environment state .

[0120] Step S53: Iterative cue point filtering. The defensive agent uses a greedy strategy ( In state The program runs on the same screen, and its built-in Q-network represents each prompt point in the diagram. Calculate the Q-value for deleting this node. (This represents the expected value of deleting the indicator point in improving segmentation performance); then, the indicator point with the highest Q value is selected and deleted. and from the current heterogeneous graph Remove the indicator point and all its associated edges to obtain the updated heterogeneous graph. .

[0121] Step S54: Termination condition determination. Repeat Step 3 for multiple rounds of iterative filtering until any preset stopping condition is met:

[0122] Reaching the preset maximum number of deletion steps ;

[0123] The maximum Q value of all remaining indicator points is lower than a preset threshold. ,Right now This indicates that the remaining cue points are either beneficial or neutral. The final set of remaining cue points is then output as the optimization result and input into the downstream segmentation model to obtain a more stable and accurate segmentation mask.

[0124] This invention provides a dot-mapping segmentation device based on an attack defense mechanism for implementing the above-described method, comprising:

[0125] The composition module is used to integrate spatial and semantic information into a heterogeneous cue point graph;

[0126] The adversarial training module is used to build and train attack and defense agents.

[0127] The model solidification module is used to solidify the policy network of the defensive agent after adversarial training is completed.

[0128] The inference optimization module is used to load the defense agent after it has been trained and perform iterative deletion optimization on new input images and cue points.

[0129] The present invention provides a readable storage medium on which a computer program is stored, and the computer program, when executed by a processor, implements the method described above.

[0130] Regarding the specific structure of this invention, it should be noted that the connection relationships between the various component modules used in this invention are definite and achievable. Except as specifically described in the embodiments, their specific connection relationships can bring about corresponding technical effects and solve the technical problems proposed by this invention without relying on the execution of corresponding software programs. The models of the components, modules, and specific components appearing in this invention, the connection methods between them, and the conventional usage methods and expected technical effects brought about by the above technical features, unless specifically described, are all publicly disclosed content in patents, journal articles, technical manuals, technical dictionaries, and textbooks that can be obtained by those skilled in the art before the application date, or belong to conventional technology, common knowledge, and other existing technologies in this field. There is no need to elaborate, which makes the technical solution provided in this case clear, complete, and achievable, and can reproduce or obtain corresponding physical products based on this technical means.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point-based suggestion segmentation method based on an attack defense mechanism, characterized in that, Includes the following steps: Step S1: Construct an initial heterogeneous graph of cue points that integrates spatial and semantic information; Step S2: Input the heterogeneous cue point graph as the environmental state into the attacking agent to complete the noise cue point injection and immediate reward calculation; Step S3: Input the updated cue point heterogeneous graph and segmentation performance feedback information into the defense agent. The defense agent evaluates the harmfulness of each cue point in the updated cue point heterogeneous graph, selects the cue point with the highest harmfulness and performs a deletion action to optimize the cue set. Based on the segmentation performance improvement brought about by the deletion action, calculate the immediate reward of the defense agent. Step S4: Based on the immediate rewards of the attacking agent and the defending agent, the policy network parameters of the attacking agent and the defending agent are updated using a deep reinforcement learning algorithm, so that the two agents can cooperate to optimize in the adversarial game and complete the dual-agent adversarial game and cooperative training; wherein, the goal of the attacking agent is to minimize the segmentation performance, and the goal of the defending agent is to maximize the segmentation performance. Step S5: Using the defense agent after solidified training, iteratively evaluate and delete cue points on the new input image and the initial cue point set, and output the optimized cue point set.

2. The point-based suggestion segmentation method based on an attack defense mechanism according to claim 1, characterized in that, In step S1, an initial cue point heterogeneous graph is constructed that integrates spatial physical edges and semantic feature edges, including the following steps: Step S11, Offline Feature Extraction: Use the pre-trained visual base model to extract features from the input image, obtain the global feature map of the input image, and extract the visual feature vector of each prompt point according to the initial prompt point position of the input image; at the same time, pre-calculate the spatial distance matrix and semantic similarity matrix between all prompt points and persist them. Step S12, Online Environment Construction: Load pre-stored features in the reinforcement learning environment; initialize all cue point features by fusing the location coordinate encoding, positive / negative label embedding, and corresponding visual features of each cue point; based on the pre-computed spatial distance matrix and semantic similarity matrix, set thresholds to construct spatially adjacent physical edges and semantically similar feature edges, thereby forming an initial cue point heterogeneous graph that simultaneously represents the spatial layout and semantic association of cue points.

3. The point-based suggestion segmentation method based on an attack defense mechanism according to claim 1, characterized in that, Step S2, which involves injecting noise cue points and calculating immediate rewards, includes the following steps: Step S21: Input the initial cue point heterogeneous graph as the environment state into the attack agent; The attack agent evaluates the value of attack actions such as adding positive or negative cue points on the preset candidate position network based on its built-in deep Q network, and selects target attack actions according to the ε-greedy policy and executes them, and finally generates noisy cue points in the current cue point heterogeneous graph. Step S22: Add the noise cue points to the initial cue point set to complete the update of the cue point heterogeneous graph; Step S23: Input the set of cue points after the attack action perturbation into the downstream segmentation model, and output the perturbation segmentation mask from the downstream segmentation model; calculate the second similarity index based on the perturbation segmentation mask and the real segmentation mask. The second similarity index characterizes the segmentation performance of the downstream segmentation model after the attack action perturbation. Step S24: Retrieve the preset first similarity index and compare it with the second similarity index; wherein, the first similarity index is the quantification result of the similarity between the initial segmentation mask output by the model and the real segmentation mask after the initial cue point set before the attack is input into the downstream segmentation model, and the immediate reward of the attacking agent is obtained.

4. The point-based suggestion segmentation method based on an attack defense mechanism according to claim 1, characterized in that, In step S3, calculating the immediate reward for the defensive agent includes the following steps: Step S31: Input the heterogeneous graph of the current cue points and the segmentation performance feedback information into the defense agent. Step S32: The defensive agent analyzes the heterogeneous graph of prompt points in the current state through its built-in deep Q-network, and encodes each prompt point and the relationship between the edges in the heterogeneous graph using a graph neural network. Step S33: Based on the encoding results, the defense agent evaluates the "harmfulness" of each existing cue point in the cue point heterogeneous graph in the current state and outputs a Q value for each existing cue point. The Q value is used to characterize the expected value of the segmentation performance improvement after deleting the corresponding existing cue point. Step S34: The defensive agent selects the cue point with the highest Q value and performs the deletion action, removing the cue point and all its associated edges from the cue point heterogeneous graph in the current state, thus completing the purification and repair of the cue point set; Step S35: Calculate the improvement in segmentation performance after the execution of the defense action and use it as an immediate reward for the defense agent.

5. The point-based suggestion segmentation method based on an attack defense mechanism according to claim 4, characterized in that, In step S31, the segmentation performance feedback information specifically includes the second similarity index corresponding to the attack action, the first similarity index corresponding to the attack action, and the immediate reward of the attacking agent. The core purpose is to provide feedback on the changes in segmentation performance caused by the current set of prompt points.

6. The point-based suggestion segmentation method based on an attack defense mechanism according to claim 4, characterized in that, The improvement in segmentation performance is obtained by comparing the third similarity index between the repaired segmentation mask and the real segmentation mask, obtained by inputting the set of cue points after defense into the downstream segmentation model, with the second similarity index before defense.

7. The point-based suggestion segmentation method based on an attack defense mechanism according to claim 4, characterized in that, Step S5, outputting the optimized set of cue points includes the following steps: Step S51: After the dual-agent adversarial game and collaborative training are completed, only the defensive agent is retained and the parameters of the defensive agent are solidified. Based on the solidified defensive agent, the final cue point optimization model is formed and then the inference stage is entered. Step S52: After entering the inference stage, for the new input image and the initial cue point set, repeat the initial cue point heterogeneous graph constructed in step S1, call the solidified defense agent to perform multiple rounds of iterative cue point evaluation and deletion operations, delete the most harmful cue point in each round, until the preset stopping condition is reached. Step S53: Output the final set of remaining cue points as the optimization result and input it into the downstream segmentation model.

8. The point-based suggestion segmentation method based on an attack defense mechanism according to claim 7, characterized in that, In step S52, the preset stopping condition is that the maximum number of deletion rounds or the highest Q value of the remaining indicator points is lower than the threshold.

9. A point-based prompting segmentation device based on an attack defense mechanism for implementing the method as described in any one of claims 1-8, characterized in that, include: The composition module is used to integrate spatial and semantic information into a heterogeneous cue point graph; The adversarial training module is used to build and train attack and defense agents; The model solidification module is used to solidify the policy network of the defensive agent after adversarial training is completed. The inference optimization module is used to load the defense agent after it has been trained and perform iterative deletion optimization on new input images and cue points.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 9.

Images