Automatic image segmentation method based on open context scenario

Abstract

The application discloses an automatic image segmentation method based on an open context scene, which comprises the following steps: a spatial perception intelligent agent selects a semantic switch action and feedback of the action based on a current state, and after iteration, an optimal prompt position is recommended; an image encoder and a prompt encoder are used in a SAM model to encode a given image and a prompt set respectively, so that image features and prompt features are obtained; the image features and the prompt features are input into a mask decoder to generate a segmentation mask; a text description is used to guide a visual-language model to encode the given image, so that an activation map is generated; the segmentation mask is multiplied by the image features to obtain a space-dominant state; the activation map is multiplied by the image features to obtain a semantic-dominant state; and the space-dominant state and the semantic-dominant state are input into a semantic calibration module to recalibrate the current state. The application improves the understanding ability of abstract concepts.

Description

Technical Field

[0001] This invention belongs to the technical field of image processing, specifically relating to an automatic prompting image segmentation method based on open context scenarios. Background Technology

[0002] Recent breakthroughs have been made in Vision Foundation Models (VFM). For example, the Segment Anything Model (SAM) has proven to have strong generalization capabilities, efficiently generalizing to tasks and data distributions not encountered during training. However, SAM is pre-trained in a class-agnostic manner. To apply SAM to various downstream scenarios (e.g., semantic segmentation, transparent object segmentation, salient object segmentation), additional manual prompts are needed based on image content. Common prompt types include points and bounding boxes. However, in real-world scenarios, constructing accurate manual prompts for each image individually is extremely tedious. Therefore, it is necessary to design appropriate transfer strategies to adapt the general SAM model to meet the needs of downstream tasks, thereby freeing the model from dependence on manual prompts.

[0003] In recent years, adaptation strategies for segmentation base models have emerged and can be roughly divided into three categories: text-guided methods, in-context learning methods, and parameter-efficient fine-tuning methods. Specifically: (1) Text-guided methods add an additional vision-language model (VLM) to the segmentation base model and focus on designing corresponding attention strategies to better integrate and interact visual and text features. (2) In-context learning methods focus on how to build better feature matching strategies so that the base model can understand and complete downstream tasks based on several input-output examples. (3) Parameter-efficient fine-tuning methods introduce additional fine-tunable components to the model, such as low-rank adaptation (LoRA) and adapters, so that the base model can achieve performance similar to fine-tuning all parameters while keeping most parameters frozen.

[0004] However, current adaptation strategies for basic segmentation models still have shortcomings. Text-guided adaptation methods struggle to handle abstract concepts in reality, such as "salient" objects in images that are visually significant to the human eye. Using abstract concepts as text cues can negatively impact model performance. Context-based adaptation methods also fall short in understanding abstract concepts, typically using only a single image as a reference, which is insufficient for the model to infer high-level abstract semantics. While parameter-efficient fine-tuning adaptation methods can handle abstract concepts well, they rely on a large number of training samples for fine-tuning, and the process of collecting and labeling these samples incurs significant overhead. Therefore, image segmentation methods that improve the understanding of abstract concepts and reduce training costs are urgently needed. Summary of the Invention

[0005] The main objective of this invention is to overcome the shortcomings and deficiencies of existing technologies and provide an automatic prompting image segmentation method based on open context scenarios. This method constructs a unified prompting framework through reinforcement learning and utilizes a spatially aware agent and a semantic recalibration module to adapt to downstream tasks involving implicit and explicit semantics, thereby improving the understanding of abstract concepts and reducing training overhead. Furthermore, the semantic recalibration module incorporates prior knowledge provided by the visual language model, ensuring the accuracy of prompt labels and ultimately achieving efficient model adaptation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides an automatic image segmentation method based on open context scenarios, comprising the following steps:

[0008] A spatially aware agent selects a semantic switching action based on its current state, receives a context-based reward signal from the environment, iterates through the process to recommend the optimal cue location, and obtains a cue set. The spatially aware agent includes a policy network and a value network. The cue set includes image location information and indicator labels. The semantic switching action includes explicit branching paths and implicit branching paths. Explicit branching paths are suitable for segmenting explicitly concrete tasks, while implicit branching paths are suitable for segmenting abstract semantic tasks. Specifically, selecting a semantic switching action based on the current state involves the spatially aware agent receiving a recalibrated state as input to predict the guided attention map for each candidate location. If the recalibrated state is semantically dominant, the explicit branch is executed; otherwise, the implicit branch is executed. The spatially aware agent requires reinforcement learning training for spatial perception. The training steps include: constructing an action space, constructing a state space, and policy optimization.

[0009] The construction of the action space specifically refers to:

[0010] Given the image Divided into image blocks And a candidate action set is constructed by collecting the center point of each image patch, as follows:

[0011] ,

[0012] in, Represents the coordinates of the center of the image patch. Indicates the height of the image. Indicates the height of the image block. Indicates the width of the image. Indicates the width of the image patch. Indicates the number of image channels;

[0013] The center point within the image patch is marked as a positive point, and points outside the image patch are marked as negative points. Select from candidate points Mark each point;

[0014] The construction of the state space is specifically as follows:

[0015] Extract a given image Visual features are embedded into spatial perception agents;

[0016] Using visual features and the previous time step The segmentation mask obtained at that time constructs the state space as follows:

[0017] ,

[0018] in, Representation of the current state, Represents the image features of a given image. Indicates the previous time step Mask encoding at time;

[0019] In the SAM model, an image encoder and a cue encoder are used to encode a given image and a cue set, respectively, to obtain image features and cue features. The image features and cue features are then input into a mask decoder to generate a segmentation mask. Text descriptions are used to guide the visual-language model to encode a given image and generate an activation map.

[0020] The segmentation mask is multiplied with the image features to obtain the spatially dominant state; the activation map is multiplied with the image features to obtain the semantically dominant state; the spatially dominant state and the semantically dominant state are input into the semantic calibration module to recalibrate the current state.

[0021] As a preferred technical solution, the strategy optimization includes:

[0022] The policy network maps the current state to the action probability distribution, and the value network determines the reward or penalty based on the action of the current state. The parameters of the spatially aware agent are iteratively optimized using the stochastic gradient descent method until the maximum cumulative reward is obtained. During each iteration, the parameters are stored in a memory buffer, including the current state, action, reward, and subsequent state.

[0023] As a preferred technical solution, the policy network maps the current state to an action probability distribution, specifically as follows:

[0024] Initialize each given image and randomly select positive and negative hints from the truth mask;

[0025] The total time steps of a certain iteration during training Inside, it interacts with the SAM model by performing a series of actions, when the time step is At that time, the spatially perceptive agent selects an action from the probability distribution. The probability distribution is as follows:

[0026] ,

[0027] in, Represents the probability distribution in the action space. (·) denotes a policy network;

[0028] The value network employs a reward mechanism, specifically: the reward function is set to the score of the target region; when the action of the spatial perception agent points to the foreground region at each time step, it receives a reward; otherwise, it is penalized.

[0029] As a preferred technical solution, the parameters of the spatially aware agent are iteratively optimized using the stochastic gradient descent method, as shown in the following equation:

[0030] ,

[0031] in, This measures the degree to which a specific action, taken under given conditions, is expected to exceed the average performance value. Representing the value network in relation to state The expected value after taking an action This represents the reward received from the environment at time step t+1. This represents the reward received from the environment at time step T-1. represents the discount factor, used to weigh the relationship between current rewards and future expected values, and T represents the total time step.

[0032] As a preferred technical solution, the strategy optimization process employs importance sampling to enhance stability, as shown in the following equation:

[0033] ,

[0034] in, This represents the probability ratio between the historical strategy and the current strategy at the current time step t. The action representing the current state. (·) represents the policy network.

[0035] As a preferred technical solution, the objective function of the policy network As shown in the following formula:

[0036] ,

[0037] in, Indicates hyperparameters, This represents the clipping function, used to control the minimum and maximum probability ratios. This represents the value of the dominance function at time t.

[0038] As a preferred technical solution, the value network is optimized by minimizing the difference between the estimated value and the actual reward. The training objective of the value network is defined as follows:

[0039] ,

[0040] in, This measures the degree to which a specific action, taken under given conditions, is expected to exceed the average performance value. This represents minimizing the estimated value, and T represents the total time steps.

[0041] As a preferred technical solution, the semantic switch action includes explicit branch paths and implicit branch paths;

[0042] Once the explicit branching path is open, convolution is used to convert the space-dominant state... and semantically dominant states Dimensionality reduction is performed, and spatially dominant states are fused using a self-attention module. and semantically dominant states The fused features are input into the convolutional block for mask decoding, and the current state is relabeled.

[0043] ,

[0044] ,

[0045] ,

[0046] This represents the prediction output of the semantic calibration module. Indicates the category header, This indicates that it is a semantic switch. Represents the image features of a given image. Indicates the previous time step Predicted segmentation mask, Represents the activation graph;

[0047] Once the implicit branching path is open, convolution is used to transform the space-dominant state. Dimensionality reduction is performed, and the given graph is then subjected to abstract semantic segmentation through another convolution to relabel the current state, as shown in the following equation:

[0048] ,

[0049] in, This represents the truth mask corresponding to the image. This indicates that no content was used as input.

[0050] As a preferred technical solution, the semantic calibration module combines Dice loss. and binary cross-entropy loss Conduct training as follows:

[0051] ,

[0052] in, This represents the prediction output of the semantic calibration module. .

[0053] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0054] (1) This invention enables a spatially perceptive agent to select a semantic switch action based on the current state, receive a context-based reward signal from the environment, and make location decisions for key points. This allows the agent to adapt to downstream tasks of implicit and explicit semantics based on a limited number of training reference samples, thereby improving the understanding of abstract concepts and reducing training overhead.

[0055] (2) The present invention defines the automatic prompting process of the pre-trained visual language model as a sequence decision process, unifies different downstream segmentation tasks in the framework of reinforcement learning, and achieves wide adaptability while ensuring the universality of the pre-trained model.

[0056] (3) This invention proposes a semantic recalibration module, which integrates the prior knowledge provided by the visual language model to ensure the high accuracy of the prompt labels and promote the efficiency of model adaptation. At the same time, the key prompt positions provided by the spatial perception agent, together with the labels corresponding to the prompt positions given by the semantic recalibration module, reduce the dependence of the visual basic model on manual annotation when applied to downstream tasks, and realize fully automatic reasoning. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart of an automatic image segmentation method based on an open context scenario, as described in an embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram of CLIP-Surgery segmentation results under different language prompts according to an embodiment of the present invention. Detailed Implementation

[0060] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0061] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0062] Please see Figure 1 This embodiment provides an automatic prompting image segmentation method based on open context scenarios, which mainly adopts a spatially aware agent, a Segment Anything Model (SAM), a visual language model CLIP-Surgery, and a semantic recalibration module SRM to achieve image segmentation.

[0063] Specifically, it includes the following steps:

[0064] S1. The spatially perceptive agent selects a semantic switch action based on the current state, receives a context-based reward signal from the environment, recommends the optimal cue position after iteration, and obtains a cue set; the spatially perceptive agent includes a policy network and a value network; the cue set includes image location information and indicator labels;

[0065] Following the standard reinforcement learning framework, this embodiment trains a spatially aware agent to interact with the environment iteratively. Therefore, the goal of the spatially aware agent is to recommend optimal cue locations to facilitate gradual improvement in the segmentation model's mask prediction. Specifically, in each round of interaction, the spatially aware agent first selects an action based on observations of the current state, and then receives a context-based reward signal from the environment.

[0066] It should be noted that, in order to enhance the reinforcement learning agent's understanding of contextual information, the recalibrated state representation takes into account both the semantic and spatial dominant features of the environment.

[0067] To achieve automatic prompting and recommend the best prompts, the spatially aware agent needs to undergo reinforcement learning training for spatial perception. The training steps include: constructing the action space, constructing the state space, and optimizing the policy.

[0068] (1) Constructing the action space.

[0069] To select suitable points, a simple approach is to iterate through every pixel in the image. However, this method is costly and inefficient. Considering that small changes in point positions do not lead to significant changes in prediction accuracy, this embodiment constructs the action space at the image patch level rather than the pixel level. Specifically, this embodiment divides the image into different regions to construct the action space for reinforcement learning. Given an input image... Divide it into image blocks A candidate action set is constructed by collecting the center points of each image patch. The construction method is represented as follows:

[0070] ,

[0071] in, Represents the coordinates of the center of the image patch. Indicates the height of the image. Indicates the height of the image block. Indicates the width of the image. Indicates the width of the image patch. Indicates the number of image channels;

[0072] Next, in order to segment the target foreground of the task, the agent is required to select as many candidate points as possible within the target object and select a small number of candidate points outside the target object as auxiliary points. In this embodiment, the center point within the image patch is marked as a positive point, and the points outside the image patch are marked as negative points. Select from candidate points Mark each point.

[0073] Furthermore, an image patch refers to several non-overlapping regions obtained after segmenting an image. For the input image... Divide it into a total of Image blocks .

[0074] (2) Construct the state space.

[0075] A reinforcement learning spatial awareness agent determines its next action at each time step t based on its current state. To help the spatial awareness agent make informed decisions, the state space must contain comprehensive information about the environment. Therefore, this embodiment combines visual feature embedding with iterative prediction results to construct the state representation. Regarding time Image samples The state can be represented as:

[0076] ,

[0077] in The mask represents the mask predicted by the mask decoder at the previous time step of the SAM. The features extracted by the image encoder reflect the intrinsic spatial properties of the samples, while the information from the model's previous prediction maintains contextual coherence in the temporal dimension. The multiplication operation emphasizes the characteristics of regions more likely to be classified as foreground in previous iterations and weakens the features of regions more likely to be classified as background, which is similar to an attention mechanism guided by historical information. In this way, the policy network maps states to the probability distribution of actions, thereby accurately determining the maximum expected reward of the current action in the current state.

[0078] (3) Train the RL model to achieve policy optimization.

[0079] This embodiment employs the widely used Proximal Policy Optimization (PPO) algorithm to implement the RL framework of this embodiment, which includes a policy network. and a value network Policy Network Mapping states to probability distributions of available actions enables agents to make decisions. Value networks... Then evaluate the value of the state-action pair. (Policy Network) Exploring and improving its strategies through interaction with the environment, and value networks This is a policy network. The action provides guidance and evaluation. (Policy Network) and value network Work together to maximize accumulated rewards.

[0080] It should be noted that the spatial awareness agent in this embodiment has a reward mechanism. Traditional reinforcement learning methods used for computer vision design involve manually designed reward functions to improve task performance between iterations. To reduce the burden of manually designing functions, this invention sets the reward function to the score of the target region. Specifically, at each time step, if the agent's action points to the foreground region, it will receive a reward of +1; otherwise, it will receive a penalty of -1 as feedback.

[0081] During the training phase, for each image, a set of prompts is initialized. This includes positive and negative cues randomly selected from their true labeled masks. In each training epoch, a discrete, finite time step is considered. The agent interacts with the segmentation module SAM by performing a series of actions. At time step... The agent selects an action from a probability distribution, which can be represented as follows:

[0082] ,

[0083] in Represents the probability distribution in the action space. Execute the action. Afterwards, the agent receives a reward from the environment. During each iteration, a tuple containing the current state, action, reward, and subsequent state is stored in a memory buffer. Once an episode is trained, the policy network is trained using stochastic gradient descent (SGD) over K epochs. and value network The parameters are optimized. To construct the optimization objective, the advantage function is first calculated. The formula is as follows:

[0084] ,

[0085] in, This measures the degree to which a specific action, taken under given conditions, is expected to exceed the average performance value. Representing the value network in relation to state The expected value after taking an action This represents the reward received from the environment at time step t+1. This represents the reward received from the environment at time step T-1. represents the discount factor, used to weigh the relationship between current rewards and future expected values, and T represents the total time step.

[0086] To address the non-stationarity issue, an importance sampling technique is employed during the strategy optimization process to enhance stability. Specifically, this embodiment uses... The probability ratio between the historical policy and the current policy at the current time step t is defined as follows:

[0087] ,

[0088] At each time point The advantage function is expressed as Then the objective function of the policy network will be calculated as follows:

[0089] ,

[0090] in, This represents the value of the dominance function at time t. It's a hyperparameter. This is a pruning function used to control the minimum and maximum values ​​of the probability ratio. Simultaneously, the value network is optimized to minimize the gap between the predicted value and the actual reward. The training objective formula for the value network is as follows:

[0091] ,

[0092] in, This indicates minimizing the estimated value.

[0093] Therefore, this embodiment applies the advantage function to auto-completion to measure the expected value of an action.

[0094] After completing the reinforcement learning training, the spatial perception agent receives the recalibrated state as input to predict the guided attention map for each candidate location, as detailed in step S3.

[0095] S2. In the SAM model, use an image encoder and a cue encoder to encode the given image and cue set respectively to obtain image features and cue features; input the image features and cue features into a mask decoder to generate a segmentation mask; use text description to guide the visual-language model to encode the given image to generate an activation map;

[0096] Regarding the Segmentation All Model (SAM), the aim is to achieve a unified capability for segmentation. Within this framework, given only a set of cues, such as foreground and background points, bounding boxes, or masks, the Segmentation All Model (SAM) can segment a wide range of targets.

[0097] Specifically, the Segmentation All Model (SAM) consists of an image encoder. A prompt encoder and a mask decoder Structure. In the model inference phase, firstly, an image encoder and a cue encoder are used to process the given image. and hint set The encoding is performed using the formula shown below:

[0098] ,

[0099] in , The total number of prompts. These represent the image's length, width, and number of channels, respectively. Then, image features... and prompt features It will be input into the mask decoder In the middle, generate the segmentation mask. .

[0100] The agent interacts iteratively with the Segmentation Model (SAM), optimizing the selection decision of candidate points through reinforcement learning training. The labels of the selected candidate points are learned through the semantic recalibration module, ultimately enabling the Segmentation Model (SAM) to handle various downstream scenarios without relying on manual prompts.

[0101] As for the visual-language model CLIP-Surgery, it is a variant model that combines the image-text pre-training model CLIP (Contrastive Language-Image Pre-training). In practical applications, it can accept text descriptions as input and output a pixel-by-pixel similarity map between the image and the text as semantic guidance.

[0102] In this embodiment, two models, Segmentation-All (SAM) and the Visual-Language (CLIP-Surgery), are used for image segmentation. They have different focuses. SAM determines the location of the segmentation target by calculating the distribution probability of image patches in a given image. Therefore, Segmentation-All focuses more on image space, and its features or states are spatially dominant. In contrast, the features or states of CLIP-Surgery are more semantically dominant.

[0103] However, please see Figure 2 Simply placing purely spatial or semantic features into a spatially perceptive agent to predict cue points and only identifying target regions at a coarse-grained level is clearly insufficient for accurate image segmentation. Figure 2 Parts (a) and (b) respectively demonstrate the use of "dog" and "visually salient objects" as language cues. "Visually salient objects" include, for example, a baseball in front of a computer desk in an image, or a turtle against a patterned background. It can be observed that the visual-language model performs well when processing explicit semantic content, but often faces challenges in interpreting implicit semantic content.

[0104] Therefore, a Semantic Recalibration Module (SRM) is needed. By integrating explicit and implicit information into the reinforcement learning framework, it further subdivides the cues selected by the spatially aware agent into positive and negative cues, providing accurate cue labels. The ultimate goal of the SRM is to output a mask as a reference for the agent's cues. If a cue falls within the mask, it is a positive cue; otherwise, it is a negative cue, as detailed in step S3.

[0105] S3. Multiply the segmentation mask with the image features to obtain the spatially dominant state; multiply the activation map with the image features to obtain the semantically dominant state; input the spatially dominant state and the semantically dominant state into the semantic calibration module to recalibrate the current state.

[0106] Specifically, to enhance the reinforcement learning agent's understanding of the environmental state, this embodiment proposes a semantically aware state representation. For space-dominant states, this embodiment uses the probability map predicted by the mask decoder at the previous time step to recalibrate the state, as it contains historical information learned from visual samples. Ultimately, semantically dominant and space-dominant states are defined as the product of their respective attention maps and the inherent properties of the samples, as shown in the following formula:

[0107] ,

[0108] in, Represents the image features of a given image. Indicates the previous time step Predicted segmentation mask, This represents the activation graph. Let... Indicates the category header, This indicates a semantic switch that selects whether to execute an implicit branch or an explicit branch.

[0109] The specific definitions for implicit and explicit branches are as follows:

[0110] Implicit branching: Implicit branching is suitable for tasks involving the segmentation of abstract semantics, such as saliency detection. These tasks cannot utilize textual features because cues like "visually salient object" inherently contain semantic information. Specifically, in the above formula... This is achieved using two convolutional blocks. This allows for the extraction of spatial features of the target without being affected by noise from text features.

[0111] Explicit branching: Explicit branching is suitable for segmenting explicitly specific tasks, such as various semantic targets (cats, dogs, cars). In this case, introducing textual information allows for a more comprehensive understanding of the specific content in the image, hence the use of explicit branching. The feature is designed as a 1x1 convolution to reduce dimensionality, and then fused with the two states using a self-attention module. Finally, this feature is fed into two convolutional blocks for mask decoding.

[0112] Once the explicit branching path is open, convolution is used to convert the space-dominant state... and semantically dominant states Dimensionality reduction is performed, and spatially dominant states are fused using a self-attention module. and semantically dominant states The fused features are input into the convolutional block for mask decoding, and the current state is relabeled.

[0113] ,

[0114] ,

[0115] ,

[0116] This represents the prediction output of the semantic calibration module, including explicit branch output and implicit branch output. Indicates the category header, This indicates that it is a semantic switch. Represents the image features of a given image. Indicates the previous time step Predicted segmentation mask, Represents the activation graph;

[0117] Once the implicit branching path is open, convolution is used to transform the space-dominant state. Dimensionality reduction is performed, and the given graph is then subjected to abstract semantic segmentation through another convolution to relabel the current state, as shown in the following equation:

[0118] ,

[0119] in, This represents the truth mask corresponding to the image. This indicates that no content was used as input.

[0120] It's important to note that when querying the corresponding positive or negative label based on the selected cue location, the truth mask result of the semantic adjustment module's explicit / image-based approach is only used as the label reference mask for the cue. This output does not recalibrate the state at the current time step, but it will affect the state at the next time step. This impact manifests in SAM's reasoning based on the labeled cue points at the current time step. This will affect the next time step. .

[0121] After recalibration via the explicit branch, the inputs to the semantic calibration module for either the explicit or implicit branch are the feature maps dominated by semantic and spatial fusion, respectively. and spatially dominant feature maps Explicit branching is a space-dominant state. and semantically dominant states After concatenation, 1x1 convolutional dimensionality reduction and self-attention operations are performed. The final result is the interaction between the space-dominant state and the semantic-dominant state. The input of the implicit branch is only the space-dominant state, that is, it only focuses on processing the space-dominant feature information.

[0122] To illustrate this more clearly, consider an example from a text-image multimodal model. Explicit branches aim to solve tasks that can be intuitively described using text, such as "dog," "airplane," and "chair." Therefore, they should be combined with the encoding obtained from the text-image multimodal model. Learning correction. Implicit branches address tasks that are difficult to describe in text, such as "a blurred background in a photograph" or "visually salient objects." Since the image similarity encoded by the image-text model is inaccurate, it is not included. Related information disturbance correction module.

[0123] Furthermore, the aforementioned classification head This can be achieved using the sigmoid function or the softmax function.

[0124] Training and testing process of the semantic recalibration module: using Dice loss. and binary cross-entropy loss The SRM module is trained using these two losses, which are widely used in image segmentation:

[0125] ,

[0126] During training, the SRM module will process the actions performed by the agent. The next iteration updates the algorithm. During testing, based on the highest and lowest probabilities predicted by the semantic recalibration module in the probability graph, a positive point and a negative point are initialized for the cue set.

[0127] It should be noted that, for the sake of simplicity, the aforementioned method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously.

[0128] Based on the same idea as the automatic image segmentation method with open context in the above embodiments, the present invention also provides an automatic image segmentation system with open context, which can be used to execute the above-described automatic image segmentation method with open context. For ease of explanation, the structural diagram of the embodiment of the automatic image segmentation system with open context only shows the parts related to the embodiments of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0129] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. An image segmentation method with automatic prompting in open context scenarios, characterized in that, Includes the following steps: A spatially aware agent selects a semantic switching action based on its current state, receives a context-based reward signal from the environment, iterates through the process to recommend the optimal cue location, and obtains a cue set. The spatially aware agent includes a policy network and a value network. The cue set includes image location information and indicator labels. The semantic switching action includes explicit branching paths and implicit branching paths. Explicit branching paths are suitable for segmenting explicitly concrete tasks, while implicit branching paths are suitable for segmenting abstract semantic tasks. Specifically, the selection of the semantic switching action based on the current state involves the spatially aware agent receiving a recalibrated state as input to predict the guided attention map for each candidate location. If the recalibrated state is semantically dominant, the explicit branch is executed; otherwise, the implicit branch is executed. The spatial perception agent needs to undergo reinforcement learning training for spatial perception. The training steps include: constructing an action space, constructing a state space, and optimizing the policy. The construction of the action space specifically refers to: Given the image Divided into image blocks And a candidate action set is constructed by collecting the center point of each image patch, as follows: ， in, Represents the coordinates of the center of the image patch. Indicates the height of the image. Indicates the height of the image block. Represents the width of the image Indicates the width of the image patch. Indicates the number of image channels; The center point within the image patch is marked as a positive point, and points outside the image patch are marked as negative points. Select from candidate points Mark each point; The construction of the state space is specifically as follows: Extract a given image Visual features are embedded into spatial perception agents; Using visual features and the previous time step The segmentation mask obtained at that time constructs the state space as follows: ， in, Representation of the current state, Represents the image features of a given image. Indicates the previous time step Mask encoding at time; In the SAM model, an image encoder and a cue encoder are used to encode a given image and a cue set, respectively, to obtain image features and cue features. The image features and cue features are then input into a mask decoder to generate a segmentation mask. Text descriptions are used to guide the visual-language model to encode a given image and generate an activation map. The segmentation mask is multiplied with the image features to obtain the spatially dominant state; the activation map is multiplied with the image features to obtain the semantically dominant state; the spatially dominant state and the semantically dominant state are input into the semantic calibration module to recalibrate the current state.

2. The automatic image segmentation method based on open context scenarios according to claim 1, characterized in that, The policy optimization includes: the policy network maps the current state to the action probability distribution, and the value network determines the reward or penalty based on the action of the current state; the parameters of the spatially aware agent are iteratively optimized using the stochastic gradient descent method until the maximum accumulated reward is obtained; during each iteration, the parameters are stored in a memory buffer, and the parameters include the current state, action, reward and subsequent state.

3. The automatic image segmentation method based on open context scenarios according to claim 2, characterized in that, The policy network maps the current state to an action probability distribution, specifically as follows: Initialize each given image and randomly select positive and negative hints from the truth mask; The total time steps of a certain iteration during training Inside, it interacts with the SAM model by performing a series of actions, when the time step is At that time, the spatially perceptive agent selects an action from the probability distribution. The probability distribution is as follows: ， in, Represents the probability distribution in the action space. (·) denotes a policy network; The value network employs a reward mechanism, specifically: the reward function is set to the score of the target region; when the action of the spatial perception agent points to the foreground region at each time step, it receives a reward, otherwise it is penalized.

4. The automatic image segmentation method based on open context scenarios according to claim 2, characterized in that, The parameters of the spatially aware agent are iteratively optimized using the stochastic gradient descent method, as shown in the following equation: ， in, This measures the degree to which a specific action, taken under given conditions, is expected to exceed the average performance value. Representing the value network in relation to state The expected value after taking an action This represents the reward received from the environment at time step t+1. This represents the reward received from the environment at time step T-1. represents the discount factor, used to weigh the relationship between current rewards and future expected values, and T represents the total time step.

5. The automatic image segmentation method based on open context scenarios according to claim 2, characterized in that, The strategy optimization process employs importance sampling to enhance stability, as shown in the following equation: ， in, This represents the probability ratio between the historical strategy and the current strategy at the current time step t. The action representing the current state. Represents the policy network.

6. The automatic image segmentation method based on open context scenarios according to claim 1, characterized in that, The objective function of the policy network As shown in the following formula: ， in, Indicates hyperparameters, This represents the clipping function, used to control the minimum and maximum probability ratios. This represents the value of the dominance function at time t. This represents the probability ratio between the historical policy and the current policy at the current time step t, where T represents the total number of time steps.

7. The automatic image segmentation method based on open context scenarios according to claim 1, characterized in that, The value network is optimized by minimizing the difference between the estimated value and the actual reward. The training objective of the value network is defined as follows: ， in, This measures the degree to which a specific action, taken under given conditions, is expected to exceed the average performance value. This represents minimizing the estimated value, and T represents the total time steps.

8. The automatic image segmentation method based on open context scenarios according to claim 1, characterized in that, Once the explicit branching path is open, convolution is used to convert the space-dominant state... and semantically dominant states Dimensionality reduction is performed, and spatially dominant states are fused using a self-attention module. and semantically dominant states The fused features are input into the convolutional block for mask decoding, and the current state is relabeled. ， ， ， This represents the prediction output of the semantic calibration module. Indicates the category header, This indicates that it is a semantic switch. Represents the image features of a given image. Indicates the previous time step Predicted segmentation mask, Represents the activation graph; Once the implicit branching path is open, convolution is used to transform the space-dominant state. Dimensionality reduction is performed, and the given graph is then subjected to abstract semantic segmentation through another convolution to relabel the current state, as shown in the following equation: ， in, This represents the truth mask corresponding to the image. This indicates that no content was used as input.

9. The automatic image segmentation method based on open context scenarios according to claim 1, characterized in that, The semantic calibration module incorporates Dice loss. and binary cross-entropy loss Conduct training as follows: ， in, This represents the prediction output of the semantic calibration module. .

Images