Method and system for automatic prompt-based generation and policy-gated object segmentation

Abstract

The application discloses a target segmentation method and system based on automatic prompt generation and policy gating, and relates to the field of image segmentation. The application determines the reliability of RGB information, Depth information and historical mask information through an imaging quality-operation state joint evaluation vector, automatically generates a prompt, and dynamically determines a segmentation strategy. After a model outputs a candidate mask, the model confidence, boundary consistency, region coherence, inter-frame stability, abnormal region coverage and operation delay evaluation results are used to re-generate a prompt, adjust an ROI, switch a strategy or perform boundary correction if necessary, and the final candidate target segmentation result, result quality score and quality abnormal type are fed back to a subsequent frame. The closed loop uses camera imaging quality and system operation state to control the prompt input, calling range, calling frequency, input resolution and feedback update mode of the basic model, thereby improving the stability, accuracy and real-time performance of target segmentation under complex imaging conditions.

Description

Technical Field

[0001] This invention relates to the field of image segmentation, and specifically to a target segmentation method and system based on automatic prompt generation and policy gating. Background Technology

[0002] Segmentation-aiding basic models such as SAM (Segment Anything Model) and SAM2 (Segment Anything Model 2) can generate segmentation results based on point, bounding box, or mask cues and can be used for image and video object segmentation. However, directly calling these models in continuous camera systems still has the following problems: First, the cues usually rely on manual interaction or external detectors, which is difficult to meet the requirements of real-time continuous operation; Second, the reliability of image edges and depth boundaries fluctuates greatly in scenarios such as reflection, transparency, occlusion, low light, depth holes, or RGB-D registration deviations in binocular, ToF (Time-of-Flight) cameras, structured light, RGB-D, endoscopes, microscopes, and industrial cameras; Third, existing automatic cues methods usually do not uniformly apply depth effectiveness, depth confidence, RGB-D edge consistency, reflective areas, depth holes, stability of historical masks (i.e., historical segmentation results), and system computing power status to cue generation, model call control, and result feedback. This makes existing cue-based segmentation models rely on manual cues or external detectors in continuous camera systems, making it difficult to automatically adjust cues and invocation strategies based on changes in camera image quality. Furthermore, they are prone to missegmentation or excessive computational overhead in environments with deep holes, reflections, occlusions, small targets, and low computing power. Summary of the Invention

[0003] To address the aforementioned shortcomings in existing technologies, the target segmentation method and system based on automatic prompting generation and policy gating provided by this invention solves the problems of missegmentation or excessive computational overhead that can easily occur in existing prompting-based segmentation models in continuous camera systems.

[0004] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A target segmentation method based on automatic prompt generation and policy gating is provided, which includes the following steps: Acquire image data captured by the camera, and obtain the corresponding image information and imaging status markers; Based on image information and imaging status markers, the camera imaging quality index and system operation status index corresponding to the image data are calculated, and then a joint evaluation vector of imaging quality and operation status is constructed. Candidate regions for the corresponding image data are generated based on the joint evaluation vector, and the reliability of the candidate regions is calculated. Based on the candidate region set, candidate region reliability, and joint evaluation vector, prompt information is generated for the promptable segmentation base model; wherein the prompt information includes at least one of positive point prompts, negative point prompts, bounding box prompts, and mask prompts; The adaptation score of the candidate segmentation strategy is calculated based on the joint evaluation vector and the reliability of the candidate region, and then the target segmentation strategy and its content are determined. The strategy content includes at least one of the following: prompt type, prompt location, call region, call frequency, input resolution, base model type and segmentation mode. Input the prompt information into the prompt segmentation base model, and perform target segmentation in the candidate region according to the strategy content to obtain the candidate target segmentation results; The quality score is calculated based on at least one of the following: confidence level, boundary consistency, region coherence, inter-frame stability, and abnormal region coverage ratio, according to the output of the promptable segmentation base model. When the quality score is lower than a preset threshold, credible boundary correction, morphological correction, and / or regeneration of prompt information are performed until the quality score is greater than or equal to the preset threshold. When the quality score is greater than or equal to the preset threshold, the candidate target segmentation result is confirmed.

[0005] A system is provided for implementing a target segmentation method based on automatic prompt generation and policy gating, comprising: The image acquisition module is used to acquire image data captured by the camera and obtain image information corresponding to the image data; The marker generation module is used to obtain the imaging state markers corresponding to the image data; The joint evaluation vector generation module is used to calculate the camera imaging quality index and system operation status index corresponding to the image data based on image information and imaging status markers, and then construct the joint evaluation vector of imaging quality-operation status. The candidate region generation module is used to generate candidate regions for the corresponding image data based on the joint evaluation vector, and to calculate the reliability of the candidate regions. The prompt generation module is used to generate prompt information for the promptable segmentation base model based on the candidate region set, candidate region reliability, and joint evaluation vector; wherein the prompt information includes at least one of positive point prompts, negative point prompts, bounding box prompts, and mask prompts; The strategy gating module is used to calculate the adaptation score of the candidate segmentation strategy based on the joint evaluation vector and the reliability of the candidate region, and then determine the target segmentation strategy and its strategy content; wherein the strategy content includes at least one of the following: prompt type, prompt location, call region, call frequency, input resolution, base model type and segmentation mode; The segmentation execution module is used to input prompt information into the promptable segmentation base model, and perform target segmentation in the candidate region according to the strategy content to obtain the candidate target segmentation results; The quality evaluation and feedback module is used to calculate a quality score based on at least one of the following: confidence level, boundary consistency, regional coherence, inter-frame stability, and abnormal region coverage ratio of the output model of the promptable segmentation base model. When the result quality score is lower than a preset threshold, it performs credible boundary correction, morphological correction, and / or regenerates prompt information until the result quality score is greater than or equal to the preset threshold. When the result quality score is greater than or equal to the preset threshold, it confirms the candidate target segmentation result. It is also used to feed back the final candidate target segmentation result, result quality score, and quality anomaly type to the candidate region generation, prompt information generation, and policy gating process of subsequent frames. The quality anomaly type includes at least one of the following: insufficient model output confidence, insufficient boundary consistency, insufficient regional coherence, insufficient inter-frame stability, and a preset coverage ratio threshold for abnormal region coverage ratio.

[0006] The beneficial effects of this invention are as follows: 1. This invention constructs a joint evaluation vector using camera imaging quality and system operating status without changing the network structure and weights of the basic model for prompting segmentation. Based on this joint evaluation vector, it automatically generates prompt information, selects segmentation strategies, and controls the model calling region and calling frequency, thereby improving the segmentation stability, accuracy, and real-time performance under complex imaging conditions.

[0007] 2. This invention eliminates the need for manual frame-by-frame clicking or selection, and can automatically generate positive points, negative points, box prompts, and mask prompts based on the joint evaluation vector.

[0008] 3. In scenarios involving reflection, occlusion, transparency, deep holes, and inconsistent RGB-D edges, this invention can automatically generate negative point prompts or switch the RGB / Depth priority strategy.

[0009] 4. This invention improves robustness under complex imaging conditions by controlling the model's calling area, calling frequency, and input resolution based on the ROI area, the number of candidate targets, and the computing power status.

[0010] 5. This invention feeds back the final candidate target segmentation result, result quality score, and quality anomaly type of the current frame to the prompt generation and policy gating of subsequent frames, forming a feedback closed loop.

[0011] 6. The basic model for suggestive segmentation of the present invention is not limited to SAM2, the deployment location is not limited to local, edge or cloud, and the components of the joint evaluation vector are not required to be all present, which makes the scope of application of the present invention wider. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the method. Detailed Implementation

[0013] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0014] Example 1: like Figure 1 As shown, the target segmentation method based on automatic prompt generation and policy gating includes the following steps: S1. Acquire image data captured by the camera, and obtain the image information and imaging status markers corresponding to the image data; In this embodiment, the camera can be an RGB camera, a binocular camera, a depth camera, a structured light camera, a ToF camera, an RGB-D camera, a multi-view camera, a microscope camera, a surgical endoscope camera, an industrial camera, or an imaging system composed of an RGB camera combined with a depth estimation module. Correspondingly, the image data acquired by the camera includes at least one of the following: RGB image, grayscale image, depth map, disparity map, infrared image, 3D point cloud, left / right eye image, and depth estimation result.

[0015] Image information includes at least one of image edges, depth edges, point cloud connected regions, reflective regions, depth hole regions, occluded regions, historical target regions, and adjacent target regions; Imaging status markers include at least one of the following: effective depth markers, reflective area markers, depth hole markers, occlusion markers, and RGB-D edge consistency markers. Imaging status markers can be provided directly by the camera, depth sensor, or camera SDK, or calculated and generated by the processor based on RGB images, depth maps, disparity maps, point clouds, depth confidence maps, historical segmentation results, or inter-frame motion information.

[0016] It should be noted that when RGB candidate region generation or RGB priority segmentation strategy is enabled, the image data includes two-dimensional imaging data and the image information includes image edges; When depth candidate region generation or depth-first segmentation strategy is enabled, the image data includes depth-related data and depth confidence maps or depth confidence information calculated from the depth-related data; the image information includes depth edges or point cloud connected regions; the imaging status markers include depth valid markers and depth hole markers. When RGB-D edge consistency evaluation is enabled, the image data includes both two-dimensional imaging data and depth-related data. The image information includes image edges and depth edges, and the imaging status marker includes RGB-D edge consistency markers. When generating negative point prompts or adjusting segmentation strategies based on reflective areas, the image information includes reflective areas, and the imaging status markers include reflective area markers. When performing video memory segmentation, history mask propagation, or keyframe re-cue, the image information includes the historical target region, and the imaging status markers include occlusion markers. Image information may also include at least one of deep hole regions, occluded regions, and adjacent target regions.

[0017] For example, the terminology and data sources for each imaging state label are shown in Table 1.

[0018] Table 1: Terminology and Data Sources

[0019] S2. Calculate the camera imaging quality index and system operation status index corresponding to the image data based on image information and imaging status markers, and then construct a joint evaluation vector of imaging quality and operation status. In this embodiment, the camera imaging quality metrics include at least one of depth efficiency, depth confidence, RGB edge reliability, RGB-D edge consistency, occlusion probability, reflectivity, texture intensity, depth hole rate, and inter-frame stability. Each component can be provided directly by the camera SDK, depth sensor, or processor, or it can be obtained by the processor based on statistics from the RGB image, depth map, disparity map, point cloud, historical segmentation results, edge map, and operating status.

[0020] It should be noted that when depth candidate region generation or depth-first segmentation strategy is enabled, the camera imaging quality metrics include depth efficiency and depth confidence, as well as RGB-D edge consistency. When RGB candidate region generation or RGB priority segmentation strategy is enabled, camera imaging quality metrics include texture intensity, RGB edge reliability, and reflectivity. When video memory segmentation, history mask propagation, or keyframe re-cue strategies are enabled, camera imaging quality metrics include inter-frame stability and occlusion probability.

[0021] The joint evaluation vector includes at least one camera imaging quality index and one system operating status index.

[0022] For details, see Table 2 for the description of the imaging quality indicators of each camera.

[0023] Table 2: Camera Image Quality Indicators and Explanations

[0024] System operating status metrics include at least one of the following: ROI area ratio, number of candidate targets, system frame rate, system latency pressure, current inference latency, latency budget, GPU usage, CPU usage, memory usage, video memory usage, and power consumption.

[0025] When the local ROI segmentation strategy or the candidate region batch segmentation strategy is enabled, the system operation status indicators include at least one of ROI area ratio and candidate target number; When the low-computing-power mode segmentation strategy is enabled, the system operation status indicators include at least one of the following: system frame rate, current inference latency, latency budget, GPU usage, CPU usage, memory usage, video memory usage, and power consumption. The system latency pressure is determined based on the relationship between the current inference latency and the latency budget. The closer the current inference latency is to or exceeds the latency budget, the higher the system latency pressure.

[0026] It should be noted that the joint evaluation vector in this embodiment is an engineering state variable, used to transform camera imaging quality and system operating status into control inputs for candidate region generation, automatic prompt generation, strategy gating, and feedback updates. In one implementation, the joint evaluation vector is constructed using a method of "no fixed formula + optional calculation method + threshold gating logic": each component can be obtained through normalized statistics, threshold judgment, segmented mapping, sliding window historical statistics, rule table mapping, or weighted summation; based on the comparison results of each component with preset thresholds, dynamic thresholds, or calibrated thresholds, it can be determined whether to enable corresponding prompts, increase or decrease the adaptation score of a certain segmentation strategy, or switch the model calling mode.

[0027] Each threshold can be determined based on camera type, target size, application scenario, historical frame statistics, or user calibration data, and can be dynamically updated during operation. When the camera does not output depth maps, depth confidence maps, or other imaging state markers, the corresponding components can be set to null values, default values, or obtained from RGB images, monocular depth estimation, historical masks, optical flow, or temporal stability estimation.

[0028] In particular, the explanations of some English terms involved in this embodiment are shown in Table 3.

[0029] Table 3: Explanation of English Terminology

[0030] It should be noted that in this embodiment, the joint evaluation vector, candidate region, reliability, etc., from input to obtaining the segmentation result and feeding back information to subsequent frames are collectively referred to as QV-PromptGate. Its functions include: determining the segmentation strategy based on the joint evaluation vector, generating positive / negative points / box hints / coarse mask hints, and feeding back the results to subsequent frames.

[0031] S3. Generate candidate regions for the corresponding image data based on the joint evaluation vector, and calculate the reliability of the candidate regions; In this implementation, candidate regions can be generated from one or more of depth candidate regions, RGB candidate regions, and historical candidate regions. Depth candidate regions can be obtained based on effective depth regions and connected component analysis; RGB candidate regions can be obtained based on color, texture, edge closure, and hole filling; historical candidate regions can be obtained from the mask dilation, propagation, or video memory results of the previous frame.

[0032] When both the depth effectiveness rate and depth confidence in the joint evaluation vector meet the preset conditions, a binary map of reliable depth regions is generated based on the depth effective markers and the depth confidence map. Connectivity analysis is then performed on the binary map of reliable depth regions to obtain an initial set of candidate depth regions. The reliability of the initial candidate depth regions is calculated based on their depth effectiveness rate, average depth confidence, and corresponding weights. The depth effectiveness rate within the initial candidate depth regions is the ratio of the number of effective depth pixels in that region to the total number of pixels in that region. When the depth efficiency in the joint evaluation vector does not meet the preset conditions, an initial RGB candidate region is generated; the reliability of the initial RGB candidate region is calculated based on the region texture intensity, RGB edge reliability and reflectivity of the initial RGB candidate region; the RGB edge reliability is determined according to the edge closure degree of the initial RGB candidate region. For video scenes with consecutive frames, the final segmentation result of the previous frame is used. By combining inter-frame motion information for propagation, the motion estimation result of the current frame can be obtained. Motion estimation results for the current frame The initial historical candidate regions are expanded to obtain the initial historical candidate regions. Based on inter-frame stability, occlusion probability and corresponding weights, the reliability of the initial historical candidate regions is calculated. The intersection of the initial depth candidate region, the initial RGB candidate region, and the initial historical candidate region is taken to obtain the final candidate region set, which is used as the candidate region for the corresponding image data. The reliability of the final candidate region is obtained by weighted summing of the reliability of the initial depth candidate region, the initial RGB candidate region, and the initial historical candidate region.

[0033] For example, the joint evaluation vector of the current frame is first obtained:

[0034] in, This indicates depth efficiency. Indicates the depth of confidence. Indicates RGB-D edge consistency. Indicates the depth void ratio. Indicates the proportion of reflective area. Indicates the stability of historical goals. This indicates pressure regarding the number of candidate targets. This indicates system latency pressure.

[0035] When the depth efficiency and depth confidence in the joint evaluation vector meet the preset conditions, a depth candidate region is generated.

[0036] First, based on the depth-effective marking and depth confidence plot Generate a binary map of a reliable region with high depth:

[0037] in, Represents a binary map of a highly reliable region. Indicates a valid depth marker. Indicates the depth of confidence. This represents the confidence threshold. Subsequently, connected component analysis is performed on the binary map of the reliable depth regions to obtain an initial set of depth candidate regions:

[0038] in, Indicates the first n Initial depth candidate regions.

[0039] For each initial depth candidate region, its reliability is calculated:

[0040] in, For the first i The reliability of the initial depth candidate region Indicates the first i The proportion of effective depths within the initial depth candidate region. Indicates the first i The average depth confidence of the initial depth candidate regions Indicates the first i The proportion of depth holes in the initial depth candidate regions This represents the weighting parameter. The formula explains that a region is more suitable for generating candidate regions when it has higher effective depth, higher depth confidence, and fewer missing depth values.

[0041] when or If the reliability of the current depth information decreases, the weight of the depth candidate region is reduced, and RGB candidate region generation is enabled. RGB candidate regions are obtained based on RGB edge information, color consistency, local texture intensity, and region closure.

[0042] in Represents the initial set of RGB candidate regions; Indicates the first m One initial RGB candidate region; For each initial RGB candidate region, calculate its reliability:

[0043] in, For the first i Reliability of the initial RGB candidate regions Indicates the first i The region texture intensity of the initial RGB candidate regions Indicates the first i RGB edge reliability of the initial RGB candidate regions Indicates the first i The reflectance ratio of the initial RGB candidate regions. When the texture is rich, the boundaries are closed, and the reflectance ratio is low, the reliability of the RGB candidate regions is improved.

[0044] For video scenes with consecutive frames, the final segmentation result of the previous frame is used. By combining inter-frame motion information for propagation, the motion estimation result of the current frame can be obtained. :

[0045] in, This represents the frame motion estimation result. This represents the propagation operation that maps the final segmentation result of the previous frame to the coordinate position of the current frame based on the inter-frame motion estimation result; Motion estimation results for the current frame The execution area is expanded to obtain the initial historical candidate area. :

[0046] in, k Represents morphological extension parameters, This indicates that a morphological dilation region expansion operation is performed on the input region; the reliability of the initial historical candidate region is determined based on inter-frame stability and occlusion probability; where inter-frame stability is calculated based on the degree of regional overlap, boundary offset, and inter-frame motion consistency between the current frame candidate region and the initial historical candidate region.

[0047] The final set of candidate regions is: ; For any candidate region Overall reliability can be calculated:

[0048] in, Indicates the depth of candidate weights. Indicates the RGB candidate weights. This represents the historical candidate weights, which are dynamically adjusted by the joint evaluation vector. Specifically, when depth efficiency is high, depth void ratio is low, and RGB-D edge consistency is high... The priority of depth candidate regions should be increased; when the depth void ratio increases and the depth confidence decreases, The priority of RGB candidate regions should be increased.

[0049] S4. Generate prompting information for the promptable segmentation base model based on the candidate region set, candidate region reliability, and joint evaluation vector; wherein the prompting information includes at least one of positive point prompting, negative point prompting, bounding box prompting, and mask prompting; The on-time prompt is generated based on the distance transformation result of pixels within the candidate region and pixel reliability, wherein pixel reliability is determined by at least one of depth confidence, reflective area marker, depth hole marker and distance to boundary; Negative point hints are obtained from at least one of the following: occluded areas, reflective areas, deep hole areas, RGB-D edge inconsistency areas, or adjacent target areas. The number of negative point hints is dynamically determined based on the reflective ratio, deep hole rate, or RGB-D edge inconsistency degree within the candidate area. RGB-D edge inconsistency areas are determined based on RGB-D edge consistency markers and are regions formed by pixels or areas where the RGB edges and depth edges do not meet the preset edge consistency conditions. The bounding box is generated from the bounding box of the candidate region; the extended margin of the bounding box is dynamically determined based on the region reliability score, reflectivity, or depth void ratio. The mask prompt consists of a coarse mask generated from candidate regions, deep connected regions, point cloud connected regions, historical target regions, or video memory results, and is enabled when the reliability score is higher than a preset threshold and the depth void ratio is lower than a preset threshold.

[0050] S5. Calculate the fit score of the candidate segmentation strategy based on the joint evaluation vector and the reliability of the candidate region, and then determine the target segmentation strategy and its content; wherein the strategy content includes at least one of the following: prompt type, prompt location, calling region, calling frequency, input resolution, basic model type and segmentation mode; In this embodiment, the candidate segmentation strategies include at least two of the following: full-image segmentation strategy, local ROI segmentation strategy, RGB-first segmentation strategy, Depth-first segmentation strategy, video memory segmentation strategy, keyframe re-encoding segmentation strategy, candidate region batch segmentation strategy, and low-computing-power mode segmentation strategy. The adaptation score of the Depth-first segmentation strategy is determined by depth efficiency, depth confidence, depth hole rate, and RGB-D edge consistency. The adaptation score of the RGB priority segmentation strategy is determined by RGB edge reliability, texture intensity, and reflectivity. Among them, RGB edge reliability is determined by the edge closure degree of the candidate region, which is used to characterize the completeness of the closed contour formed by the edge of the candidate region. The adaptation scores of the local ROI segmentation strategy, the candidate region batch segmentation strategy, and the low computing power mode segmentation strategy are all determined by the ROI area ratio, the number of candidate targets, the reliability of the candidate region, and the system latency pressure. The adaptation scores of the video memory segmentation strategy, the history mask propagation strategy, and the keyframe re-encoding segmentation strategy are calculated based on inter-frame stability, the quality score of the previous frame result, and the occlusion probability. When the depth efficiency, depth confidence or RGB-D edge consistency are higher than the preset threshold, and the depth hole rate or reflectivity is lower than the pre-review threshold, the adaptation score of the Depth-first segmentation strategy is increased. When the texture intensity is higher than the preset threshold, the depth confidence is lower than the preset threshold, the depth hole rate is higher than the preset threshold, or the RGB-D edge consistency is lower than the preset threshold, the adaptation score of the RGB priority segmentation strategy is increased. When the proportion of candidate region area is lower than the preset threshold, the reliability score of candidate region is higher than the preset threshold, the system latency pressure is higher than the preset threshold, or the number of candidate targets is higher than the preset threshold, the adaptation score of the local ROI segmentation strategy is increased. When the inter-frame stability is higher than the preset inter-frame stability threshold, the quality score of the previous frame result is higher than the preset result quality threshold, and the occlusion probability is lower than the preset occlusion probability threshold, the adaptation score of the video memory segmentation strategy or the history mask propagation strategy is increased. When the quality score of the previous frame is lower than the preset threshold, the inter-frame stability is lower than the preset threshold, the occlusion probability is higher than the preset threshold, the depth hole rate is higher than the preset threshold, or the candidate region reliability score is lower than the preset threshold, the key frame re-segmentation strategy is triggered. When the number of candidate regions or candidate targets reaches a preset threshold, the candidate region batch segmentation, ROI segmentation strategy, or low computing power mode segmentation strategy is triggered. When system latency pressure, number of candidate targets, ROI area ratio, CPU usage, GPU usage, memory usage, VRAM usage, or power consumption exceeds a preset threshold, a low-computing-power segmentation strategy is triggered, which reduces the input resolution, decreases the number of candidate regions, shrinks the calling region, reduces the model calling frequency, and / or switches to a lightweight, prompt-enabled segmentation base model.

[0051] S6. Input the prompt information into the promptable segmentation base model, and perform target segmentation in the candidate region according to the strategy content to obtain the candidate target segmentation results; S7. Calculate the quality score of the output result based on at least one of the following: confidence level, boundary consistency, regional coherence, inter-frame stability, and abnormal region coverage ratio, according to the promptable segmentation base model output; when the result quality score is lower than a preset threshold, perform credible boundary correction, morphological correction, and / or regenerate prompt information until the result quality score is greater than or equal to the preset threshold; when the result quality score is greater than or equal to the preset threshold, confirm the candidate target segmentation result.

[0052] In one implementation, the expression for calculating the result quality score is:

[0053] in, This indicates the confidence level of the model output. This indicates boundary consistency, which is the degree of consistency between the mask boundary and the reliable boundary of the candidate target segmentation result. Indicates the stability of consecutive frames. Indicates the coverage ratio of abnormal areas. When When the current segmentation result meets the quality requirements, it can be output directly; when... This indicates that the current segmentation result has degraded in quality and requires further adaptive correction.

[0054] Specifically, in this embodiment, the source of the credible boundary E can be determined based on the comparison between the adaptation scores of the Depth-first segmentation strategy and the RGB-first segmentation strategy. The credible boundary E is used to evaluate the boundary consistency of the candidate target segmentation results and serves as the basis for performing credible boundary correction.

[0055] When the adaptation score of the Depth-first segmentation strategy is higher than that of the RGB-first segmentation strategy, the depth edge is taken as the reliable boundary. .in, Indicates depth edge; When the adaptation score of the RGB-first segmentation strategy is higher than that of the Depth-first segmentation strategy, the RGB edges are taken as reliable boundaries. .in, Indicates the RGB edge.

[0056] Morphological correction includes at least one of erosion, dilation, opening, and closing operations. Erosion is used to remove over-segmented regions caused by reflections, background textures, or mis-spotting of boundaries; dilation is used to restore missing target regions caused by depth holes, low texture, or occlusion; opening is used to remove isolated noise regions; and closing is used to fill in internal holes in the target, improving the coherence of the target region.

[0057] This embodiment can select one or more morphological operations to generate multiple correction candidates based on the anomaly type of the current segmentation result:

[0058] Each of them This represents the candidate segmentation results obtained by using different correction methods.

[0059] Subsequently, a corrected quality score is calculated based on at least one of the following indicators: boundary consistency, regional coherence, inter-frame stability, and coverage ratio of anomalous regions in the candidate results. And select the correction result with the highest score:

[0060] The final candidate target segmentation result is the candidate target segmentation result obtained by performing morphological correction.

[0061] S8. Feed back the final candidate target segmentation result, result quality score and quality anomaly type of the current frame to the candidate region generation, prompt information generation and strategy content determination process of subsequent frames.

[0062] The quality anomaly type is determined based on the evaluation index that fails to meet the preset quality requirements during the result quality evaluation process, including at least one of the following: insufficient model output confidence, insufficient boundary consistency, insufficient regional coherence, insufficient inter-frame stability, and abnormal region coverage ratio exceeding the preset coverage ratio threshold.

[0063] In this embodiment, for the current frame that receives the final candidate target segmentation result, result quality score, and quality anomaly type from the previous frame, the following operations are performed: Based on the inter-frame stability and occlusion probability of the current frame, as well as the quality score and quality anomaly type of the previous frame, determine whether the current frame meets the key frame triggering conditions. When the current frame does not meet the keyframe triggering conditions, the final candidate target segmentation result of the previous frame is used as the basis for the generation of candidate regions and initialization of prompts in the current frame. The current frame prompt information is generated and target segmentation is performed using the historical mask propagation strategy, video memory segmentation strategy or local ROI segmentation strategy. When the current frame meets the keyframe triggering condition, the joint evaluation vector construction, candidate region generation, prompt information generation, target segmentation strategy determination, and target segmentation steps are re-executed based on the current frame to obtain the candidate target segmentation result for the current frame.

[0064] In this embodiment, when the target state remains stable between consecutive frames, historical information is used preferentially to reduce redundant calculations.

[0065] Specifically, when the following conditions are met: high inter-frame stability, high target segmentation quality score in the current frame, low occlusion probability, and satisfactory reliability of the current candidate region, the current target tracking state is maintained, and the next frame's prompt information is generated using historical mask propagation, video memory strategy, or ROI local segmentation strategy. The final candidate target segmentation result of the previous frame can be used as the basis for candidate region generation and prompt initialization in the next frame, thereby reducing the number of times full-image segmentation is re-executed and lowering computational overhead.

[0066] When the target state changes significantly in consecutive frames, historical feedback information may no longer guarantee the effectiveness of the prompts, thus triggering a keyframe to provide a new prompt.

[0067] In one implementation, keyframe judgment indicators are generated based on inter-frame stability, boundary consistency, and occlusion status.

[0068] in, Indicates the stability of the historical target area. This indicates the boundary consistency of the candidate target segmentation results in the current frame. Indicates the occlusion probability of the current frame. , , This indicates the corresponding weight. When inter-frame stability decreases, the results of the previous frame cannot be reliably propagated; when boundary consistency decreases, the mask boundary of the candidate target segmentation result in the current frame may shift; when the occlusion probability increases, the appearance or spatial position of the target may change.

[0069] when When this condition is met, it indicates that the current frame meets the keyframe triggering condition. At this time, the re-initialization process is executed, including: (1) Regenerate the initial candidate region set based on the joint evaluation vector of the current frame; (2) Recalculate the reliability of the initial candidate region; (3) Generate positive point hints, negative point hints, box hints or mask hints based on the new initial candidate regions; (4) Re-execute the policy gating and redetermine the segmentation policy, calling region, input resolution or model calling mode used in the current frame.

[0070] In this way, the output of the current frame is not directly passed to the next frame as a fixed template, but rather forms a dynamic feedback state after quality evaluation. The overall process is as follows: Image input → Joint evaluation vector generation → Candidate region generation → Hint generation → Model segmentation → Quality evaluation → Feedback update → Next frame hint and strategy adjustment.

[0071] Therefore, this method can automatically adjust the prompt generation method and strategy gating results of subsequent frames based on changes in imaging quality, target stability, and computational resource status in consecutive frames, thereby improving the stability of continuous segmentation in complex imaging environments.

[0072] Example 2: This embodiment is a further extension of Embodiment 1. This embodiment provides a system for implementing a target segmentation method based on automatic prompt generation and policy gating, which includes: The image acquisition module is used to acquire image data captured by the camera and obtain image information corresponding to the image data; The marker generation module is used to obtain the imaging state markers corresponding to the image data; The joint evaluation vector generation module is used to calculate the camera imaging quality index and system operation status index corresponding to the image data based on image information and imaging status markers, and then construct the joint evaluation vector of imaging quality-operation status. The candidate region generation module is used to generate candidate regions for the corresponding image data based on the joint evaluation vector, and to calculate the reliability of the candidate regions. The prompt generation module is used to generate prompt information for the promptable segmentation base model based on the candidate region set, candidate region reliability, and joint evaluation vector; wherein the prompt information includes at least one of positive point prompts, negative point prompts, bounding box prompts, and mask prompts; The strategy gating module is used to calculate the adaptation score of the candidate segmentation strategy based on the joint evaluation vector and the reliability of the candidate region, and then determine the target segmentation strategy and its strategy content; wherein the strategy content includes at least one of the following: prompt type, prompt location, call region, call frequency, input resolution, base model type and segmentation mode; The segmentation execution module is used to input prompt information into the promptable segmentation base model, and perform target segmentation in the candidate region according to the strategy content to obtain the candidate target segmentation results; The quality evaluation and feedback module is used to calculate a quality score based on at least one of the following: confidence level, boundary consistency, regional coherence, inter-frame stability, and abnormal region coverage ratio of the output model of the promptable segmentation base model. When the result quality score is lower than a preset threshold, it performs credible boundary correction, morphological correction, and / or regenerates prompt information until the result quality score is greater than or equal to the preset threshold. When the result quality score is greater than or equal to the preset threshold, it confirms the candidate target segmentation result. It is also used to feed back the final candidate target segmentation result, result quality score, and quality anomaly type to the candidate region generation, prompt information generation, and policy gating process of subsequent frames. The quality anomaly type includes at least one of the following: insufficient model output confidence, insufficient boundary consistency, insufficient regional coherence, insufficient inter-frame stability, and a preset coverage ratio threshold for abnormal region coverage ratio.

[0073] It should be noted that suggestive segmentation base models include SAM, SAM2, SAM2.1, EdgeSAM, MobileSAM, FastSAM, MedSAM, or other suggestive segmentation base models.

[0074] For example, in a surgical vision-assisted scenario, the camera can be an endoscopic camera, a microscopic camera, an RGB-D camera, a structured light camera, a ToF camera, or a binocular microscopic imaging system. The system calculates the reflectance ratio, depth efficiency, depth hole rate, RGB-D edge consistency, and inter-frame stability. When the reflectance ratio is high but the depth boundary is reliable, a depth-first strategy is selected, generating bounding box cues or coarse mask cues from the depth-connected regions, and generating negative point cues in high-reflectance areas and areas with inconsistent edges. After the model outputs masks of instruments, tissues, blood vessels, or lesions, the system evaluates boundary consistency based on reliable depth edges and triggers keyframe re-cues or boundary correction when consistency is insufficient.

[0075] In industrial inspection or warehouse sorting scenarios, cameras can be RGB-D cameras, ToF cameras, structured light cameras, multi-view cameras, or RGB cameras combined with depth estimation modules. The system selects full image segmentation, local ROI segmentation, or batch segmentation of candidate regions based on target size, number of candidate regions, depth effectiveness, depth confidence, and latency budget. When the number of candidate targets is large and the ROI area is relatively small, the system generates bounding box hints and positive point hints for the candidate targets and batch calls a lightweight hint-enabled segmentation model. When the depth hole rate increases, it switches to an RGB-first strategy and generates negative point hints in depth holes and adjacent target regions.

[0076] In obstacle avoidance or grasping scenarios for mobile robots, the system acquires RGB and depth maps using binocular or RGB-D cameras. If the target is stable across consecutive frames, the system enables video memory or historical mask propagation to reduce the frequency of full model calls; if rapid movement causes a decrease in inter-frame stability, the current frame is set as a keyframe, cues are regenerated, and the full model is called. The system can switch between different sizes of cueable segmentation base models based on latency budget.

[0077] This embodiment also includes localized deployment and real RGB-D experimental verification.

[0078] Specifically, SAM2.1 Tiny is deployed on a local computer as the base model for cue segmentation. This embodiment does not change the network structure, model weights, training data, or loss function of SAM2.1 Tiny, but only adds a joint evaluation vector generation module, a cue generation module, a policy gating module, an ROI invocation module, and a quality evaluation and feedback module to its input and output sides. The above modifications do not involve the internal parameters of the base model, but through a closed loop of "image quality evaluation - cue generation - policy gating - base model invocation - result scoring - feedback update", the base model can automatically select a more appropriate cue and invocation method in reflective, transparent, deep hole, small target, and low computing power scenarios.

[0079] The experimental data used was publicly available RGB-D data from the Washington RGB-D Object Dataset. As shown in Table 4, this dataset contains RGB images, depth maps, and target masks captured by a real Kinect, and is used only to illustrate the feasibility and technical effects of the present invention, without limiting the application scenario, camera type, base model model, or scope of protection.

[0080] Table 4: Composition of Supplementary Experimental Data for Real RGB-D

[0081] The actual RGB-D experimental results are shown in Table 5.

[0082] Table 5: Summary of Real RGB-D Experimental Results

[0083] Experimental Results Analysis: The QV-PromptGate mechanism of this invention demonstrates significant technical effects in various scenarios. The baseline is uniformly set to "fixed full-image SAM2.1 Tiny (no joint evaluation vector, no policy gating, no ROI, no feedback)". First, in deep hole scenarios, the baseline IoU is only 0.0781, while the qv_promptgate mechanism improves the IoU to 0.7366 (an improvement of 58.55%), while significantly reducing abnormal overlap. Second, in reflective scenarios, the baseline IoU is 0.8562, while the qv_promptgate mechanism improves the IoU to 0.8573, significantly reducing the abnormal overlap rate and optimizing the Boundary F1 score, demonstrating the fine adaptability of policy gating under complex lighting conditions. Furthermore, in low-computational-power mode, compared to fixed full-image access, the QV-PromptGate+ROI mechanism of this invention reduces the access area by 97.56% to 0.0244, while improving IoU by 29.68% to 0.5887 and Boundary F1 by 70.11%. Overall, this invention does not alter the original model structure, employs zero-retraining, and achieves external control solely through the joint evaluation vector Q_t, candidate region generation, QV-PromptGate closure, and result feedback. It significantly improves segmentation performance and resource utilization even in complex scenarios such as deep holes, low computational power, and reflectivity.

[0084] In summary, this invention establishes a regularized control link between the camera data processing module and the cue-based segmentation model. It uses a joint evaluation vector of imaging quality and system operating status to determine the reliability of RGB, Depth, and historical mask information, automatically generating positive, negative, bounding box, or coarse mask cues. It dynamically determines strategies such as full-image segmentation, local ROI segmentation, RGB priority, Depth priority, video memory, keyframe re-cues, candidate region batch segmentation, or low-computing-power mode. After the model outputs candidate masks, it regenerates cues, adjusts ROIs, switches strategies, or performs boundary corrections as needed, based on model confidence, boundary consistency, regional coherence, inter-frame stability, abnormal region coverage, and runtime latency evaluation results. Finally, it feeds back the final candidate target segmentation results, result quality scores, and quality anomaly types to subsequent frames. This closed-loop approach does not rely on the model structure of SAM, SAM2, or other cue-based segmentation models as innovation. Instead, it utilizes camera imaging quality and system operating status to control the cue input, calling range, calling frequency, input resolution, and feedback update method of the basic model, thereby improving the stability, accuracy, and real-time performance of target segmentation under complex imaging conditions.

Claims

1. A target segmentation method based on automatic prompt generation and policy gating, characterized in that, Includes the following steps: Acquire image data captured by the camera, and obtain the corresponding image information and imaging status markers; Based on image information and imaging status markers, the camera imaging quality index and system operation status index corresponding to the image data are calculated, and then a joint evaluation vector of imaging quality and operation status is constructed. Candidate regions for the corresponding image data are generated based on the joint evaluation vector, and the reliability of the candidate regions is calculated. Based on the candidate region set, candidate region reliability, and joint evaluation vector, prompt information is generated for the promptable segmentation base model; The prompts include at least one of the following: positive point prompts, negative point prompts, box prompts, and mask prompts; The fit score of the candidate segmentation strategy is calculated based on the joint evaluation vector and the reliability of the candidate region, and then the target segmentation strategy and its content are determined. The strategy includes at least one of the following: prompt type, prompt location, call area, call frequency, input resolution, base model type, and segmentation mode; Input the prompt information into the prompt segmentation base model, and perform target segmentation in the candidate region according to the strategy content to obtain the candidate target segmentation results; A quality score is calculated based on at least one of the following: confidence, boundary consistency, region coherence, inter-frame stability, and proportion of abnormal region coverage, from the output of the suggestible segmentation base model. When the result quality score is lower than the preset threshold, perform confidence boundary correction, morphological correction and / or regenerate prompt information until the result quality score is greater than or equal to the preset threshold. When the result quality score is greater than or equal to the preset threshold, the candidate target segmentation result is confirmed.

2. The target segmentation method based on automatic prompt generation and policy gating according to claim 1, characterized in that, The image data acquired by the camera includes two-dimensional imaging data and depth-related data; Two-dimensional imaging data includes at least one of RGB images, grayscale images, and infrared images; Depth-related data include at least one of the following: depth map, disparity map, 3D point cloud, left / right eye image, and depth estimation results; At least one of the following: RGB image, grayscale image, depth map, disparity map, infrared image, 3D point cloud, left / right eye image, and depth estimation result; Image information includes at least one of image edges, depth edges, point cloud connected regions, reflective regions, depth hole regions, occluded regions, historical target regions, and adjacent target regions; Imaging status markers include at least one of the following: effective depth markers, reflective area markers, depth hole markers, occlusion markers, and RGB-D edge consistency markers.

3. The target segmentation method based on automatic prompt generation and policy gating according to claim 1, characterized in that, Camera imaging quality metrics include at least one of the following: depth efficiency, depth confidence, RGB edge reliability, RGB-D edge consistency, occlusion probability, reflectivity, texture intensity, depth hole rate, and inter-frame stability. System operating status metrics include at least one of the following: ROI area ratio, number of candidate targets, system frame rate, system latency pressure, current inference latency, latency budget, GPU usage, CPU usage, memory usage, video memory usage, and power consumption. The joint evaluation vector includes at least one camera imaging quality index and at least one system operating status index. Specific construction methods include: By normalizing statistics, threshold judgment, segmented mapping, sliding window historical statistics, rule table mapping and / or weighted summation, specific values ​​of camera imaging quality indicators and / or system operating status indicators are obtained; the obtained indicator values ​​are used as elements of a vector to obtain a joint evaluation vector of imaging quality and operating status.

4. The target segmentation method based on automatic prompt generation and policy gating according to claim 1, characterized in that, The specific methods for generating candidate regions for the corresponding image data based on the joint evaluation vector and calculating the reliability of the candidate regions include: When both the depth effectiveness rate and depth confidence in the joint evaluation vector meet the preset conditions, a binary map of reliable depth regions is generated based on the depth effective markers and the depth confidence map. Connectivity analysis is then performed on the binary map of reliable depth regions to obtain an initial set of candidate depth regions. The reliability of the initial candidate depth regions is calculated based on their depth effectiveness rate, average depth confidence, and corresponding weights. The depth effectiveness rate within the initial candidate depth regions is the ratio of the number of effective depth pixels in that region to the total number of pixels in that region. When the depth efficiency in the joint evaluation vector does not meet the preset conditions, an initial RGB candidate region is generated; the reliability of the initial RGB candidate region is calculated based on the region texture intensity, RGB edge reliability and reflectivity of the initial RGB candidate region; the RGB edge reliability is determined according to the edge closure degree of the initial RGB candidate region. For video scenes with consecutive frames, the final segmentation result of the previous frame is used. By combining inter-frame motion information for propagation, the motion estimation result of the current frame can be obtained. Motion estimation results for the current frame The initial historical candidate regions are expanded to obtain the initial historical candidate regions. Based on inter-frame stability, occlusion probability and corresponding weights, the reliability of the initial historical candidate regions is calculated. The intersection of the initial depth candidate region, the initial RGB candidate region, and the initial historical candidate region is taken to obtain the final candidate region set, which is used as the candidate region for the corresponding image data. The reliability of the final candidate region is obtained by weighted summing of the reliability of the initial depth candidate region, the initial RGB candidate region, and the initial historical candidate region.

5. The target segmentation method based on automatic prompt generation and policy gating according to claim 1, characterized in that, The on-time prompt is generated based on the distance transformation result of pixels within the candidate region and pixel reliability, wherein pixel reliability is determined by at least one of depth confidence, reflective area marker, depth hole marker and distance to boundary; Negative point hints are obtained from at least one of the following samples: occluded areas, reflective areas, deep hole areas, RGB-D edge inconsistency areas, or adjacent target areas; The number of negative points is dynamically determined based on the reflectance ratio, depth hole rate, or RGB-D edge inconsistency within the candidate area; the RGB-D edge inconsistency area is determined based on the RGB-D edge consistency marker, which is the area formed by pixels or areas whose RGB edges and depth edges do not meet the preset edge consistency conditions. The bounding box is generated from the bounding box of the candidate region; the extended margin of the bounding box is dynamically determined based on the region reliability score, reflectivity, or depth void ratio. The mask prompt consists of a coarse mask generated from candidate regions, deep connected regions, point cloud connected regions, historical target regions, or video memory results, and is enabled when the reliability score is higher than a preset threshold and the depth void ratio is lower than a preset threshold.

6. The target segmentation method based on automatic prompt generation and policy gating according to claim 1, characterized in that, Candidate segmentation strategies include at least two of the following: full-image segmentation strategy, local ROI segmentation strategy, RGB-first segmentation strategy, Depth-first segmentation strategy, video memory segmentation strategy, keyframe re-encoding segmentation strategy, candidate region batch segmentation strategy, and low-computing-power mode segmentation strategy. The adaptation score of the Depth-first segmentation strategy is determined by depth efficiency, depth confidence, depth hole rate, and RGB-D edge consistency. The adaptation score of the RGB priority segmentation strategy is determined by RGB edge reliability, texture intensity, and reflectivity. Among them, RGB edge reliability is determined by the edge closure degree of the candidate region, which is used to characterize the completeness of the closed contour formed by the edge of the candidate region. The adaptation scores of the local ROI segmentation strategy, the candidate region batch segmentation strategy, and the low computing power mode segmentation strategy are all determined by the ROI area ratio, the number of candidate targets, the reliability of the candidate region, and the system latency pressure. The adaptation scores of the video memory segmentation strategy, the history mask propagation strategy, and the keyframe re-encoding segmentation strategy are calculated based on inter-frame stability, the quality score of the previous frame result, and the occlusion probability. When the depth efficiency, depth confidence or RGB-D edge consistency are higher than the preset threshold, and the depth hole rate or reflectivity is lower than the pre-review threshold, the adaptation score of the Depth-first segmentation strategy is increased. When the texture intensity is higher than the preset threshold, the depth confidence is lower than the preset threshold, the depth hole rate is higher than the preset threshold, or the RGB-D edge consistency is lower than the preset threshold, the adaptation score of the RGB priority segmentation strategy is increased. When the proportion of candidate region area is lower than the preset threshold, the reliability score of candidate region is higher than the preset threshold, the system latency pressure is higher than the preset threshold, or the number of candidate targets is higher than the preset threshold, the adaptation score of the local ROI segmentation strategy is increased. When the inter-frame stability is higher than the preset inter-frame stability threshold, the quality score of the previous frame result is higher than the preset result quality threshold, and the occlusion probability is lower than the preset occlusion probability threshold, the adaptation score of the video memory segmentation strategy or the history mask propagation strategy is increased. When the quality score of the previous frame is lower than the preset threshold, the inter-frame stability is lower than the preset threshold, the occlusion probability is higher than the preset threshold, the depth hole rate is higher than the preset threshold, or the candidate region reliability score is lower than the preset threshold, the key frame re-segmentation strategy is triggered. When the number of candidate regions or candidate targets reaches a preset threshold, the candidate region batch segmentation, ROI segmentation strategy, or low computing power mode segmentation strategy is triggered. When system latency pressure, number of candidate targets, ROI area ratio, CPU usage, GPU usage, memory usage, VRAM usage, or power consumption exceeds a preset threshold, a low-computing-power segmentation strategy is triggered, which reduces the input resolution, decreases the number of candidate regions, shrinks the calling region, reduces the model calling frequency, and / or switches to a lightweight, prompt-enabled segmentation base model.

7. The target segmentation method based on automatic prompt generation and policy gating according to claim 6, characterized in that, When performing trusted boundary correction, if the adaptation score of the Depth-first segmentation strategy is higher than that of the RGB-first segmentation strategy, then the depth edge is taken as the trusted boundary; if the adaptation score of the RGB-first segmentation strategy is higher than that of the Depth-first segmentation strategy, then the RGB edge is taken as the trusted boundary. And perform boundary correction based on the trusted boundary; When performing morphological corrections, one or more morphological correction operations are selected to generate one or more correction candidate results; The correction quality score is calculated based on at least one of the following indicators: boundary consistency, regional coherence, inter-frame stability, and abnormal region coverage ratio of the corrected candidate results. The corrected result with the highest correction quality score is selected as the candidate target segmentation result obtained by performing morphological correction. The morphological correction operations include erosion, dilation, opening, and closing operations.

8. The target segmentation method based on automatic prompt generation and policy gating according to claim 1, characterized in that, It also includes the following steps: The final candidate target segmentation result, result quality score, and quality anomaly type of the current frame are fed back to the candidate region generation, prompt information generation, and strategy content determination process of subsequent frames; The quality anomaly types include at least one of the following: insufficient model output confidence, insufficient boundary consistency, insufficient regional coherence, insufficient inter-frame stability, and an abnormal region coverage ratio exceeding a preset coverage ratio threshold.

9. The target segmentation method based on automatic prompt generation and policy gating according to claim 8, characterized in that, For the current frame that receives the final candidate target segmentation result, result quality score, and quality anomaly type from the previous frame, perform the following operations: Based on the inter-frame stability and occlusion probability of the current frame, as well as the quality score and quality anomaly type of the previous frame, determine whether the current frame meets the key frame triggering conditions. When the current frame does not meet the keyframe triggering conditions, the final candidate target segmentation result of the previous frame is used as the basis for the generation of candidate regions and initialization of prompts in the current frame. The current frame prompt information is generated and target segmentation is performed using the historical mask propagation strategy, video memory segmentation strategy or local ROI segmentation strategy. When the current frame meets the keyframe triggering condition, the joint evaluation vector construction, candidate region generation, prompt information generation, target segmentation strategy determination, and target segmentation steps are re-executed based on the current frame to obtain the candidate target segmentation result for the current frame.

10. A system for implementing the target segmentation method based on automatic prompt generation and policy gating as described in any one of claims 1 to 9, characterized in that, include: The image acquisition module is used to acquire image data captured by the camera and obtain image information corresponding to the image data; The marker generation module is used to obtain the imaging state markers corresponding to the image data; The joint evaluation vector generation module is used to calculate the camera imaging quality index and system operation status index corresponding to the image data based on image information and imaging status markers, and then construct the joint evaluation vector of imaging quality-operation status. The candidate region generation module is used to generate candidate regions for the corresponding image data based on the joint evaluation vector, and to calculate the reliability of the candidate regions. The prompt generation module is used to generate prompt information for the promptable segmentation base model based on the candidate region set, candidate region reliability, and joint evaluation vector; The prompts include at least one of the following: positive point prompts, negative point prompts, box prompts, and mask prompts; The strategy gating module is used to calculate the adaptation score of the candidate segmentation strategy based on the joint evaluation vector and the reliability of the candidate region, and then determine the target segmentation strategy and its content. The strategy includes at least one of the following: prompt type, prompt location, call area, call frequency, input resolution, base model type, and segmentation mode; The segmentation execution module is used to input prompt information into the promptable segmentation base model, and perform target segmentation in the candidate region according to the strategy content to obtain the candidate target segmentation results; The quality evaluation and feedback module is used to output a quality score based on at least one of the following calculated results: confidence, boundary consistency, regional coherence, inter-frame stability, and abnormal region coverage ratio, according to the promptable segmentation base model. When the result quality score is lower than the preset threshold, perform confidence boundary correction, morphological correction and / or regenerate prompt information until the result quality score is greater than or equal to the preset threshold. When the result quality score is greater than or equal to the preset threshold, the candidate target segmentation result is confirmed; It is also used to feed back the final candidate target segmentation results, result quality scores and quality anomaly types to the candidate region generation, prompt information generation and policy gating processes of subsequent frames; The quality anomaly types include at least one of the following: insufficient model output confidence, insufficient boundary consistency, insufficient regional coherence, insufficient inter-frame stability, and a preset coverage ratio threshold for abnormal regions.

Images