A video stream few-sample segmentation method based on perspective consistent prototype memory

Abstract

The application discloses a video stream few-sample segmentation method based on a perspective consistent prototype memory, and relates to the technical fields of computer vision and video target segmentation. The method acquires a support image, a support mask and a query video, generates a perspective consistent initial visual prototype based on a target region of the support image; analyzes the query video frame by frame, generates a current frame visual prototype, and fuses the initial visual prototype, the current frame visual prototype and historical high-confidence frame prototypes in a prototype memory library to obtain a fused visual prompt to generate a current frame target segmentation mask; calculates a quality score according to a mask area, an inter-frame overlap degree and a prototype similarity, and only writes the current frame visual prototype meeting a preset condition into the prototype memory library; and finally outputs a segmentation mask video and a segmentation visualized video. The application can realize continuous segmentation of a video stream under the condition of providing only a few support samples, and can relieve the problems of segmentation jump, target drift and error accumulation caused by frame-by-frame independent inference.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, video object segmentation, few-shot learning, and basic visual model application technology, specifically to a few-shot segmentation method for video streams based on viewpoint-consistent prototype memory. Background Technology

[0002] Image segmentation is a crucial task in computer vision, aiming to identify target regions in images at the pixel level. With the development of deep learning technology, semantic segmentation, instance segmentation, and interactive segmentation methods have been widely applied in scenarios such as industrial quality inspection, security monitoring, robot vision, drone inspection, medical image analysis, and video content understanding. However, traditional supervised segmentation methods typically rely on large amounts of pixel-level labeled data, resulting in high annotation costs, and their generalization ability is limited when facing new categories, new scenes, or long-tailed targets.

[0003] Few-shot segmentation methods segment similar targets in a query image by providing a small number of labeled support samples, thus reducing reliance on large-scale labeled data. Existing few-shot image segmentation methods typically extract target category prototypes conditioned on support images and support masks, and then match these prototypes with features in the query image to obtain the target segmentation result. In recent years, cue-based base segmentation models and viewpoint-consistent prototype learning methods have further improved the generalization ability of few-shot segmentation. These methods can generate visual cues using target masks in support images and, to some extent, alleviate the prototype offset problem between support images and query images caused by changes in viewpoint, pose, scale, or appearance.

[0004] However, most existing few-shot segmentation methods are geared towards single query images and are difficult to adapt directly to continuous video stream scenarios. In practical applications, query objects often appear in video form, such as surveillance videos, robot camera videos, industrial production line videos, or drone inspection videos. If the video is split into independent images frame by frame and few-shot segmentation is performed on each, the historical information between adjacent frames cannot be fully utilized, which can easily lead to problems such as segmentation result jumps, target drift, weak recovery ability after occlusion, and unstable boundaries between consecutive frames.

[0005] On the other hand, existing video object segmentation methods typically require finely labeled first frames or a large number of video training samples. Some methods rely on complex temporal modeling, optical flow estimation, or video memory networks, resulting in high deployment costs and not necessarily being suitable for few-shot segmentation of new object categories. For scenarios where only one supporting image and its object mask are provided, how to extend the segmentation capability of static few-shot images to video stream input and output without retraining the model on a large scale, and how to utilize historical high-confidence frame information to improve the segmentation stability of subsequent frames, remains a technical problem that needs to be solved in this field.

[0006] Furthermore, during video stream segmentation, if the segmentation results of all historical frames are directly written to the memory, and a frame is mis-segmented due to motion blur, occlusion, lighting changes, or background interference, the erroneous prototype will contaminate the segmentation cues of subsequent frames, leading to error accumulation and target drift. Therefore, it is necessary to design a quality gating mechanism that can determine the reliability of the current frame's segmentation result, writing only high-confidence historical frame prototypes to the memory, thereby reducing the risk of error propagation while utilizing historical information.

[0007] Therefore, how to construct a few-sample segmentation method for video stream input and output, which can generate viewpoint-consistent prototypes based on supporting images and supporting masks, and achieve stable segmentation in continuous video frames through high-confidence historical frame prototype memory and quality gating update mechanism, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0008] The purpose of this invention is to provide a few-sample segmentation method for video streams based on viewpoint-consistent prototype memory, in order to solve the problems of existing few-sample image segmentation methods being unable to directly adapt to video stream input and output, failing to fully utilize historical frame information due to frame-by-frame independent inference, and being prone to segmentation jumps, target drift, and error accumulation in consecutive video frames.

[0009] In this invention, "viewpoint consistency" refers to the ability of the target category prototype to maintain consistency in category semantics and structural features across different viewpoint representations obtained from the supporting image, query video frame, and the supporting image or video frame, even when changes occur in shooting angle, target pose, scale, or local deformation. This viewpoint-consistent prototype reduces prototype shift caused by viewpoint differences between supporting samples and query video frames, enabling the same target category to generate stable segmentation cues under different viewing angles.

[0010] Another objective of this invention is to provide a method for extending video stream processing capabilities during the inference phase without requiring large-scale retraining. This method enables the system to take as input images, masks, and query videos, and output target segmentation masked videos and overlaid visualization videos, thereby improving the ease of deployment of few-sample target segmentation in real-world video scenarios.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows.

[0012] A few-shot segmentation method for video streams based on viewpoint-consistent prototype memory includes the following steps:

[0013] 1) Obtain supporting images, supporting masks, and query videos. The supporting mask is used to identify target regions in the supporting images, and the query videos include consecutive video frames to be segmented.

[0014] 2) Preprocess the supporting image and supporting mask, and generate an initial visual prototype based on the target region in the supporting image; the initial visual prototype is a viewpoint-consistent visual prototype, which is used to represent the semantic and structural features of the target category that remain consistent under different viewpoints, poses or scale changes.

[0015] 3) Perform video stream parsing on the query video, obtain the current video frame in the order of video frames, and perform image preprocessing on the current video frame in the same way as the supported images to generate the visual prototype of the current frame.

[0016] 4) For the current video frame, fuse the initial visual prototype, the current video frame visual prototype, and the historical high-confidence frame prototypes in the prototype memory to generate a fused visual cue. Input the fused visual cue into the segmentation decoder to obtain the target segmentation mask for the current video frame.

[0017] 5) Calculate a quality score based on the current frame visual prototype, the initial visual prototype, and the current frame segmentation mask. The quality score is used to determine whether the current frame visual prototype meets the conditions for writing to the prototype memory.

[0018] 6) When the quality score meets the preset conditions, the visual prototype of the current frame is written into the prototype memory; when the quality score does not meet the preset conditions, writing the visual prototype of the current frame into the prototype memory is prohibited.

[0019] 7) Generate a segmentation mask video based on the frame-by-frame target segmentation mask, and overlay the target segmentation mask with the original video frames to generate a segmentation visualization video.

[0020] Furthermore, in step 2), the viewpoint-consistent visual prototype can be obtained through supporting image target region features, supporting mask constraints, structural feature extraction, and viewpoint-consistent feature alignment. The viewpoint-consistent feature alignment may include feature modeling of the supporting image or its transformed views to enhance the stability of the target prototype under viewpoint changes.

[0021] Furthermore, in step 3), the query video can be an mp4, avi, mov, mkv or other video format file. The system reads the query video frame by frame through the video decoding module and retains the original video frame rate, frame width, frame height and frame sequence information.

[0022] Further, in step 5), the quality score is determined based on the current frame target segmentation mask, the current frame visual prototype, and the initial visual prototype; wherein, the current frame target segmentation mask is used to calculate the segmentation mask area ratio and the overlap with the segmentation masks of adjacent historical frames, and the current frame visual prototype and the initial visual prototype are used to calculate the prototype similarity. The quality score can be determined based on one or more of the segmentation mask area ratio, the overlap, and the prototype similarity.

[0023] Furthermore, the segmentation mask area ratio is used to determine whether the current frame segmentation result has too small a missed detection or too large a false detection; the overlap between the current frame target segmentation mask and the adjacent historical frame segmentation masks is used to determine whether the current frame segmentation result has undergone abnormal jumps; the similarity between the current frame visual prototype and the initial visual prototype is used to determine whether the current frame segmentation target still maintains the same category as the supporting sample target. When the above indicators meet the preset conditions, the current frame visual prototype is written into the prototype memory as a high-confidence frame query visual prototype; when the above indicators do not meet the preset conditions, the update of the prototype memory is skipped.

[0024] Furthermore, the prototype memory is used to store a preset number of historical high-confidence frame query visual prototypes. These historical high-confidence frame query visual prototypes are query visual prototypes written to previous video frames after their quality scores meet preset conditions, and are used to characterize the appearance and structural state of historically confirmed reliable targets in the video stream. The prototype memory can be updated using a first-in, first-out (FIFO) approach, or it can selectively retain historical high-confidence frame query visual prototypes based on quality scores, time intervals, prototype similarity, or target integrity.

[0025] Furthermore, the fused visual cue is obtained by fusing an initial visual prototype, a current frame visual prototype, and a historical high-confidence frame query visual prototype from a prototype memory. The initial visual prototype provides stable category constraints for the target category, the current frame visual prototype provides the real-time appearance and structural state of the target in the current video frame, and the historical high-confidence frame query visual prototype provides reliable historical target representations in the video stream. Preferably, the fused visual cue includes a weighted combination of the means of the initial visual prototype, the current frame visual prototype, and the historical high-confidence frame query visual prototype.

[0026] Furthermore, the method also includes outputting frame-by-frame segmentation masks, frame-by-frame overlay visualization images, segmented mask videos, segmented overlay videos, and prototype memory update logs. The prototype memory update log includes at least one or more of the following: frame number, quality score, whether it is written to the prototype memory, prototype memory size, current frame segmentation mask area ratio, overlap of adjacent frame masks, and similarity between the current frame visual prototype and the initial visual prototype.

[0027] Compared with the prior art, the present invention has at least the following beneficial effects:

[0028] 1) This invention extends static few-sample image segmentation to video stream few-sample segmentation, enabling the system to directly input query videos and output segmented videos, thereby improving the automation processing capability in practical applications.

[0029] 2) This invention represents target categories using a viewpoint-consistent visual prototype, enabling a relatively stable target cues capability even when there are changes in viewpoint, pose, or scale between images and video frames.

[0030] 3) This invention stores historical high-confidence frame visual prototypes in a prototype memory bank, enabling subsequent video frame segmentation to utilize historically stable target representations, thus alleviating the segmentation jumps and target drift problems caused by frame-by-frame independent reasoning.

[0031] 4) This invention uses a quality gating mechanism to write only the prototype video frames that meet the preset quality conditions into the prototype memory, thereby reducing the risk of mis-segmented frames polluting the memory and propagating errors to subsequent frames.

[0032] 5) This invention can output segmented masked video, segmented overlay video, and prototype memory update log, which facilitates subsequent manual inspection, system integration, and engineering deployment. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the overall process of the video stream few-sample segmentation method based on viewpoint consistent prototype memory of the present invention. Detailed Implementation

[0034] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0035] This embodiment provides a few-shot segmentation method for video streams based on viewpoint-consistent prototype memory, applied to video target segmentation scenarios. The method can be deployed on servers, workstations, edge computing devices, industrial camera computing platforms, robot control platforms, or other electronic devices with image processing capabilities. The method enables continuous segmentation of targets of the same category in a query video with only a small number of supporting samples.

[0036] In this embodiment, the input data includes at least:

[0037] 1. Supports images;

[0038] 2. Support mask corresponding to the supporting image;

[0039] 3. Search for videos.

[0040] The support images provide visual, semantic, and structural references for the target category; the support mask identifies the target regions within the support images; and the query video provides the video stream data to be segmented. The query video can be in mp4, avi, mov, mkv, or other video formats, or it can be a sequence of already segmented video frames.

[0041] The main process of the video stream few-shot segmentation method based on viewpoint-consistent prototype memory proposed in this invention is shown in Figure 1, and specifically includes the following steps:

[0042] 1) Obtain supported images, supported masks, and query videos;

[0043] 2) Generate an initial visual prototype based on the supporting images and supporting masks;

[0044] 3) Parse the query video and read the current video frame by frame;

[0045] 4) Generate a visual prototype of the current video frame based on the current video frame;

[0046] 5) Merge the initial visual prototype, the historical high-confidence frame prototype, and the current frame visual prototype to generate a merged visual cue;

[0047] 6) Generate the current frame segmentation mask based on fused visual cues;

[0048] 7) Assess the quality of the visual prototype in the current frame;

[0049] 8) Update or skip the update of the prototype memory based on the quality score results;

[0050] 9) Encode the frame-by-frame segmentation results into a segmented video and output it.

[0051] Specifically, step 1) includes:

[0052] First, obtain the supporting image and the supporting mask. The supporting image can be an RGB image, and the supporting mask can be a binary mask, where the pixel values ​​of the target area are foreground and the pixel values ​​of the non-target area are background. The size of the supporting mask can be the same as the supporting image; when the size of the supporting mask is inconsistent with the supporting image, the supporting mask can be adjusted to the same size as the supporting image using nearest neighbor interpolation to avoid the mask boundary being contaminated by continuous grayscale interpolation.

[0053] Subsequently, the query video is retrieved. The query video contains consecutive video frames that require target segmentation. The system reads the query video through the video input module, obtaining the video frame rate, video width, video height, total number of video frames, and video encoding information, and reads the video frames one by one in chronological order. For video frames that have already been segmented into image sequences, they can also be read sequentially according to the natural order of the frame filenames.

[0054] In this embodiment, the support image, support mask, and query video together constitute the input for the few-shot video segmentation task. Compared with ordinary video segmentation methods, this embodiment does not require manual annotation of each frame in the query video; it only requires the support image and its target mask to segment similar targets in the query video frame by frame.

[0055] Specifically, step 2) includes:

[0056] The system extracts features from the supporting images to obtain semantic and structural features. The semantic features characterize high-level semantic information of the target category, while the structural features characterize the boundaries, shape, and local structural information of the target region. Based on a support mask, the system extracts target region features from the supporting image features and aggregates them to obtain an initial visual prototype.

[0057] In one optional implementation, the system can further perform viewpoint consistency processing on the supporting image or its features to generate a viewpoint-consistent visual prototype. This viewpoint consistency processing may include generating transformed views of the supporting image, aligning target structural features under different views, or enhancing the stability of the target prototype under viewpoint changes through structural feature propagation. In this way, the system can reduce prototype offset issues between the supporting image and the query video frame caused by changes in shooting angle, target pose, or scale.

[0058] In this embodiment, the initial visual prototype is denoted as... The initial visual prototype can be obtained either by directly aggregating the features of the target region in the supporting image, or by enhancing the features of the target region in the supporting image with viewpoint consistency features. This initial visual prototype serves as a stable anchor point prototype in the subsequent video stream segmentation process.

[0059] Specifically, step 3) includes:

[0060] The system reads the current frame of the queried video through the video input module, denoted as... ,in Indicates the current frame number. For each frame... The system retains the original width and height for subsequent restoration of the segmentation mask to the original video size. Then, the system performs image preprocessing on the current frame, which may include scaling, tensor transformation, normalization, or other processing steps adapted to the few-shot segmentation model input.

[0061] By using a frame-by-frame parsing method, the system does not need to pre-split the entire query video into image files; it can directly perform inference processing according to the video stream sequence. This method reduces intermediate file storage overhead and makes the invention more suitable for practical video stream processing scenarios.

[0062] Specifically, step 4) includes:

[0063] For the current video frame The system inputs the supporting image, supporting mask, and the current video frame into the few-shot segmentation model. The few-shot segmentation model may include a feature encoder, a prototype generation module, a visual cue generation module, and a segmentation decoder. The feature encoder extracts image features from the supporting image and the current video frame; the prototype generation module generates a visual prototype of the target category; the visual cue generation module generates cue information for the segmentation decoder based on the visual prototype; and the segmentation decoder outputs the target segmentation mask for the current video frame.

[0064] In one implementation, the system first obtains an initial visual prototype based on supporting images and supporting masks. Then, based on the features of the current video frame, the visual prototype of the current frame is obtained. Subsequently, the system generates segmentation prompts for the current frame based on the initial visual prototype, the current frame visual prototype, and historical prototype memories obtained in subsequent steps. If the current frame is the first frame of the queried video and the prototype memory is empty, the system can output the segmentation result of the first frame based solely on the initial visual prototype or the visual prototype generated from the current frame.

[0065] The current frame segmentation result is denoted as ,in This is a segmentation mask in binary or probabilistic form. If the segmentation model outputs a probability map, it can be converted into a binary mask using a preset threshold. The converted binary mask is used for subsequent quality scoring, prototype memory updates, and video output.

[0066] Specifically, step 5) includes:

[0067] For the current video frame, the system will initialize the visual prototype. Current frame visual prototype and prototype memory bank The historical high-confidence frames are queried and fused with visual prototypes to generate a fused visual cue. The fused visual cue is denoted as... Among them, the initial visual prototype Used to provide stable category constraints for the target category, the current frame visual prototype The historical high-confidence frame query visual prototype is used to characterize the real-time appearance and structural state of targets in the current video frame and to provide a reliable historical target representation in the video stream.

[0068] In a preferred embodiment, the system first calculates the mean of historical high-confidence frame visual prototypes in the prototype memory:

[0069]

[0070] in, This indicates the number of historical high-confidence frame query visual prototypes actually stored in the current prototype memory. Indicates the first A visual prototype for querying historical high-confidence frames. When the prototype memory is empty, the system can proceed based solely on the initial visual prototype. and the current frame visual prototype Generate blended visual cues.

[0071] When the prototype memory is not empty, the system fuses the average values ​​of the initial visual prototype, the current frame visual prototype, and the historical high-confidence frame query visual prototypes according to preset fusion weights:

[0072]

[0073] in, Indicates the integration of visual cues. , , These represent the fusion weights of the initial visual prototype, the current frame visual prototype, and the mean of the historical high-confidence frame query visual prototypes, respectively. Preferably, Larger This can enhance the constraint of supporting samples on the target class, and a larger [size / size] This can enhance the adaptability of the real-time appearance of the current frame to the segmentation results, resulting in a larger... It can enhance the constraint of historical high-confidence frames on video continuity.

[0074] It should be noted that the fusion method is not limited to the weighted summation described above. It can also employ splicing fusion, gated fusion, similarity-weighted fusion, attention fusion, or other methods that combine the initial visual prototype, the current frame visual prototype, and historical high-confidence frames to query the visual prototype. The above embodiments are merely illustrative of a preferred implementation of the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0075] By fusing three types of prototypes, the system can simultaneously utilize the category constraints provided by supporting samples, the real-time target state provided by the current video frame, and the video continuity information provided by historical high-confidence frames, thereby improving the accuracy, continuity, and anti-drift capability of the current frame segmentation. Current Frame Visual Prototype Only after the current frame segmentation result passes the quality score is it written into the prototype memory as a visual prototype for querying historical high-confidence frames of subsequent video frames.

[0076] Specifically, step 6) includes:

[0077] The system will integrate visual cues The input is processed by a segmentation decoder to obtain the target segmentation mask for the current video frame. The segmentation decoder can be a visual cue-based segmentation decoder or any other decoding structure capable of outputting a target mask based on a visual prototype or cue information.

[0078] In one implementation, the segmentation decoder generates a mask prediction map based on fused visual cues and image features of the current video frame. If the mask prediction map contains continuous probability values, the system generates a binary segmentation mask based on a preset threshold. Subsequently, the system restores the segmentation mask from its internal processing size to the original size of the current video frame to ensure that the output mask is spatially aligned with the original video frame.

[0079] When outputting the overlay visualization result, the system restores the segmentation mask to its original size and overlays it with the original video frame to obtain the overlay result of the current frame. The overlay result can be visualized using color overlay, boundary depiction, transparency blending, or other methods.

[0080] Specifically, step 7) includes:

[0081] The system uses the target segmentation mask for the current frame. Current frame visual prototype and the initial visual prototype Perform quality scoring to determine the visual prototype of the current frame. Whether it is suitable to be written into the prototype memory as a historical high-confidence frame query visual prototype. The quality score can be determined by one or more quality metrics.

[0082] In a preferred embodiment, the quality indicators include:

[0083] 1. Mask area ratio 2. Overlap of adjacent frame masks 3. Prototype Similarity .

[0084] Among them, the mask area ratio The proportion of foreground pixels in the target segmentation mask of the current frame to the total number of pixels in the current frame can be represented as:

[0085]

[0086] in, This indicates the number of foreground pixels in the target segmentation mask of the current frame. and These represent the height and width of the current video frame, respectively. If If the area is less than the preset minimum area threshold, it indicates that the current frame may have missed segmentation; if If the area exceeds the preset maximum area threshold, it means that the current frame may have misclassified too much background as the target.

[0087] Adjacent frame mask overlap This is used to measure the regional consistency between the target segmentation mask of the current frame and the target segmentation mask of the previous frame or the previous high-confidence frame. The overlap can be expressed using the intersection-over-union ratio (IoU).

[0088]

[0089] in, This represents the target segmentation mask of the previous frame or the previous high-confidence frame. When When the overlap is below a preset threshold, it indicates that the target segmentation region in the current frame may have an abnormal jump compared to historical frames, and the visual prototype of the current frame... It is not suitable for writing to the prototype memory.

[0090] Prototype similarity Used to measure the visual prototype of the current frame Compared with the initial visual prototype The similarity between categories. This similarity can be represented using cosine similarity:

[0091]

[0092] in, Indicates the visual prototype of the current frame. This represents the initial visual prototype generated from the supporting image and the supporting mask. When If the similarity is below the preset threshold, it means that the query visual prototype extracted in the current frame may have deviated from the target category represented by the supporting sample and is not suitable for writing into the prototype memory.

[0093] Furthermore, the system can obtain a comprehensive quality score based on the above indicators. In one implementation, the overall quality score can be expressed as:

[0094]

[0095] in, , , These represent the area quality, overlap quality, and prototype similarity quality after normalization or thresholding, respectively. , , The weights are preset. It should be noted that the calculation method of the comprehensive quality score is not limited to the above formula, and can also adopt rule judgment, weighted summation, threshold gating or other methods that can determine the reliability of the current frame segmentation.

[0096] When the overall quality score When preset conditions are met, the system will display the visual prototype of the current frame. Write it into the prototype memory as a visual prototype for querying historical high-confidence frames of subsequent video frames; when the overall quality score is... If the preset conditions are not met, the system skips updating the prototype memory. Through the above quality scoring mechanism, the system can avoid writing obviously erroneous visual prototypes of the current frame into the prototype memory, thereby reducing the risk of error accumulation and target drift.

[0097] Specifically, step 8) includes:

[0098] The system scores based on quality. The prototype memory is updated based on the results of judgments from one or more quality indicators. The prototype memory is denoted as... This is used to store historical high-confidence frame visual prototypes. The prototype memory can store one or more historical frame visual prototypes:

[0099]

[0100] in, This indicates the maximum number of prototypes that can be stored in the prototype memory. Indicates the first A historical high-confidence frame visual prototype.

[0101] When the quality score of the current frame meets the preset conditions, the system will display the visual prototype of the current frame. Write to the prototype memory. If the prototype memory is not full, write directly; if the prototype memory is full, delete the oldest written historical frame prototype and then write the current frame visual prototype. This method constitutes a first-in, first-out prototype update method.

[0102] When the quality score of the current frame does not meet the preset conditions, the system prohibits writing the visual prototype of the current frame into the prototype memory and continues to use the existing prototype memory for subsequent frame segmentation. In this way, even if the current frame has segmentation abnormalities due to occlusion, blurring, or background interference, the erroneous prototype will not be propagated to subsequent video frames.

[0103] In one alternative implementation, the prototype memory can also record the frame number, quality score, mask area ratio, prototype similarity, and update time corresponding to each historical prototype for subsequent log output, visualization analysis, or memory maintenance.

[0104] Specifically, step 9) includes:

[0105] The system saves frame-by-frame segmentation masks and frame-by-frame overlay visualization images in the order of video frames. Furthermore, the system can encode the frame-by-frame segmentation mask into a segmentation mask video and the frame-by-frame overlay visualization image into a segmentation overlay video based on the original frame rate, original width, and original height of the query video.

[0106] In one implementation, the system output includes at least:

[0107] 1. Segment the mask image frame by frame;

[0108] 2. Segment and overlay images frame by frame;

[0109] 3. Segment and mask the video;

[0110] 4. Segment and overlay video;

[0111] 5. Prototype memory update log.

[0112] The prototype memory update log can include information such as frame number, current frame mask area ratio, adjacent frame mask overlap, current frame prototype similarity, overall quality score, whether it has been written to the prototype memory, and the current size of the prototype memory. By outputting this log, users can check the prototype memory update process and determine whether the system stops updating at low-quality frames, thereby improving the interpretability and debuggability of the method.

[0113] The invention will now be described in conjunction with a specific application scenario.

[0114] In industrial quality inspection video scenarios, users first provide a supporting image containing the target part to be inspected, along with a binary support mask corresponding to the part's region. Then, the user inputs a video containing similar parts. The system generates an initial viewpoint-consistent visual prototype based on the supporting image and support mask, and processes the query video frame by frame. When the segmentation results of the first few frames of the video are relatively stable, the system writes the visual prototypes of these high-confidence frames into the prototype memory. If a frame's segmented region becomes abnormally small due to motion blur or occlusion, has too low an overlap with the previous frame, or its prototype similarity decreases, the system does not write the prototype of that frame into the memory. Subsequent frames continue to be segmented by fusing the initial visual prototype and historical high-confidence frame prototypes. Finally, the system outputs a segmented mask video of the target part and an overlaid visualization video.

[0115] Through this implementation, the present invention enables continuous segmentation of targets in videos with only a small number of supporting samples. Compared to frame-by-frame independent segmentation, the present invention enhances the target representation in subsequent frames by using historical high-confidence frame prototype memory; compared to directly using all historical frames to update the memory bank, the present invention reduces the risk of erroneous segmentation results contaminating the memory bank through a quality gating mechanism.

[0116] Furthermore, this invention can also be applied to robot grasping scenarios. In this scenario, the robot only needs to obtain a supporting image of the target object and its mask to segment similar target objects in real-time camera video. Since the target will undergo changes in viewpoint and scale during robot movement, the initial viewpoint-consistent visual prototype can provide stable constraints for the target category, while the historical high-confidence frame prototype memory can supplement the target appearance information according to the current video environment, thereby improving the stability of continuous frame segmentation.

[0117] Furthermore, this invention can also be applied to drone inspection scenarios. During drone flight, the shooting angle, target scale, and background texture continuously change. Using the method of this invention, the system can specify the target type using support samples and output segmented video of the target area in the inspection video, providing pixel-level results for subsequent target localization, area measurement, defect detection, or behavior analysis.

[0118] This invention does not limit the specific network structure of the few-shot segmentation model. The few-shot segmentation model can employ convolutional neural networks, visual Transformers, graph neural networks, cue-based segmentation models, or combinations of the above. Any model capable of generating a visual prototype based on supporting images, masks, and query video frames, and outputting target segmentation results based on the visual prototype, can be applied to this invention.

[0119] This invention does not limit the specific storage quantity and update rules of the prototype memory. The size of the prototype memory can be set according to the video length, device video memory, target motion speed, and application scenario. For short videos, a smaller memory capacity can be set; for long videos, a larger memory capacity can be set, or the memory can be maintained using methods such as time decay, quality priority, and similarity redundancy removal.

[0120] This invention does not limit the specific thresholds for each quality indicator in the quality scoring. For scenes with relatively stable target area, the weight of the area ratio indicator can be increased; for scenes with strong target motion continuity, the weight of the adjacent frame mask overlap indicator can be increased; for scenes with strong background interference, the weight of the prototype similarity indicator can be increased. Each threshold can be set empirically or adaptively adjusted based on actual video data.

[0121] This invention does not limit the specific encoding format of the output video. The system can output mp4, avi, mov, or other video formats according to the actual deployment environment. The output segmented video can be a black and white masked video or a color category masked video; the output overlay video can have a semi-transparent overlay effect, a boundary contour effect, or other visualization forms that facilitate the observation of the target segmentation results.

[0122] The method described in this embodiment is applicable to video stream target segmentation tasks with few samples. By supporting image and mask generation to create visual prototypes with consistent initial viewpoints, and maintaining a historical high-confidence frame prototype memory bank during video inference, this invention can improve the stability of continuous frame segmentation while reducing the cost of manual video annotation. Through a quality-gated update mechanism, this invention can reduce the risk of mis-segmented frames polluting the memory bank and causing subsequent frame drift. Through video input / output adaptation, this invention can directly receive query videos and output segmentation result videos, making it more suitable for practical engineering deployments.

[0123] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A few-shot segmentation method for video streams based on viewpoint-consistent prototype memory, characterized in that, Includes the following steps: 1) Obtain supporting images, supporting masks, and query videos, wherein the supporting mask is used to identify target regions in the supporting images, and the query videos include consecutive video frames to be segmented; 2) Preprocess the supporting image and the supporting mask, and generate an initial visual prototype based on the target region in the supporting image. The initial visual prototype is a viewpoint-consistent visual prototype, which is used to represent the semantic and structural features of the target category that remain consistent under different viewpoints, poses or scale changes. 3) Perform video stream parsing on the query video, obtain the current video frame in the order of video frames, and perform image preprocessing on the current video frame to generate the visual prototype of the current frame; 4) The initial visual prototype, the current frame visual prototype, and the historical high-confidence frame query visual prototype in the prototype memory are fused to generate a fused visual cue. 5) Input the fused visual cues into the segmentation decoder to obtain the target segmentation mask of the current video frame; 6) Calculate a quality score based on the current frame visual prototype, the initial visual prototype, and the target segmentation mask of the current video frame. The quality score is used to determine whether the current frame visual prototype meets the conditions for being written into the prototype memory. 7) When the quality score meets the preset conditions, the visual prototype of the current frame is written into the prototype memory; when the quality score does not meet the preset conditions, writing the visual prototype of the current frame into the prototype memory is prohibited. 8) Generate a segmentation mask video based on the target segmentation mask obtained frame by frame, and overlay the target segmentation mask with the corresponding original video frame to generate a segmentation visualization video.

2. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 1, characterized in that, In step 2), the initial visual prototype is obtained by extracting features from the supporting image, extracting target region features based on the supporting mask, and aggregating the target region features; and the initial visual prototype is enhanced with viewpoint consistency feature processing to improve its stability under different shooting angles, target poses, scale changes, or local deformation conditions.

3. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 2, characterized in that, The viewpoint consistency feature processing includes structural feature modeling and feature alignment of the supporting image, the transformed view of the supporting image, or the feature representation corresponding to the supporting image, in order to reduce the prototype offset between the supporting image and the query video frame caused by viewpoint differences.

4. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 1, characterized in that, The query video is a video file or a sequence of video frames. When the query video is a video file, the query video is read frame by frame by the video decoding module, and the frame rate, frame width, frame height and frame sequence number information of the query video are retained. When the query video is a sequence of video frames, the video frames are read sequentially according to the order of the frame file names or the time order.

5. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 1, characterized in that, The fused visual cues are obtained by fusing the initial visual prototype, the current frame visual prototype, and the historical high-confidence frame query visual prototype in the prototype memory; wherein, the initial visual prototype is used to provide target category constraints, the current frame visual prototype is used to provide the target appearance and structural state in the current video frame, and the historical high-confidence frame query visual prototype is used to provide a reliable historical target representation in the video stream.

6. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 5, characterized in that, When the prototype memory is empty, the fused visual cue is generated based on the initial visual prototype and the current frame visual prototype. When the prototype memory is not empty, the fused visual cue is generated based on the average of the initial visual prototype, the current frame visual prototype, and the visual prototypes queried from historical high-confidence frames in the prototype memory. The fusion method includes one or more of weighted fusion, splicing fusion, gated fusion, similarity-weighted fusion, or attention fusion.

7. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 1, characterized in that, The quality score is determined based on one or more of the following: mask area ratio, adjacent frame mask overlap, and prototype similarity. Specifically, the mask area ratio is used to determine whether there are any missed or incorrect segments in the current frame segmentation result; the adjacent frame mask overlap is used to determine whether the current frame segmentation result has an abnormal jump relative to historical frames; and the prototype similarity is used to determine whether the current frame visual prototype maintains category consistency with the initial visual prototype.

8. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 7, characterized in that, When the mask area ratio is within a preset area range, the overlap of the adjacent frame masks is greater than a preset overlap threshold, and the prototype similarity is greater than a preset similarity threshold, the quality score is determined to meet the preset conditions, and the current frame visual prototype is written into the prototype memory as a historical high-confidence frame query visual prototype; otherwise, the quality score is determined not to meet the preset conditions, and the update of the prototype memory is skipped.

9. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 1, characterized in that, The prototype memory is used to store a preset number of historical high-confidence frame query visual prototypes; When the prototype memory library has not reached the preset capacity, the current frame visual prototype that meets the preset conditions is directly written. When the prototype memory reaches a preset capacity, the visual prototypes in the historical high-confidence frames of the prototype memory are updated or selectively retained based on one or more of the following: first-in-first-out rule, quality score, time interval, prototype similarity, or target integrity.

10. The video stream few-shot segmentation method based on viewpoint-consistent prototype memory according to claim 1, characterized in that, The method further includes outputting frame-by-frame segmentation masks, frame-by-frame overlay visualization images, segmented mask videos, segmented overlay videos, and prototype memory update logs; the prototype memory update logs include at least one or more of the following: frame number, quality score, whether to write to the prototype memory bank, prototype memory bank size, current frame mask area ratio, overlap of adjacent frame masks, and similarity between the current frame visual prototype and the initial visual prototype.

Images