Training-free subjective and objective video object correspondence method driven by spatial feature alignment

Abstract

This invention discloses a training-free subjective-objective video object mapping method driven by spatial feature alignment. First, videos with target object annotations in both subjective and objective perspectives are designated as source perspective videos, while the other perspective video is designated as the target perspective video. A pre-trained dense feature matching model is used to process the source and target perspective videos, obtaining a bidirectional feature point mapping relationship and a matching confidence map. Based on the target object mask of the source perspective video, an effective feature point set is obtained. Based on this effective feature point set, keyframes and the target object masks of the corresponding frames in the target perspective video are selected and used as spatiotemporal cues. A pre-set cue-based video object segmentation model is then used for target segmentation, yielding the target object mask of the target perspective video frame, thus obtaining the subjective-objective video object mapping relationship. This invention requires no training and significantly improves the accuracy of subjective-objective video object mapping tasks on large-scale datasets by utilizing feature matching and cue-based video object segmentation.

Description

Technical Field

[0001] This invention belongs to the field of video processing technology, and more specifically, relates to a training-free subjective-object mapping method driven by spatial feature alignment. Background Technology

[0002] The subjective-object mapping task aims to establish the correspondence between objects in different video perspectives (first-person perspective "ego" and third-person perspective "exo") within the same scene. Existing methods for subjective-object mapping mainly achieve this through the following approaches: The first approach is an extension of object segmentation methods. These methods, building upon models originally designed for monocular vision, introduce text modality assistance through multimodal conditional fusion and cross-viewpoint feature alignment to align visual features in the two perspectives, thus achieving object mapping between subjective and objective perspectives. Another approach, based on large-scale subjective-object viewpoint video pairs, uses self-supervised learning. It learns viewpoint consistency through viewpoint-conditional grayscale coloring and trains the model using pseudo-labels generated by a self-supervised model to establish the subjective-object mapping.

[0003] Existing technologies have several significant limitations when processing tasks involving both subjective and objective video objects.

[0004] First, existing supervised learning methods often require massive amounts of labeled data to train model parameters in order to achieve accurate matching of objects across different perspectives. The collection and labeling of such data not only consumes a lot of human and material resources, but also needs to deal with the labeling complexity brought about by the diversity of objects in different scenarios. At the same time, the model training process relies on high-performance computing equipment, and long-term iterative optimization further increases the computing cost.

[0005] Secondly, subjective and objective perspectives, as different shooting angles of the same scene, have some spatial overlap. These areas contain rich pixel-level correspondences, which are important clues for establishing cross-perspective object associations. However, most existing methods ignore this key information and fail to effectively mine and utilize the pixel-level features of the overlapping areas, reducing the utilization rate of spatial information and thus affecting the accuracy of object correspondence results.

[0006] Furthermore, one of the core characteristics of video data is its temporal dynamism; objects in a video exhibit continuous motion and shape changes. However, existing methods often process videos as discrete frame pairs, failing to consider the temporal correlation between frames and thus failing to capture the consistent characteristics of objects over time. This makes it difficult for models to maintain the temporal coherence of segmentation results in dynamic scenes when objects move, occlude, or change shape, easily leading to jumps in results between frames. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a training-free subjective-object mapping method driven by spatial feature alignment. This method does not require a training framework and utilizes feature matching and cue-based video target segmentation to fully leverage the spatial information of overlapping regions and the temporal information in the video frame stream, thereby significantly improving the accuracy of subjective-object mapping tasks on large-scale datasets.

[0008] To achieve the above-mentioned objective, the training-free subjective-object video object mapping method driven by spatial feature alignment of the present invention includes the following steps:

[0009] S1: For the acquired first-person subjective perspective video and third-person objective perspective video ,in, , These represent video frames from the subjective perspective and video frames from the objective perspective, respectively. , This indicates the video frame number. Based on actual needs, target objects are labeled for one of the video perspectives, and the video perspective with labeled target objects is used as the source video perspective. , This represents the source view video frame, and each source view video frame is generated based on the target object annotation. Corresponding target object mask Then, the video from another perspective is used as the target perspective video. , Indicates a video frame from the target's perspective;

[0010] S2: Use a pre-trained dense feature matching model to analyze the source view video. and target perspective video Corresponding time in the middle video frames and The process is performed to generate a bidirectional dense correspondence field, resulting in a mapping that transforms the source viewpoint pixel coordinates to the target viewpoint. And mapping the target view pixel coordinates back to the source view. Output two matching confidence graphs simultaneously. and Then, source view video frames are generated based on the matching confidence map. feature point set According to the target object mask From feature point set The source view effective feature point set is constructed by selecting feature points belonging to the target object. ;

[0011] S3: Generate spatiotemporal cues for guiding segmentation, including source view keyframes and target object masks for corresponding frames in the target view. The specific method is as follows:

[0012] S3.1: Video from the source perspective Selected from Keyframes , Indicates the original frame number of the keyframe. , The value should be set according to actual needs;

[0013] S3.2: Set each keyframe Effective feature point set Using valid feature points as conditions, a cue-based mask generation model is employed to generate video frames from the target viewpoint. Generate a set of class-independent mask proposals ,in , Indicates the number of mask proposals;

[0014] S3.3: Based on the pre-set mask proposal evaluation score calculation method, for each target viewpoint video frame... Each mask proposal in Calculate its evaluation score ;

[0015] S3.4: For each target viewpoint video frame From its A mask proposal Select evaluation score The optimal mask proposal is used as the target object mask. ;

[0016] S4: Obtain the result from step S3 Individual source perspective keyframes and the target object mask from the target's perspective As a spatiotemporal cue, a pre-defined cue-based video target segmentation model is input to segment the target, thereby obtaining video frames from each target's perspective. Target object mask Obtain the mask pair of the target object in the subjective and objective video. This yields the correspondence between subjective and objective video objects.

[0017] This invention discloses a training-free subjective-objective video object correspondence method driven by spatial feature alignment. First, videos with target object annotations in both subjective and objective perspectives are designated as source perspective videos, while another perspective video is designated as the target perspective video. A pre-trained dense feature matching model is used to process the source and target perspective videos, obtaining a bidirectional feature point mapping relationship and a matching confidence map. Based on the target object mask of the source perspective video, an effective feature point set is obtained. Based on this effective feature point set, keyframes and the target object masks of corresponding frames in the target perspective video are selected and used as spatiotemporal cues. A pre-set cue-based video object segmentation model is then used for target segmentation, resulting in the target object mask of the target perspective video frame, thus obtaining the subjective-objective video object correspondence relationship.

[0018] The present invention has the following beneficial effects:

[0019] 1) This invention can achieve the correspondence between the same target in first-person (subjective view) and third-person (objective view) videos in the same scene without the need for annotation data and model training of the correspondence between subjective and objective video objects. This solves the problem of target correspondence caused by the significant differences between subjective and objective viewpoints in imaging geometry, field of view and motion mode.

[0020] 2) Experiments have shown that the performance of this invention is close to that of existing supervised methods and significantly better than zero-shot methods;

[0021] 3) This invention can serve as the core basic visual task module for embodied intelligent subjects (such as autonomous robots), supporting their first-person interaction and environmental perception. It also provides efficient training-free technical support for spatial relationship cognition and modeling in cross-perspective scenes in spatial intelligence. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a specific implementation of the spatial feature alignment-driven training-free subjective-object video object mapping method of the present invention.

[0023] Figure 2 This is a schematic diagram of the cycle consistency filtering in this embodiment;

[0024] Figure 3 This is a flowchart of the process for generating spatiotemporal prompts in this invention;

[0025] Figure 4 This is a flowchart of the heuristic keyframe selection method in this embodiment;

[0026] Figure 5 This is a schematic diagram illustrating the calculation of semantic similarity score in this embodiment. Detailed Implementation

[0027] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0028] Example

[0029] Figure 1 This is a flowchart illustrating a specific implementation of the training-free subjective-object video object mapping method driven by spatial feature alignment of the present invention. Figure 1 As shown, the training-free subjective-object video object mapping method driven by spatial feature alignment of the present invention includes the following steps:

[0030] S101: Determine the source viewpoint and target viewpoint:

[0031] For the first-person subjective perspective video obtained and third-person objective perspective video ,in, , These represent video frames from the subjective perspective and video frames from the objective perspective, respectively. , This indicates the video frame number. Based on actual needs, target objects are labeled for one of the video perspectives, and the video perspective with labeled target objects is used as the source video perspective. , This represents the source view video frame, and each source view video frame is generated based on the target object annotation. Corresponding target object mask Then, the video from another perspective is used as the target perspective video. , This represents a video frame from the target's perspective.

[0032] S102: Cross-view feature matching:

[0033] The key to achieving the target-objective video correspondence task lies in establishing the correlation features between different viewpoints. Feature matching methods can achieve cross-view feature fusion and focus on overlapping areas within the viewpoints, thereby fully utilizing spatial information. Therefore, cross-view feature matching is first performed on the source and target viewpoint videos to establish pixel-level correlations between the subjective and objective viewpoints, and reliable feature points related to the target are selected. The specific method of cross-view feature matching in this invention is as follows:

[0034] A pre-trained dense feature matching model (RoMa (Robust Dense Feature Matching) is used in this embodiment to match the source viewpoint video. and target perspective video Corresponding time in the middle frames and The process is performed to generate a bidirectional dense correspondence field, resulting in a mapping that transforms the source viewpoint pixel coordinates to the target viewpoint. And mapping the target view pixel coordinates back to the source view. Output two matching confidence graphs simultaneously. and This is used to characterize the reliability of each mapping relationship. Then, source-view video frames are generated based on the matching confidence map. feature point set According to the target object mask From feature point set The source view effective feature point set is constructed by selecting feature points belonging to the target object. .

[0035] To ensure the correlation between feature points and the target, this embodiment employs a target feature mask. Cyclic consistency verification is performed on the pixels to verify the effective feature point set. The optimization process is as follows:

[0036] For the effective feature point set Each pixel ,pass Pixels are obtained by mapping to the target viewpoint Then through Pixels Map back to the source viewpoint to obtain pixels If pixels Target mask located in the source view within, that is Then retain pixels Pixels are considered valid feature points; otherwise, they are discarded. .

[0037] Figure 2 This is a schematic diagram of the cycle consistency filtering in this embodiment. As can be seen, cycle consistency filtering can effectively filter out noise points that are irrelevant to the target object, retaining only the target object feature points that are consistent between the subjective and objective perspectives.

[0038] S103: Generate a spacetime prompt:

[0039] Although the feature points of the target object have been obtained in step S102, two challenges remain: bridging the gap between pixels and objects, and modeling the spatiotemporal dynamics beyond a single frame. To address the first challenge, class-agnostic object segmentation can transform the representation from the pixel level to the object level and provide interpretable masks. However, object proposals generated in a single frame cannot capture the temporal information in the video. To enhance temporal modeling capabilities, previous video object segmentation methods have introduced memory modules to propagate features across frames. Nevertheless, cue-based VOS (Video Object Segmentation) methods heavily rely on cue quality; therefore, robust keyframe selection and mask generation are crucial. This invention transforms the effective feature points of the target object obtained in step S102 into spatiotemporal cues that can be used to guide segmentation, including source view keyframes and target object masks for the corresponding frames of the target view. Figure 3 This is a flowchart illustrating the generation of spatiotemporal prompts in this invention. For example... Figure 3 As shown, the specific steps for generating spatiotemporal prompts in this invention include:

[0040] S301: Select keyframe:

[0041] From the source perspective video Selected from Keyframes , Indicates the original frame number of the keyframe. , The value should be set according to actual needs.

[0042] To balance the richness of spatial information and the uniformity of temporal distribution, and to select keyframes that can represent the spatiotemporal characteristics of the video, this embodiment proposes a heuristic keyframe selection method. Figure 4 This is a flowchart of the heuristic keyframe selection method in this embodiment. For example... Figure 4 As shown, the heuristic keyframe selection method in this embodiment includes the following steps:

[0043] S401: Calculate frame weights:

[0044] effective feature point set The size of the source view video frame weight A higher weight indicates that the frame contains richer target spatial information. Then, the weights are assigned to all source viewpoint video frames. Normalization is performed to obtain the normalized weights for each frame. To avoid the influence of absolute numerical differences.

[0045] S402: Select the first keyframe:

[0046] Select normalized weights from the source video. The largest video frame is used as the first keyframe. Make the keyframe set ,Right now .

[0047] S403: Calculate candidate frame scores:

[0048] For each candidate frame in the source view video The following formula is used to calculate and add it to the keyframe set. The score after :

[0049] ,

[0050] in, Indicates the preset weight. Indicates candidate frames Add to keyframe set The resulting set of keyframes The time uniformity score. Normalized weights. The spatial importance of candidate frames is represented by the temporal uniformity score. This represents the temporal importance of candidate frames, and keyframes selected in this way have stronger spatiotemporal representativeness.

[0051] In this embodiment, the time uniformity score It is calculated using the relative standard deviation (RSD). The specific calculation method is as follows: Let the ideal time interval be... Calculate the set of keyframes The time interval between two adjacent video frames , , Represents the set of keyframes The number of frames in the data, and then the logarithm of the time interval is calculated. ,calculate The relative standard deviation of each log value Finally, the time uniformity score is calculated. ,in Represents the natural constant. This represents the attenuation coefficient. It can be seen that the smaller the relative standard deviation (RSD) (the more uniform the distribution), the higher the time uniformity.

[0052] S404: Select new keyframes:

[0053] Score is selected from all candidate frames. The highest one is added to the keyframe set as a new keyframe. .

[0054] S405: Determine the current keyframe set Has the number of keyframes reached the required level? If yes, the keyframe selection ends; otherwise, return to step S403.

[0055] S302: Proposal for Generating Keyframe Target View Masks:

[0056] Each keyframe Effective feature point set Using valid feature points as conditions, a prompt-based mask generation model (using the SAM2 mask generator in this embodiment) is employed to generate a mask from the target viewpoint video frame. Generate a set of class-independent mask proposals ,in , Indicates the number of mask proposals.

[0057] To improve spatial coverage and reduce redundancy, the effective feature point set can be optimized before generating the mask proposal. Perform density-based sampling, retain the valid feature points obtained from the sampling, and remove other valid feature points.

[0058] S303: Calculate the mask proposal evaluation score:

[0059] Based on the pre-set mask proposal evaluation score calculation method, for each target viewpoint video frame Each mask proposal in Calculate its evaluation score In this embodiment, the mask proposal evaluation score is... The calculation method is as follows:

[0060] ,

[0061] in, The weight represents the balance between geometric alignment and semantic similarity. , These represent mask proposals. The geometric projection score and semantic similarity score are calculated using the following methods:

[0062] For the effective feature point set of the source view Each pixel According to the matching confidence graph and Obtain its two-way confidence level , Calculate the sum of their two-way confidence scores. Then, the mask proposal is calculated using the following formula. Geometric projection fraction :

[0063] ,

[0064] in, Indicates the indicator function, when pixel Mapped to target viewpoint video frames pixels It belongs to the mask proposal When the target object is, ,otherwise .

[0065] Semantic similarity scores are used to evaluate mask proposals. The semantic similarity between candidate objects and target objects. Figure 5 This is a schematic diagram illustrating the calculation of semantic similarity scores in this embodiment. For example... Figure 5 As shown, the preset feature extraction function The specific feature extraction method can be set according to actual needs. This embodiment uses a pre-trained MobileNetV3 model. A feature extraction function is used. From the target video mask proposal and source view mask Extracting feature embeddings of the target object region , Then, the similarity between feature embeddings is calculated as a semantic similarity score. In this embodiment, the feature embedding similarity uses cosine similarity, therefore:

[0066] .

[0067] S304: Determine the target view mask:

[0068] For each target viewpoint video frame From its A mask proposal Select evaluation score The optimal mask proposal is used as the target object mask. In this embodiment, the mask proposal with the higher the evaluation score is considered better.

[0069] S104: Complete the mapping between subjective and objective video objects:

[0070] The result obtained in step S103 Individual source perspective keyframes and the target object mask from the target's perspective As a spatiotemporal cue, a pre-defined cue-based video target segmentation model is input to segment the target, resulting in video frames for each target's viewpoint. Target object mask This allows us to obtain the mask pair of the target object in the subjective and objective video. This yields the correspondence between subjective and objective video objects.

[0071] In this embodiment, the SAM2 model is used for the prompt-based video target segmentation model. As can be seen, the target view mask in the keyframes provides spatial appearance information of the target, and its timestamp serves as a time anchor. The SAM2 model utilizes a memory module to propagate the mask information of the keyframes throughout the entire video sequence, completing the segmentation of the target in all frames from the target viewpoint while maintaining temporal consistency.

[0072] To better illustrate the technical solution of this invention, specific embodiments are used to experimentally verify the invention. In this embodiment, testing is conducted on the Ego-Exo4D dataset, which contains 295 scenes, including 139,134 subjective-to-objective viewpoint video frame pairs and 233,500 subjective-to-objective viewpoint video frame pairs. This embodiment uses two existing methods as comparison methods (denoted as Matching2Seg) for comparative verification with this invention, including methods that do not require training and methods that require training, respectively including the following methods:

[0073] Methods that do not require training:

[0074] XsegTX: See the document "Grauman, K., et al. 2024. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition."

[0075] XMem: See the document "Grauman, K., et al. 2024. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InProceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition."

[0076] PSALM: See the document "Z. Zhang, Y. Ma, E. Zhang, and X. Bai, "PSALM: Pixelwise segmentAtion with large multi-modal model," in Proceedings of the European Conference on Computer Vision (ECCV), 2024, pp. 74–91."

[0077] DINOv2+SAM2: See the literature "Espinosa, M.; Yang, C.; Ericsson, L.; McDonagh,

[0078] S.; and Crowley, EJ 2025. No time to train! Training-FreeReference-Based Instance Segmentation. arXiv:2507.02798.”

[0079] RoMa+SAM2 (point): The baseline model that uses the point with the highest confidence among the feature points obtained by the dense feature matching model RoMa as the SAM2 cue to perform separate segmentation for each frame.

[0080] RoMa+SAM2(KMG): Uses all feature points obtained by the dense feature matching model RoMa as SAM2 cue, and calculates the mask proposal score on the obtained mask to obtain the baseline model of the segmentation result for each frame.

[0081] Training methods required:

[0082] XsegTX,

[0083] ObjectRelator: See the document "Fu, Y., et al. 2024. ObjectRelator: EnablingCross-View Object Relation Understanding in Ego-Centric and Exo-CentricVideos. arXiv:2411.19083."

[0084] PCC: See the document "Baade, A., and Chen, C. 2025. Self-Supervised Cross-ViewCorrespondence with Predictive Cycle Consistency. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition."

[0085] In this embodiment, the following indicators are used to compare the present invention and the comparative method:

[0086] Balancing Accuracy (Balanced Accuracy) is used to evaluate the accuracy of the model in determining the existence of a target object; a higher value is better.

[0087] IoU (Intersection over Union) measures the degree of overlap between the predicted object mask and the real mask; a higher value indicates a better segmentation effect.

[0088] Loc. Score (Location Score) quantifies the normalized deviation between the predicted object center and the actual object center. The lower the value, the more accurate the positioning.

[0089] CA (Contour Accuracy): Calculated by the harmonic mean of contour precision and recall, it is used to evaluate the shape consistency of the predicted mask. The higher the value, the closer the contour is to the real situation.

[0090] Table 1 is a comparison table of the performance indicators of the present invention and the comparative method in this embodiment.

[0091]

[0092] Table 1

[0093] In Table 1, ZSL is marked with √ indicating no training is required and × indicating training is required. As shown in Table 1, the present invention outperforms previous methods in most metrics among training-free methods. Compared with training-based methods, the present invention achieves superior performance on most metrics under both subjective and objective settings, and achieves a comparable level under objective and subjective conditions.

[0094] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A training-free subjective-object video object mapping method driven by spatial feature alignment, characterized in that, Includes the following steps: S1: For the acquired first-person subjective perspective video and third-person objective perspective video ,in, , These represent video frames from the subjective perspective and video frames from the objective perspective, respectively. , This indicates the video frame number. Based on actual needs, target objects are labeled for one of the video perspectives, and the video perspective with labeled target objects is used as the source video perspective. , This represents the source view video frame, and each source view video frame is generated based on the target object annotation. Corresponding target object mask Then, the video from another perspective is used as the target perspective video. , Indicates a video frame from the target's perspective; S2: Use a pre-trained dense feature matching model to analyze the source view video. and target perspective video Corresponding time in the middle video frames and The process is performed to generate a bidirectional dense correspondence field, resulting in a mapping that transforms the source viewpoint pixel coordinates to the target viewpoint. And mapping the target view pixel coordinates back to the source view. Output two matching confidence graphs simultaneously. and Then, source view video frames are generated based on the matching confidence map. feature point set According to the target object mask From feature point set The source view effective feature point set is constructed by selecting feature points belonging to the target object. ; S3: Generate spatiotemporal cues for guiding segmentation, including source view keyframes and target object masks for corresponding frames in the target view. The specific method is as follows: S3.1: Video from the source perspective Selected from Keyframes , Indicates the original frame number of the keyframe. , The value should be set according to actual needs; S3.2: Set each keyframe Effective feature point set Using valid feature points as conditions, a cue-based mask generation model is employed to generate video frames from the target viewpoint. Generate a set of class-independent mask proposals ,in , Indicates the number of mask proposals; S3.3: Based on the pre-set mask proposal evaluation score calculation method, for each target viewpoint video frame... Each mask proposal in Calculate its evaluation score ; S3.4: For each target viewpoint video frame From its A mask proposal Select evaluation score The optimal mask proposal is used as the target object mask. ; S4: Obtain the result from step S3 Individual source perspective keyframes and the target object mask from the target's perspective As a spatiotemporal cue, a pre-defined cue-based video target segmentation model is input to segment the target, thereby obtaining video frames from each target's perspective. Target object mask Obtain the mask pair of the target object in the subjective and objective video. This yields the correspondence between subjective and objective video objects.

2. The training-free subjective-object video object mapping method according to claim 1, characterized in that, The dense feature matching model adopts the RoMa model.

3. The training-free subjective-object video object mapping method according to claim 1, characterized in that, For the effective feature point set Optimization is achieved using circular consistency verification, as detailed below: For the effective feature point set Each pixel ,pass Pixels are obtained by mapping to the target viewpoint Then through Pixels Map back to the source viewpoint to obtain pixels If pixels Target mask located in the source view within, that is Then retain pixels Pixels are considered valid feature points; otherwise, they are discarded. .

4. The training-free subjective-object video object mapping method according to claim 1, characterized in that, The specific method for keyframe selection is as follows: S3.1.1: Set the effective feature points The size of the source view video frame weight Then, the weights of all source view video frames are calculated. Normalization is performed to obtain the normalized weights for each frame. ; S3.1.2: Selecting Normalized Weights from the Source View Video The largest video frame is used as the first keyframe. Make the keyframe set ; S3.1.3: For each candidate frame in the source view video The following formula is used to calculate and add it to the keyframe set. The score after : ， in, Indicates the preset weight. Indicates candidate frames Add to keyframe set The resulting set of keyframes The time uniformity score; S3.1.4: Filter scores from all candidate frames The highest one is added to the keyframe set as a new keyframe. ; S3.1.5: Determine the current set of keyframes Has the number of keyframes reached the required level? If yes, the keyframe selection ends; otherwise, return to step 3.1.

3.

5. The training-free subjective-object video object mapping method according to claim 4, characterized in that, The time uniformity score The calculation method is as follows: Let the ideal time interval Calculate the set of keyframes The time interval between two adjacent video frames , , Represents the set of keyframes The number of frames in the data, and then the logarithm of the time interval is calculated. ,calculate The relative standard deviation of each log value Finally, the time uniformity score is calculated. ,in Represents the natural constant. This represents the attenuation coefficient.

6. The training-free subjective-object video object mapping method according to claim 1, characterized in that, The prompt-based mask generation model uses the SAM2 mask generator.

7. The training-free subjective-object video object mapping method according to claim 1, characterized in that, In step S3.2, the effective feature point set is processed before generating the mask proposal. Perform density-based sampling, retain the valid feature points obtained from the sampling, and remove other valid feature points.

8. The training-free subjective-object video object mapping method according to claim 1, characterized in that, The mask proposal evaluation score The calculation method is as follows: ， in, Indicates weight, , These represent mask proposals. The geometric projection score and semantic similarity score are calculated using the following methods: For the effective feature point set of the source view Each pixel According to the matching confidence graph and Obtain its two-way confidence level , Calculate the sum of their two-way confidence scores. Then, the mask proposal is calculated using the following formula. Geometric projection fraction : ， in, Indicates the indicator function, when pixel Mapped to target viewpoint video frames pixels It belongs to the mask proposal When the target object is, ,otherwise ; Preset feature extraction function Feature extraction function From the target video mask proposal and source view mask Extracting feature embeddings of the target object region , Then calculate the feature embedding. , The similarity between them is used as a semantic similarity score. .

9. The training-free subjective-object video object mapping method according to claim 8, characterized in that, The feature extraction function A pre-trained MobileNetV3 model was used.

10. The training-free subjective-object video object mapping method according to claim 1, characterized in that, The prompt-based video target segmentation model uses the SAM2 model.

Images