Interactive click-based video object segmentation method

Abstract

The application discloses a video target segmentation method based on interactive clicking, comprising the following steps: receiving a click point of a user on a video frame as an initial target label; extracting local features around the click point through a deep learning model to construct a preliminary representation of a target region; further extracting target features of the target region and propagating the features in two dimensions of space and time; performing long-time sequence target mapping and tracking on the target region; performing cross-frame target optimization on the target region to realize dynamic adjustment and optimization of a final target region; and adjusting a target segmentation boundary and a target position in each frame by combining a PID control algorithm and an LQR optimization strategy. The application ensures stability and high efficiency of target segmentation.

Description

TECHNICAL FIELD

[0001] The application relates to an interactive click-based video target segmentation method and belongs to the technical field of image processing. BACKGROUND

[0002] Video target segmentation technology has been widely applied in fields such as monitoring, automatic driving and video editing. Some existing algorithms can achieve high-precision target segmentation in short video sequences, but for long video sequences, especially in complex background and scenes with significant light changes, the effect of video target segmentation still has great challenges. Traditional video target segmentation methods mostly rely on mask-based target tracking and target recognition technology, which realizes target segmentation by mask propagation or feature matching between target regions of video frames. However, this method is prone to error accumulation when processing long videos due to unstable target appearance changes and motion trajectories, resulting in a decline in target segmentation effect.

[0003] Some new algorithms have begun to focus on improving segmentation effect through deep learning and feature propagation technology. For example, the video segmentation method based on deep learning model can realize real-time target segmentation in dynamic scenes. However, this method has high demand for computing resources, and the processing effect for long videos is still limited. Especially in complex outdoor scenes, due to the lack of sufficient texture information and unstable light conditions, traditional deep learning-based video segmentation algorithms often have difficulty in stable target segmentation.

[0004] Therefore, existing video target segmentation algorithms need to improve computing efficiency, accuracy and robustness when facing long video sequences and complex dynamic scenes. SUMMARY

[0005] The purpose of the present application is to overcome the problems in the prior art and provide an interactive click-based video target segmentation method that can use deep learning algorithms and feature propagation mechanisms to dynamically model and segment targets in videos, thereby achieving higher segmentation accuracy, enhancing the continuity and consistency of the segmentation process, improving the target tracking ability in long video sequences, ensuring the stability and efficiency of target segmentation, and addressing complex backgrounds, light changes, target occlusion and motion blur.

[0006] To solve the above technical problems, the interactive click-based video target segmentation method of the present application comprises the following steps:

[0007] S1: receiving a click point of a user on a video frame as an initial target label;

[0008] S2: extracting local features around the click point through a deep learning model , constructing a preliminary representation of the target region;

[0009] S3: further extracting target features of the target region and propagating features in both spatial and temporal dimensions;

[0010] S4: long-term sequential target mapping and tracking of the target region;

[0011] S5: cross-frame target optimization of the target region to achieve dynamic adjustment and optimization of the final target region;

[0012] S6: adjusting the target segmentation boundary and target position in each frame by combining the PID control algorithm and LQR optimization strategy;

[0013] S7: determining the final target segmentation box according to the final confirmed target segmentation boundary and target position in S6, and generating the mask of the target.

[0014] Further, S2 specifically includes the following steps:

[0015] S2.1: extracting local features around the image region of the click point through a deep learning model to obtain local features , the local features including multi-dimensional information of the target region's color, texture, and shape,

[0016] S2.2: constructing a preliminary representation of the target in the current frame through the multi-dimensional information.

[0017] Further, S3 specifically includes the following steps:

[0018] S3.1: target feature extraction: extracting features of the click point and its adjacent regions through a deep learning model to obtain richer spatial feature representations than the preliminary representation;

[0019] S3.2: target feature propagation:

[0020] 1) In the spatial dimension, propagate feature information to surrounding areas by analyzing the feature similarity of adjacent regions to dynamically expand the target region: calculate the similarity measure between the target region and its adjacent regions, identify adjacent regions with similar features to the target region, and fuse the adjacent regions with similar features into the existing target region to form a new target region.

[0021] 2) In the temporal dimension, combine the continuity and motion information between frames to propagate target region information from the current frame to the next frame.

[0022] Further, the target feature propagation process is as follows:

[0023] in, is the feature representation of the current frame, is the feature propagation function, which depends on the previous frame and the current frame feature information; the target propagation also needs to model the target's motion trend and appearance changes through time convolution or recurrent neural network, so that the target can be dynamically adjusted and optimized between consecutive frames.

[0024] Furthermore, S4 specifically includes the following steps:

[0025] S4.1: A convolutional neural network based on a deep learning model processes each frame in the video and extracts key feature information of the target, including the target's shape, edges, texture, and color.

[0026] S4.2: Fusing the key feature information of the target across multiple frames, and establishing a global target feature map by integrating temporal information;

[0027] S4.3: Finally, a three-dimensional model of the target is constructed based on the global target feature map, and the predicted position of the target is automatically adjusted by updating the spatial position and motion trajectory of the target in the three-dimensional model in real time. The three-dimensional model can accurately feedback the position, direction, and motion state of the target in three-dimensional space.

[0028] Furthermore, cross-frame target optimization specifically includes the following operations:

[0029] 1) Combine feature similarity metrics for region matching to further improve the accuracy of cross-frame targets;

[0030] 2) After using the attention mechanism to enhance the key areas in the initial stage of object segmentation, background modeling technology is used to separate the foreground object and the dynamic background;

[0031] 3) Optimizing the object boundary through morphological operations in the post-processing stage of object segmentation;

[0032] 4) Combining the granularity information of the click point, the attention mechanism, and the morphological operation to dynamically adjust the segmentation details of the target area to obtain the final optimized target area , where: is the weight matrix, is the feature map at the current moment.

[0033] Furthermore, in the process of building the target's 3D model, the inter-frame optical flow estimation method is used to establish pixel-level correspondences to track the target's motion trajectory and achieve precise alignment of the target between different frames.

[0034] a given optical flow field of the inter-frame optical flow estimation method wherein respectively represent the horizontal and vertical direction optical flow components, and the cross-frame consistency optimization of the target region is expressed as: ,

[0035] wherein: is a smoothing coefficient, controlling the fusion degree of the current frame features and the propagated features, is the local feature of the click point p at time t, is the optical flow field of the click point p at time t, and t-1 represents the previous frame at time t.

[0036] Further, the three-dimensional model of the target can be reconstructed through the relative position information, motion trajectory and appearance change of the target between consecutive frames.

[0037] Further, the target features in each frame are extracted through a deep learning model,

[0038] and the matching is performed by calculating the feature similarity of the target region between adjacent frames;

[0039] For regions with high similarity, accurate region matching is performed to ensure the consistency of the target in the video sequence;

[0040] For regions with low similarity or occlusion, a dynamically updated matching strategy is introduced to further optimize the cross-frame matching effect. The specific feature similarity calculation formula is as follows:

[0041] wherein: f(q) is the representation of f(p) in the feature space in the adjacent frame, is a scale parameter.

[0042] Further, S6 specifically includes real-time adjustment of the segmentation boundary of the target region through a PID control algorithm, adjustment of the target position and speed relying on the feedback signal of the target motion trajectory; global adjustment and optimization of the target motion trajectory through an LQR optimization strategy, specifically including the following steps:

[0043] S6.1: dynamically adjust the deviation between the segmentation boundary and the actual boundary of the target through a PID control algorithm, to ensure that the target segmentation boundary always follows the motion trajectory of the target and maintains accuracy: according to the difference between the segmentation boundary and the actual position of the target, calculate the three parameters of proportion, integral and differential, and adjust the boundary position in real time according to the proportion, integral and differential, and strengthen the correction of the target speed and position when the target moves quickly or deviates;

[0044] S6.2: Adjust the speed and position of the target through PID control and motion analysis: track the motion trajectory of the target in real time, analyze the position and speed changes of the target and predict the speed and position of the target in the next frame according to the motion law of the target, and dynamically adjust the segmentation boundary of the target region;

[0045] S6.3: LQR optimization strategy establishes a state space model of target position and speed, comprehensively considers the smoothness and accuracy of target motion trajectory, calculates the optimal control input, and adjusts the motion trajectory of the target according to the calculated optimal control input, so that the motion trajectory of the target is optimal between multiple frames.

[0046] Compared with the prior art, the present application has the following beneficial effects:

[0047] 1. By using the click point provided by the user as the initial annotation, the present application can accurately capture the target region. The feature propagation mechanism not only gradually refines the target region, but also dynamically adjusts the propagation direction and range to ensure that the target boundary in each frame is reasonably expanded and optimized. The temporal optimization not only considers the feature relationship between the previous and subsequent frames, but also comprehensively considers the motion information between the frames to ensure the consistency and accuracy of target segmentation in long video sequences. When processing the target segmentation task in long period videos, the segmentation accuracy can be effectively improved and the accumulation of errors can be reduced, thereby achieving higher quality video target segmentation effect.

[0048] 2. By combining deep learning model, optical flow estimation and background modeling, not only the target segmentation accuracy in each frame is improved, but also the spatio-temporal consistency of cross-frame target segmentation results is ensured. This method can cope with complex background, light change, target occlusion and motion blur, etc., providing strong ability for target segmentation in dynamic scenes, and can realize accurate tracking and real-time segmentation of fast moving targets, and can adapt to various complex environmental conditions, greatly improving the robustness and practicality of the segmentation model.

[0049] 3. The feature propagation mechanism ensures the continuity and consistency of the target in time, which effectively solves the segmentation errors caused by the motion, appearance change or occlusion of the target between video frames, and avoids the shift of the target position and the loss of the appearance. Through continuous feature propagation and update, the segmentation results of the model in the video sequence show stability and consistency, thereby improving the overall performance of the video target segmentation task.

[0050] 4. The three-dimensional model of the target can accurately reflect the position, direction and motion state of the target in three-dimensional space, avoiding tracking errors caused by target motion, occlusion or appearance change. For example, when facing high-speed motion or complex background, the future position of the target can be accurately predicted relying on the information between consecutive frames, ensuring the spatio-temporal consistency of target segmentation.

[0051] 5、PID control and LQR optimization strategy can work together to achieve accurate adjustment of video target segmentation. By continuously updating the target position and motion trajectory information, and combining the strategy of adjusting the segmentation boundary in real time, this method ensures the accurate segmentation of the target in dynamic scenes. This not only significantly improves the accuracy of target segmentation, but also enhances the stability and robustness of the segmentation results in long video sequences, ensuring high-quality segmentation results in various complex dynamic environments. BRIEF DESCRIPTION OF DRAWINGS

[0052] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, which are provided for reference and illustration only, and are not intended to limit the present application.

[0053] Figure 1 The flowchart of the present application is shown in the figure;

[0054] Figure 2 The interactive click schematic diagram is shown in the figure;

[0055] Figure 3 The interactive video segmentation scene is shown in the figure;

[0056] Figure 4 The flowchart of step 2 in the present application is shown in the figure;

[0057] Figure 5 The flowchart of step 3 in the present application is shown in the figure;

[0058] Figure 6 The flowchart of step 4 in the present application is shown in the figure;

[0059] Figure 7 The specific implementation schematic diagram of interactive click is shown in the figure;

[0060] Figure 8 The segmentation effect of interactive click in video is shown in the figure;

[0061] Figure 9 The segmentation effect of interactive click in the first frame of video is shown in the figure. DETAILED DESCRIPTION

[0062] In order to make the technical means, creative features, purposes and effects realized by the present application easy to understand, the present application will be further described below in conjunction with specific figures.

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0064] The detailed implementation of the present application from video input to mask output is shown in the figureFigure 1 As shown in the accompanying drawings.

[0065] As shown in the accompanying drawings. Figure 2 The segmentation method of the present application only needs to click the annotation information without relying on additional information such as optical flow.

[0066] The application of the present application on instruments such as mechanical arms is as shown in the accompanying drawings. Figure 3 The interactive click-based video target segmentation method of the present application specifically includes the following steps: S1: receiving a user's click point on a video frame As an initial target annotation.

[0067] S2: extracting local features around the click point through a deep learning model , constructing a preliminary representation of the target region; as shown in the accompanying drawings, S2 specifically includes the following steps: Figure 4 S2.1: extracting local features around the image region through a deep learning model, obtaining local features

[0068] ,

[0069] The local features include multi-dimensional information of the target region, including color, texture and shape,

[0070] S2.2: constructing a preliminary representation of the target in the current frame through multi-dimensional information.

[0071] S3: further target feature extraction of the target region and feature propagation in both spatial and temporal dimensions; as shown in the accompanying drawings, S3 specifically includes the following steps: Figure 5

[0072] S3.1: target feature extraction: through the convolutional neural network CNN of the deep learning model, the features of the click point and its adjacent region are extracted, obtaining a more rich spatial feature representation than the preliminary representation, i.e. extracting the semantic features of the target, including the category, texture pattern and semantic association between targets and other high-level information of the target;

[0073] S3.2: target feature propagation:

[0074] 1) in the spatial dimension, the feature information is propagated to the surrounding region by analyzing the feature similarity of the adjacent region, to dynamically expand the target region: by calculating the similarity measure between the target region and its adjacent region, the adjacent region with similar features to the target region is identified and the adjacent region with similar features is fused into the existing target region to form a new target region;

[0075] ​​2) In the time dimension, the target region information is propagated from the current frame to the next frame by combining the continuity and motion information between frames. Through the feature spatio-temporal propagation method, the target region can be dynamically tracked and refined in consecutive video frames, solving the challenges caused by target motion, external interference, etc.

[0076] The target feature propagation process is as follows:

[0077] wherein, is the feature representation of the current frame, is the feature propagation function, which depends on the feature information of the previous frame and the current frame ; the target propagation also needs to model the motion trend and appearance change of the target through time convolution or recurrent neural network, so that the target can be dynamically adjusted and optimized between consecutive frames.

[0078] S4: Long-term sequential target mapping and tracking of the target region;

[0079] As shown in Figure 7 , the target segmentation model in the deep learning model adopted by the application is a Maskformer model, as shown in Figure 6 , S4 specifically includes the following steps:

[0080] S4.1: Based on the convolutional neural network of the deep learning model, the key feature information of the target is extracted by processing each frame in the video, including the shape, edge, texture and color of the target;

[0081] S4.2: Fuse the key feature information of the target between multiple frames, and establish a global target feature atlas through the integration of time sequence information; the global target feature atlas can not only describe the features of the target in a single frame image, but also capture the change trend of the target in the entire video sequence.

[0082] S4.3: According to the global target feature atlas, a three-dimensional model of the target is constructed, and the predicted position of the target is automatically adjusted by updating the spatial position and motion trajectory of the target in the three-dimensional model in real time. The three-dimensional model can accurately feedback the position, direction and motion state of the target in the three-dimensional space. And through the relative position information, motion trajectory and appearance change of the target between consecutive frames, the three-dimensional model of the target can be reconstructed. Avoid tracking errors caused by target motion, occlusion or appearance change. For example, when facing high-speed motion or complex background, the future position of the target can still be accurately predicted relying on the information between consecutive frames, ensuring the spatio-temporal consistency of target segmentation.

[0083] In the process of establishing the three-dimensional model of the target, the inter-frame optical flow estimation method is used to establish the pixel-level correspondence relationship to track the motion trajectory of the target, and the target is accurately aligned between different frames;

[0084] Given optical flow field of inter-frame optical flow estimation method , wherein , respectively represent the horizontal and vertical direction flow components, and the cross-frame consistency optimization of the target region is represented as: ,

[0085] In the formula: is a smoothing coefficient, which controls the fusion degree of the current frame features and the propagated features, is the local feature of the click point p at time t, is the optical flow field of the click point p at time t, and t-1 represents the last frame at time t.

[0086] S5: Cross-frame target optimization is performed on the target region to realize dynamic adjustment and optimization of the final target region;

[0087] The cross-frame target optimization specifically includes the following operations:

[0088] 1) Combine feature similarity measurement for region matching to further improve the accuracy of cross-frame target;

[0089] 2) After using attention mechanism to strengthen key regions in the initial stage of target segmentation, use background modeling technology to separate foreground target and dynamic background, which is specifically shown in the following ways: in the process of video target segmentation, the saliency and importance of different regions usually differ greatly. By introducing attention mechanism, the deep learning model can automatically focus on the most recognizable regions in the target and give higher weights to these key regions, so as to more accurately identify and segment the core part of the target. As for the unimportant or easily confused regions, lower weights are given to reduce their influence, so as to improve the overall segmentation quality;

[0090] 3) In the post-processing stage of target segmentation, the target boundary is optimized through morphological operation;

[0091] 4) Combine the granularity information of click points, attention mechanism and morphological operation to dynamically adjust the segmentation details of the target region to obtain the final optimized target region , wherein: is a weight matrix, is the feature map at the current time. In view of the granularity information of the click points, the system can dynamically adjust the segmentation details of the target region to ensure that the final segmentation result is more consistent with the morphology and characteristics of the target.

[0092] The target features in each frame are extracted through a deep learning model,

[0093] And by calculating the feature similarity of the target area between adjacent frames to match;

[0094] For regions with high similarity, accurate region matching is performed to ensure consistency of the target in the video sequence;

[0095] For regions with low similarity or occlusion, a dynamic updating matching strategy is introduced to further optimize the cross-frame matching effect, such as context-based matching or correction of low-confidence regions. The specific feature similarity calculation formula is as follows:

[0096] In the formula: f(q) is the representation of f(p) adjacent frame in feature space, is a scale parameter.

[0097] In this way, the system not only improves the stability of the target area in long video sequences, but also effectively deals with various complex movements and changes, reduces the interference of factors such as occlusion and deformation on the segmentation effect, and thus realizes more accurate and consistent video target segmentation.

[0098] Morphological operation is an image processing method that refines and corrects target regions through expansion, contraction, opening operation and closing operation, etc. Through these operations, the noise of the target edge can be eliminated, the irregular boundary can be smoothed, and the defects caused by occlusion or light change can be repaired. Morphological operation can effectively improve the integrity and continuity of the segmentation result, and ensure that the final target region conforms to the real physical form.

[0099] In the present application, the deep learning model adopts a decomposition algorithm when performing feature extraction, which converts the global target segmentation problem into a local target segmentation task, gradually optimizes the segmentation accuracy of each frame, and thus realizes efficient segmentation of the global target. This method reduces the difficulty of segmentation in each frame and effectively avoids the error propagation and accumulation problems that may occur in the global segmentation process.

[0100] Specifically, the decomposition algorithm first performs rough segmentation on the target in the video frame and extracts possible target regions. Then, using a local optimization strategy, the target boundary is fine-tuned within each frame to ensure that the target segmentation result of each frame has high accuracy. To achieve this process, the algorithm uses a deep learning model to extract features of the target in each local region.

[0101] Through frame-by-frame optimization, the decomposition algorithm can continuously refine the target segmentation result in each frame, and ultimately achieve efficient target segmentation in the global range. This step-by-step optimization strategy enables the model to maintain high segmentation accuracy in complex backgrounds, while avoiding error propagation problems in the global target segmentation process. In addition, the decomposition algorithm can also flexibly adjust the local segmentation strategy according to the actual situation of each frame, to adapt to complex situations such as target shape changes, occlusions, and dynamic movements.

[0102] S6: Adjust the target segmentation boundary and target position in each frame by combining the PID control algorithm and the LQR optimization strategy;

[0103] S7: Determine the final target segmentation frame according to the final confirmed target segmentation boundary and target position in S6, and generate the mask of the target.

[0104] S6 specifically includes real-time adjustment of the segmentation boundary of the target area by the PID control algorithm, and adjustment of the target position and speed by the feedback signal of the target motion trajectory; global adjustment and optimization of the target motion trajectory by the LQR optimization strategy, which specifically includes the following steps:

[0105] S6.1: Dynamic adjustment of the deviation between the segmentation boundary and the actual boundary of the target by the PID control algorithm to ensure that the target segmentation boundary always follows the motion trajectory of the target and maintains accuracy: according to the difference between the segmentation boundary and the actual position of the target, calculate the proportional, integral, and differential parameters, and adjust the boundary position in real time according to the proportional, integral, and differential parameters, and strengthen the correction of the target speed and position when the target moves quickly or deviates;

[0106] S6.2: Adjust the speed and position of the target by PID control and motion analysis: real-time tracking of the motion trajectory of the target, analysis of the position and speed change of the target and prediction of the speed and position of the target in the next frame according to the motion law of the target, dynamic adjustment of the segmentation boundary of the target area;

[0107] S6.3: LQR optimization strategy establishes a state space model of the target position and speed, considers the smoothness and accuracy of the target motion trajectory, calculates the optimal control input, and adjusts the motion trajectory of the target according to the calculated optimal control input, so that the motion trajectory of the target remains optimal between multiple frames through global planning, and the calculation method is as follows:

[0108] First, calculate , where: is the coordinate of the center point of the target in the three-dimensional model, is the motion speed of the target, is the predicted point of the target; then define the predicted point of the target in the next frame , process noise for simulating uncertainty in target motion, is the acceleration variable of the target, and A and B are both target motion matrix models in the LQR system. .

[0109] The present application combines PID control and LQR optimization strategies, and can effectively adjust the target segmentation boundary and target position in each frame according to the target trajectory and the segmentation region information of the current frame, thereby ensuring the continuity and consistency of the target in the entire video sequence.

[0110] The PID control algorithm is used to adjust the boundary of target segmentation in real time, and relies on the feedback signal of the target trajectory to adjust the position and speed. Through real-time detection and error analysis of the target, the PID algorithm quickly responds to the deviation of the target position according to the adjustment mechanism of the three items of proportion, integral and differential, so that the target segmentation boundary always remains accurate during the movement process. For example, when the target moves faster or has a larger deviation, the PID control algorithm will strengthen the correction of the speed and position to reduce the error. The PID algorithm can effectively handle the dynamic changes in the target movement process and reduce the segmentation error caused by external disturbances such as occlusion, background noise, etc.

[0111] At the same time, the LQR optimization strategy is used to globally optimize the target trajectory, ensuring that the segmentation effect of the target at each time in space can reach the optimal. The LQR strategy combines the prediction of the target trajectory and position in multiple frames of video through accurate modeling of the target motion state, global adjustment and optimization, thereby improving the spatio-temporal consistency of target segmentation. By introducing LQR strategy optimization, the position deviation and morphological transformation problems of the target during the movement process can be effectively reduced, and the accuracy of the segmentation result can be improved.

[0112] Therefore, the present application adopts PID control algorithm and LQR optimization strategy, combines the target trajectory and segmentation region, accurately adjusts the segmentation boundary and target position, and ensures the accurate segmentation of the target in the entire video sequence.

[0113] The overall video segmentation effect of the present application is shown in Figure 8 The segmentation effect of the first frame of the video of the present application is shown in Figure 9 .

[0114] Different from the traditional static image target segmentation method, the video target segmentation method needs to consider not only the target information of each frame, but also effectively process the time sequence dependence between video frames to ensure that the target segmentation result can remain consistent in the continuity of the video. By combining the target region output by the deep learning model and the optical flow estimation result, the dynamic change of the target can be effectively captured, thereby improving the consistency of the target in time and avoiding segmentation errors caused by motion blur or rapid movement. This optimization based on motion information can still maintain high segmentation accuracy even in the case of rapid target movement or occlusion. Through multi-level feature extraction and spatio-temporal information fusion, the method of the present application can effectively reduce the computational cost while ensuring accuracy when processing continuous video frames. Finally, through a real-time feedback mechanism, the system can quickly correct the segmentation result to ensure the accuracy and consistency of the target segmentation in the time dimension, adapting to various complex dynamic environments.

[0115] The above is only a preferred embodiment of the present application, which shows and describes the basic principles, main features and advantages of the present application, and does not limit the patent protection scope of the present application. Those skilled in the art should understand that the present application is not limited by the above embodiments. In addition to the above embodiments, the present application can have other implementation manners without departing from the spirit and scope of the present application. The present application can also have various changes and improvements. Any technical solution formed by equivalent replacement or equivalent transformation falls within the scope of protection required by the present application. The scope of protection required by the present application is defined by the appended claims and their equivalents. The technical features not described in the present application can be realized by or using existing technology, which will not be described here.

Claims

1. A video object segmentation method based on interactive click, comprising the following steps: S1: Receive the user's click point on the video frame As the initial target annotation; S2: Extract local features around the click point through deep learning model , construct a preliminary representation of the target area; S3: Further target feature extraction is performed on the target area and feature propagation is performed in both spatial and temporal dimensions. S3 specifically includes the following steps: S3.1: Target feature extraction: Extract features from the click point and its surrounding area through a deep learning model to obtain a richer spatial feature representation than the initial representation; S3.2: Target Feature Propagation: 1) In the spatial dimension, by analyzing the feature similarity of neighboring regions, feature information is propagated to surrounding areas to dynamically expand the target region: by calculating the similarity measure between the target region and its neighbors, neighboring regions with similar features are identified and merged into the existing target region to form a new target region; 2) In the temporal dimension, the target region information is propagated from the current frame to the next frame by combining the continuity and motion information between frames; S4: Long-term target mapping and tracking of the target area; S5: Perform cross-frame target optimization on the target area to achieve dynamic adjustment and optimization of the final target area; S6: Combine the PID control algorithm and LQR optimization strategy to adjust the target segmentation boundary and target position in each frame; S7: Determine the final target segmentation frame based on the target segmentation boundary and target position finally confirmed in S6, and generate a target mask.

2. The method for video object segmentation based on interactive click according to claim 1, characterized in that: S2 specifically includes the following steps: S2.1: Use the deep learning model to extract local features from the image area around the click point to obtain local features. , The local features Multi-dimensional information including the color, texture and shape of the target area, S2.2: Construct a preliminary representation of the target in the current frame using the multi-dimensional information.

3. The video target segmentation method based on interactive click according to claim 2, characterized in that: The target feature propagation process formula is as follows: in, is the feature representation of the current frame, is the feature propagation function, which depends on the previous frame and the current frame feature information; the target propagation also needs to model the target's motion trend and appearance changes through time convolution or recurrent neural network, so that the target can be dynamically adjusted and optimized between consecutive frames.

4. The method for video object segmentation based on interactive click according to claim 3, characterized in that: S4 specifically includes the following steps: S4.1: A convolutional neural network based on a deep learning model processes each frame in the video and extracts key feature information of the target, including the target's shape, edges, texture, and color. S4.2: Fusing the key feature information of the target across multiple frames, and establishing a global target feature map by integrating temporal information; S4.3: Finally, a three-dimensional model of the target is constructed based on the global target feature map, and the predicted position of the target is automatically adjusted by updating the spatial position and motion trajectory of the target in the three-dimensional model in real time. The three-dimensional model can accurately feedback the position, direction, and motion state of the target in three-dimensional space.

5. The method for video object segmentation based on interactive click according to claim 4, characterized in that: Cross-frame target optimization specifically includes the following operations: 1) Combine feature similarity metrics for region matching to further improve the accuracy of cross-frame targets; 2) After using the attention mechanism to enhance the key areas in the initial stage of object segmentation, background modeling technology is used to separate the foreground object and the dynamic background; 3) Optimizing the object boundary through morphological operations in the post-processing stage of object segmentation; 4) Combining the granularity information of the click point, the attention mechanism, and the morphological operation to dynamically adjust the segmentation details of the target area to obtain the final optimized target area , where: is the weight matrix, is the feature map at the current moment.

6. The method for video object segmentation based on interactive click according to claim 4, characterized in that: In the process of building the target's 3D model, the inter-frame optical flow estimation method is used to establish pixel-level correspondence to track the target's motion trajectory and achieve precise alignment of the target between different frames. The given optical flow field of the inter-frame optical flow estimation method ,in Represent the optical flow components in the horizontal and vertical directions respectively, and the cross-frame consistency optimization of the target area is expressed as: ， Where: is the smoothing coefficient, which controls the degree of fusion between the current frame features and the propagation features. is the local feature of the click point p at time t, is the optical flow field of the click point p at time t, and t-1 represents the previous frame at time t.

7. The method for video object segmentation based on interactive click according to claim 6, characterized in that: The three-dimensional model of the target can be reconstructed through the relative position information, motion trajectory and appearance changes of the target between consecutive frames.

8. The method for video object segmentation based on interactive click according to claim 2, characterized in that: The target features in each frame are extracted through the deep learning model. Matching is performed by calculating the feature similarity of the target area between adjacent frames; Perform accurate region matching for regions with high similarity to ensure the consistency of the target in the video sequence; For areas with low similarity or occlusion, a dynamically updated matching strategy is introduced to further optimize the cross-frame matching effect. The specific feature similarity calculation formula is as follows: Where: f(q) is the representation of the adjacent frames of f(p) in the feature space, is the scale parameter.

9. The method for video object segmentation based on interactive click according to claim 1, characterized in that: S6 specifically includes adjusting the segmentation boundary of the target area in real time through the PID control algorithm, adjusting the target position and speed based on the feedback signal of the target motion trajectory; and globally adjusting and optimizing the target motion trajectory through the LQR optimization strategy, which specifically includes the following steps: S6.1: Dynamically adjust the deviation between the segmentation boundary and the actual target boundary using a PID control algorithm to ensure that the target segmentation boundary always follows the target's motion trajectory and maintains accuracy. Based on the difference between the segmentation boundary and the actual target position, the proportional, integral, and differential parameters are calculated. The boundary position is adjusted in real time based on these proportional, integral, and differential parameters. This strengthens the correction of the target's speed and position when the target moves quickly or deviates. S6.2: Adjust the target's speed and position through PID control and motion analysis: Track the target's motion trajectory in real time, analyze the target's position and speed changes, predict the target's speed and position in the next frame based on the target's motion patterns, and dynamically adjust the target area segmentation boundaries; S6.3: The LQR optimization strategy establishes a state-space model of the target position and velocity, comprehensively considers the smoothness and accuracy of the target trajectory, calculates the optimal control input, and adjusts the target trajectory according to the calculated optimal control input. Through global planning, the target trajectory is kept optimal across multiple frames. The calculation method is as follows: First calculate , where: is the coordinate of the center point of the target in the three-dimensional model, is the target's movement speed, The predicted point of the target; then the LQR system defines the predicted point of the target in the next frame , Used to simulate the uncertainty of target motion process noise, is the acceleration variable of the target, and A and B are the target motion matrix models in the LQR system.

Images