Self-supervised video object segmentation method and system based on foreground object dense feature enhancement

Abstract

The present invention discloses a self-supervised video target segmentation method and system based on foreground target dense feature enhancement. The method includes the following steps: obtaining a video target segmentation dataset, dividing the video target segmentation dataset into dataset A and dataset B, pre-training ResNet-18 based on dataset A, extracting foreground target dense feature maps of sample pairs, and inputting them into a dense projection head to obtain dense feature vectors; constructing a pixel-guided loss function based on Huber Loss and Mask-IoU Loss; performing inter-frame relationship training based on dataset B, minimizing the calculated result of the pixel-guided loss function to train ResNet-18, and reconstructing the next frame image based on the previous frame image; and inputting the segmentation result of the previous frame image into the trained ResNet-18 to obtain the segmentation result of the next frame image. The present invention can improve the performance of the model video target segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of video target segmentation, and in particular to a self-supervised video target segmentation method and system based on foreground target dense feature enhancement. Background Art

[0002] Video target segmentation has a wide range of potential application scenarios. In the field of visual navigation, it can identify and predict the target's motion trajectory, thereby realizing automatic navigation. In the field of video surveillance, it can capture key targets and track and monitor them. In the field of film and television production, such as movie and television special effects and short video live broadcasts, video target segmentation technology can be used to separate the foreground from the background, modify or replace the background, and create scenes that do not exist or are difficult to achieve in reality, thereby enhancing the impact of the information.

[0003] However, the task of video object segmentation still faces several challenges and problems in current practical applications. First, there is the issue of complex backgrounds. When there are complex interactions and occlusions between the object and the background in the video, the task of video object segmentation becomes more difficult. Second, the objects in the video may deform, transform, or move, which causes the appearance and shape of the object to change between frames. For these dynamic objects, accurately segmenting the object is a challenging task. In addition, the task of video object segmentation also faces problems such as lack of data and difficulty in annotation. Compared with image segmentation, video object segmentation requires more training data and annotation information.

[0004] Video object segmentation is a complex, high-level visual task in computer vision research, with significant research significance and value. In practical applications, specifically addressing complex and difficult problems, improving algorithm accuracy, efficiency, and robustness, and exploring more effective training methods and data augmentation techniques all hold high research value and practical implications for real-world applications, worthy of further in-depth research. Summary of the Invention

[0005] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a self-supervised video target segmentation method and system based on foreground target dense feature enhancement. The present invention uses a pixel-guided loss function to train the model, so that the network pays more attention to those pixels that have a greater contribution to the model performance, thereby improving the model's video target segmentation performance.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The present invention provides a self-supervised video object segmentation method based on foreground object dense feature enhancement, comprising the following steps:

[0008] Obtain a video target segmentation dataset and divide it into dataset A and dataset B. Dataset A is a set of two randomly selected frames of images used for fine-grained feature pre-training, and dataset B is a set of two adjacent frames of images used for inter-frame relationship learning.

[0009] Two frames from the same video in dataset A are considered a positive sample pair, while any two frames from different videos are considered a negative sample pair. The positive or negative sample pairs in dataset A are input into the backbone network ResNet-18 for pre-training, from which dense feature maps of the foreground and target of the sample pairs are extracted. The dense feature maps of the foreground and target are input into the dense projection head, which outputs a dense feature vector.

[0010] Use the smooth L1 loss Huber Loss and the mask intersection-over-union loss Mask-IoU Loss combination as the supervision signal to construct the pixel-guided loss function;

[0011] Calculate the pixel-by-pixel similarity of the two frames, rank the similarities, select pixels with a set ranking ratio, calculate the smooth L1 loss Huber Loss, and use Mask-IoU Loss to optimize pixel-level changes;

[0012] The feature differences between adjacent frames are calculated based on the pixel-guided loss function. Inter-frame relationship training is performed based on dataset B. The calculated result of the pixel-guided loss function is minimized to train the backbone network ResNet-18, and the next frame image is reconstructed based on the previous frame image.

[0013] Obtain the segmentation result of the previous frame image, input the segmentation result of the previous frame image into the trained backbone network ResNet-18, and obtain the segmentation result of the next frame image.

[0014] As a preferred technical solution, the dense projection head adopts a 1*1 convolution layer.

[0015] As a preferred technical solution, pre-training uses contrast loss, which is specifically expressed as:

[0016]

[0017] Among them, r·t + is a positive pixel sample pair, r·t - is a negative pixel pair, and τ is a temperature hyperparameter used to adjust the proportion of positive and negative sample learning.

[0018] As a preferred technical solution, the pixel-guided loss function is constructed as follows:

[0019]

[0020]

[0021]

[0022] Where L represents the pixel-guided loss function, l h Represents smooth L1 loss Huber Loss, l m Represents the Mask-IoU Loss, λ represents the trade-off parameter, k represents the threshold of the prediction deviation in the smooth L1 loss Huber Loss, Ω represents the set of all pixels in the real next frame image, and θ(Ω) is the function used to guide the most difficult area of ​​Huber Loss. Specifically, the most difficult area is the pixel area where the difference between adjacent frames ranks in the top 40%. Represents the next frame of image reconstructed, Represents the actual next frame image.

[0023] As a preferred technical solution, the feature differences between adjacent frame images are calculated based on the pixel-guided loss function, and inter-frame relationship training is performed based on dataset B. The calculated result of the pixel-guided loss function is minimized to train the backbone network ResNet-18, and the next frame image is reconstructed based on the previous frame image. Specifically, it includes:

[0024] Take two consecutive frame images I t-1 and I t As input, use the encoder ￡(I, θ) from I t-1 and I t Extract features from t-1 , X t ∈R H×W×C , feature X t-1 and feature X t The pairwise difference between them is used as the reconstruction feature to represent the spatial difference between the two frame images;

[0025] The feature X t-1 Neutral and Feature X t The area around the pixel i at the same position in is defined as the response area Z of pixel i, based on the feature X t-1 With feature X t Generate affinity matrix for all pixels Specifically expressed as:

[0026]

[0027]

[0028] Where t represents the index of the frame in the video, H and W represent the height and width of the frame respectively, c represents the number of channels, <·,·> represents the dot product operation of two vectors, i is the pixel point of the image feature of the previous frame, j is the pixel point of the image feature of the next frame in the Z region, They respectively represent the longitudinal slice vector features along the channel direction in the pixel points of the previous frame image features and the pixel points of the next frame image features in the Z area.

[0029] The present invention also provides a self-supervised video target segmentation system based on foreground target dense feature enhancement, comprising: a data acquisition module, a data partitioning module, a pre-training module, a pixel-guided loss function construction module, an inter-frame relationship training module, and a target segmentation module;

[0030] The data acquisition module is used to acquire a video target segmentation data set;

[0031] The data partitioning module is used to divide the video object segmentation dataset into dataset A and dataset B, wherein dataset A is a set of two randomly selected frames of images used for fine-grained feature pre-training, and dataset B is a set of two adjacent frames of images used for inter-frame relationship learning;

[0032] The pre-training module is used to input positive sample pairs or negative sample pairs in dataset A into the backbone network ResNet-18 for pre-training, extract the foreground target dense feature map of the sample pair, input the foreground target dense feature map into the dense projection head, and output a dense feature vector. Two frames of images in dataset A belonging to the same video are considered as positive sample pairs, and any two frames from different videos are considered as negative sample pairs;

[0033] The pixel-guided loss function construction module is used to use the smooth L1 loss Huber Loss and the mask intersection-over-union loss Mask-IoU Loss combination as a supervision signal to construct a pixel-guided loss function;

[0034] Calculate the pixel-by-pixel similarity of the two frames, rank the similarities, select pixels with a set ranking ratio, calculate the smooth L1 loss Huber Loss, and use Mask-IoU Loss to optimize pixel-level changes;

[0035] The inter-frame relationship training module is used to calculate the feature differences between adjacent frame images based on the pixel-guided loss function, perform inter-frame relationship training based on data set B, minimize the calculation result of the pixel-guided loss function to train the backbone network ResNet-18, and reconstruct the next frame image based on the previous frame image;

[0036] The target segmentation module is used to obtain the segmentation result of the previous frame image, input the segmentation result of the previous frame image into the trained backbone network ResNet-18, and obtain the segmentation result of the next frame image.

[0037] As a preferred technical solution, the dense projection head adopts a 1*1 convolution layer.

[0038] As a preferred technical solution, pre-training uses contrast loss, which is specifically expressed as:

[0039]

[0040] Among them, r·t + is a positive pixel sample pair, r·t - is a negative pixel pair, and τ is a temperature hyperparameter used to adjust the proportion of positive and negative sample learning.

[0041] As a preferred technical solution, the pixel-guided loss function is constructed as follows:

[0042]

[0043]

[0044]

[0045] Where L represents the pixel-guided loss function, l h Represents smooth L1 loss Huber Loss, l m Represents the Mask-IoU Loss, λ represents the trade-off parameter, k represents the threshold of the prediction deviation in the smooth L1 loss Huber Loss, Ω represents the set of all pixels in the real next frame image, and θ(Ω) is the function used to guide the most difficult area of ​​Huber Loss. Specifically, the most difficult area is the pixel area where the difference between adjacent frames ranks in the top 40%. Represents the next frame of image reconstructed, Represents the actual next frame image.

[0046] As a preferred technical solution, the feature differences between adjacent frame images are calculated based on the pixel-guided loss function, and inter-frame relationship training is performed based on dataset B. The calculated result of the pixel-guided loss function is minimized to train the backbone network ResNet-18, and the next frame image is reconstructed based on the previous frame image. Specifically, it includes:

[0047] Take two consecutive frame images I t-1 and I t As input, use the encoder ￡(I, θ) from I t-1 and I tExtract features from t-1 , X t ∈R H×W×C , feature X t-1 and feature X t The pairwise difference between them is used as the reconstruction feature to represent the spatial difference between the two frame images;

[0048] The feature X t-1 Neutral and Feature X t The area around the pixel i at the same position in is defined as the response area Z of pixel i, based on the feature X t-1 With feature X t Generate affinity matrix for all pixels Specifically expressed as:

[0049]

[0050]

[0051] Where t represents the index of the frame in the video, H and W represent the height and width of the frame respectively, c represents the number of channels, <·,·> represents the dot product operation of two vectors, i is the pixel point of the image feature of the previous frame, j is the pixel point of the image feature of the next frame in the Z region, They respectively represent the longitudinal slice vector features along the channel direction in the pixel points of the previous frame image features and the pixel points of the next frame image features in the Z area.

[0052] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0053] (1) The present invention uses a dense feature pre-training method to improve the fine-grained feature extraction capability of ResNet-18, and uses a pixel-guided loss function to train the model, so that the network pays more attention to pixels that contribute more to the model performance, thereby improving the model's video target segmentation performance.

[0054] (2) The present invention adopts a multi-scale pixel-level feature fusion strategy to solve the problem of low model robustness in difficult sample scenarios. When facing different samples, features of different scales are selected for fusion, which improves the practical segmentation effect of the model in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 Schematic diagram of the process of the self-supervised video object segmentation method based on foreground object dense feature enhancement of the present invention;

[0056] Figure 2 Schematic diagram of the implementation process of fine-grained feature pre-training of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] Example 1

[0059] like Figure 1 As shown, this embodiment provides a self-supervised video object segmentation method based on foreground object dense feature enhancement, including the following steps:

[0060] Step 1: Obtain a video object segmentation dataset and divide it into dataset A and dataset B. Dataset A is a set of two randomly selected frames of images used for fine-grained feature pre-training, and dataset B is a set of two adjacent frames of images used for inter-frame relationship learning.

[0061] Step 2: Consider any two frames of images in the same video in dataset A as a positive sample pair, and any two frames in different videos as a negative sample pair, such as Figure 2 As shown in the figure, the positive or negative sample pairs in dataset A are input into the backbone network ResNet-18 for fine-grained feature pre-training, from which the foreground target dense feature map of the sample pair is extracted. The foreground target dense feature map is input into the dense projection head (1*1 convolutional layer). The backbone network ResNet-18 and the series dense projection head are jointly learned to extract more discriminative and rich feature representations, and finally a dense feature vector is output;

[0062] In the pre-training stage, the ResNet-18 network is trained using the A dataset for 40 epochs. The weight decay is set to 0.0001, the initial learning rate is set to 0.05, and the loss function uses the contrast loss, which is specifically expressed as:

[0063]

[0064] Among them, r·t + is a positive pixel sample pair, r·t - is a negative pixel pair, τ is a temperature hyperparameter that controls the proportion of positive sample learning and negative sample learning, and its value is preferably 1;

[0065] In this embodiment, pre-training the backbone network ResNet-18 based on dataset A is a process of strengthening the dense features of foreground objects, allowing the backbone network ResNet-18 to focus more on fine-grained information, providing an effective self-supervised learning method for the video object segmentation task. By inputting an image with the segmentation result of the previous frame into the trained network, the segmentation result of the next frame image can be output;

[0066] Step 3: Construct a pixel-guided loss function, using a combination of smooth L1 loss Huber Loss and mask intersection-over-union loss Mask-IoU Loss as the supervisory signal, and select a fixed 40% of the more difficult pixels for Huber Loss calculation. That is, first calculate the pixel-by-pixel similarity of the two images, then rank the similarities, and take the top 40% pixels with higher values ​​for subsequent Huber Loss calculation.

[0067] The pixel-guided loss function is used during the formal training phase of the ResNet-18 backbone network. The training dataset is Dataset B from Step 1. When reconstructing the next frame from the previous one, the pixel-guided loss function is used to calculate the feature differences between adjacent frames. Minimizing the calculated result of this loss function to train the ResNet-18 backbone network enables the ResNet-18 backbone network to reconstruct the next frame from the previous one during the testing phase.

[0068] In step 3, for the next frame reconstructed and the real next frame The pixel-guided loss function forces the network to focus on difficult and valuable pixels during training, and selects a fixed portion of the most difficult pixels for HuberLoss calculation. In this loss function, the Mask-IoU loss is used to optimize pixel-level changes, and a combination of Huber Loss and Mask-IoU loss is used as the supervisory signal. The loss function specifically includes:

[0069] Huber Loss is expressed as follows:

[0070]

[0071] Wherein, k represents a threshold of prediction deviation in Huber Loss. Preferably, k is 1.

[0072] Mask-IoU Loss is expressed as follows:

[0073]

[0074] The pixel-guided loss function is expressed as follows:

[0075]

[0076] Where λ is a trade-off parameter, preferably 1, Ω represents the set of all pixels in the next real frame, and θ(Ω) is a function used to guide the most difficult area of ​​Huber Loss, specifically, the pixels that rank in the top 40% of the pixel differences between adjacent frames.

[0077] Take two consecutive frames I t-1 and I t As input, I t-1 and I t Indicated as I t-1 , I t ∈R H×W×3 , where H and W represent the height and width of the frame respectively, the subscript t represents the index of the frame in the video, and 3 represents 3 channels. t-1 and I t Extract features from t-1 , X t ∈R H×W×C , where c represents the number of channels. t-1 and X t The pairwise difference between X is used as the reconstruction feature to represent the spatial difference between the two frames. t-1 Zhong and X t A small area around the pixel i at the same position in is defined as the "response area" of pixel i (denoted as Z). The size of Z is controlled by the hyperparameter P. For convenience, Z is a square area centered on i:

[0078]

[0079] At the same time, X is defined t-1 The pixel at position i in X t The relationship between pixels in Z:

[0080]

[0081] Among them, the operation <·,·> represents the dot product of two vectors. Then, X t-1 and X t All pixels are input into the above formula to generate the affinity matrix i is the pixel point of the previous frame image feature, j is the pixel point of the next frame image feature in the Z area, They respectively represent the longitudinal slice vector features along the channel direction in the pixel points of the previous frame image features and the pixel points of the next frame image features in the Z area.

[0082] By matrix Reconstruct the next frame of the current frame and use the pixel-guided loss function to train the relationship between frames. The trained ResNet-18 network can be used for target segmentation of unlabeled videos. Multiplying them yields the reconstructed representation of the next frame. For a feature extractor, such as a ResNet-18 network, pre-training is performed using dataset A to enhance dense features. This is followed by formal training using dataset B to learn the encoder's reconstruction capabilities. Finally, during the test phase, the segmentation results of the previous frame are input and the segmentation results of the next frame are output, completing the segmentation in a frame-by-frame manner (the first frame of the video object segmentation dataset is labeled).

[0083] Step 4: After pre-training is completed, use dataset B to train the reconstruction capability of the backbone network ResNet-18, and use the pixel-guided loss function to make the network converge.

[0084] In step 4, the details of inter-frame relationship training are end-to-end training with a batch size of 12. The learning rate is initialized to 0.001, and the weight decay is set to 0.0001. Training is performed on a single NVIDIA TiTan GPU. Inter-frame relationship training enables the encoder to reconstruct the previous and next frames. When the input is the segmentation result of the previous frame, the segmentation result of the next frame is also output.

[0085] The present invention uses fine-grained pre-training and designs a dedicated foreground pixel-guided loss function. Since the difference between two adjacent frames is mainly reflected in the movement of the foreground target, focusing on 40% difficult pixels naturally pays more attention to the foreground target, allowing the network's learning to focus on the segmented targets and improving the network's ability to solve problems such as foreground target disappearance, occlusion, and rapid deformation.

[0086] Example 2

[0087] This embodiment provides a self-supervised video object segmentation system based on foreground object dense feature enhancement, including: a data acquisition module, a data partitioning module, a pre-training module, a pixel-guided loss function construction module, an inter-frame relationship training module, and an object segmentation module;

[0088] In this embodiment, the data acquisition module is used to obtain a video object segmentation dataset;

[0089] In this embodiment, the data partitioning module is used to divide the video object segmentation dataset into dataset A and dataset B, where dataset A is a set of two randomly selected frames of images used for fine-grained feature pre-training, and dataset B is a set of two adjacent frames of images used for inter-frame relationship learning;

[0090] In this embodiment, the pre-training module is used to input positive sample pairs or negative sample pairs in dataset A into the backbone network ResNet-18 for pre-training, extract the foreground target dense feature map of the sample pair from it, input the foreground target dense feature map into the dense projection head, and output a dense feature vector. Two frames of images in the same video in dataset A are considered as positive sample pairs, and any two frames in different videos are considered as negative sample pairs.

[0091] In this embodiment, the pixel-guided loss function construction module is used to use the smooth L1 loss Huber Loss and the mask intersection-over-union loss Mask-IoU Loss combination as the supervision signal to construct the pixel-guided loss function;

[0092] Calculate the pixel-by-pixel similarity of the two frames, rank the similarities, select pixels with a set ranking ratio, calculate the smooth L1 loss Huber Loss, and use the Mask-IoU Loss to optimize pixel-level changes;

[0093] In this embodiment, the inter-frame relationship training module is used to calculate the feature differences between adjacent frame images based on the pixel-guided loss function, perform inter-frame relationship training based on dataset B, minimize the calculated result of the pixel-guided loss function to train the backbone network ResNet-18, and reconstruct the next frame image based on the previous frame image;

[0094] In this embodiment, the target segmentation module is used to obtain the segmentation result of the previous frame image, input the segmentation result of the previous frame image into the trained backbone network ResNet-18, and obtain the segmentation result of the next frame image.

[0095] In this embodiment, the dense projection head adopts a 1*1 convolutional layer.

[0096] In this embodiment, pre-training uses contrast loss, which is specifically expressed as:

[0097]

[0098] Among them, r·t + is a positive pixel sample pair, r·t - is a negative pixel pair, and τ is a temperature hyperparameter used to adjust the proportion of positive and negative sample learning.

[0099] In this embodiment, a pixel-guided loss function is constructed, which is specifically expressed as:

[0100]

[0101]

[0102]

[0103] Where L represents the pixel-guided loss function, l h Represents smooth L1 loss Huber Loss, l m Represents the Mask-IoU Loss, λ represents the trade-off parameter, k represents the threshold of the prediction deviation in the smooth L1 loss Huber Loss, Ω represents the set of all pixels in the real next frame image, and θ(Ω) is the function used to guide the most difficult area of ​​Huber Loss. Specifically, the most difficult area is the pixel area where the difference between adjacent frames ranks in the top 40%. Represents the next frame of image reconstructed, Represents the actual next frame image.

[0104] In this embodiment, the feature differences between adjacent frame images are calculated based on the pixel-guided loss function, inter-frame relationship training is performed based on dataset B, and the calculated result of the pixel-guided loss function is minimized to train the backbone network ResNet-18. The next frame image is reconstructed based on the previous frame image, specifically including:

[0105] Take two consecutive frame images I t-1 and I t As input, use the encoder ￡(I, θ) from I t-1 and I t Extract features from t-1 , X t ∈R H×W×C , feature X t-1 and feature X t The pairwise difference between them is used as the reconstruction feature to represent the spatial difference between the two frame images;

[0106] The feature X t-1 Neutral and Feature X t The area around the pixel i at the same position in is defined as the response area Z of pixel i, based on the feature X t-1 With feature X t Generate affinity matrix for all pixels Specifically expressed as:

[0107]

[0108]

[0109] Where t represents the index of the frame in the video, H and W represent the height and width of the frame respectively, c represents the number of channels, <·,·> represents the dot product operation of two vectors, i is the pixel point of the image feature of the previous frame, j is the pixel point of the image feature of the next frame in the Z region, They respectively represent the longitudinal slice vector features along the channel direction in the pixel points of the previous frame image features and the pixel points of the next frame image features in the Z area.

[0110] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A self-supervised video object segmentation method based on foreground object dense feature enhancement, characterized by: The steps include: Obtain a video target segmentation dataset and divide it into dataset A and dataset B. Dataset A is a set of two randomly selected frames of images used for fine-grained feature pre-training, and dataset B is a set of two adjacent frames of images used for inter-frame relationship learning. Two frames from the same video in dataset A are considered a positive sample pair, while any two frames from different videos are considered a negative sample pair. The positive or negative sample pairs in dataset A are input into the backbone network ResNet-18 for pre-training, from which dense feature maps of the foreground and target of the sample pairs are extracted. The dense feature maps of the foreground and target are input into the dense projection head, which outputs a dense feature vector. The smooth L1 loss Huber Loss and the mask intersection-over-union loss Mask-IoU Loss are combined as supervisory signals to construct a pixel-guided loss function, which is specifically expressed as: Where L represents the pixel-guided loss function, Represents smooth L1 loss Huber Loss, Represents the Mask-IoU Loss, λ represents the trade-off parameter, k represents the threshold of the prediction deviation in the smooth L1 loss Huber Loss, Ω represents the set of all pixels in the real next frame image, and θ(Ω) is the function used to guide the most difficult area of ​​Huber Loss. Specifically, the most difficult area is the pixel area where the difference between adjacent frames ranks in the top 40%. Represents the next frame of image reconstructed, Indicates the real next frame image; Calculate the pixel-by-pixel similarity of the two frames, rank the similarities, select pixels with a set ranking ratio, calculate the smooth L1 loss Huber Loss, and use the Mask-IoULoss loss to optimize pixel-level changes; The feature differences between adjacent frames are calculated based on the pixel-guided loss function. Inter-frame relationship training is performed based on dataset B. The calculated results of the pixel-guided loss function are minimized to train the backbone network ResNet-18. The next frame is reconstructed based on the previous frame. Specifically, the following steps are performed: Take two consecutive frame images I t-1 and I t As input, use the encoder ￡(I, θ) from I t-1 and I t Extract features from t-1 , X t ∈R H×W×C , feature X t-1 and feature X t The pairwise difference between them is used as the reconstruction feature to represent the spatial difference between the two frame images; The feature X t-1 Neutral and Feature X t The area around the pixel i at the same position in is defined as the response area Z of pixel i, based on the feature X t-1 With feature X t Generate affinity matrix for all pixels Specifically expressed as: Where t represents the index of the frame in the video, H and W represent the height and width of the frame respectively, c represents the number of channels, <·,·> represents the dot product operation of two vectors, i is the pixel point of the image feature of the previous frame, j is the pixel point of the image feature of the next frame in the Z region, Respectively represent the longitudinal slice vector features along the channel direction in the pixel position of the previous frame image feature and the pixel position of the next frame image feature in the Z area; Obtain the segmentation result of the previous frame image, input the segmentation result of the previous frame image into the trained backbone network ResNet-18, and obtain the segmentation result of the next frame image.

2. The self-supervised video object segmentation method based on foreground object dense feature enhancement according to claim 1 is characterized in that The dense projection head uses a 1*1 convolutional layer.

3. The self-supervised video object segmentation method based on foreground object dense feature enhancement according to claim 1 is characterized in that Pre-training uses contrast loss, which is specifically expressed as: Among them, r·t + is a positive pixel sample pair, r·t - is a negative pixel pair, and τ is a temperature hyperparameter used to adjust the proportion of positive and negative sample learning.

4. A self-supervised video object segmentation system based on foreground object dense feature enhancement, characterized by: include: Data acquisition module, data partitioning module, pre-training module, pixel-guided loss function construction module, inter-frame relationship training module, and target segmentation module; The data acquisition module is used to acquire a video target segmentation data set; The data partitioning module is used to divide the video object segmentation dataset into dataset A and dataset B, wherein dataset A is a set of two randomly selected frames of images used for fine-grained feature pre-training, and dataset B is a set of two adjacent frames of images used for inter-frame relationship learning; The pre-training module is used to input positive sample pairs or negative sample pairs in dataset A into the backbone network ResNet-18 for pre-training, extract the foreground target dense feature map of the sample pair, input the foreground target dense feature map into the dense projection head, and output a dense feature vector. Two frames of images in dataset A belonging to the same video are considered as positive sample pairs, and any two frames from different videos are considered as negative sample pairs; The pixel-guided loss function construction module is used to use the smooth L1 loss Huber Loss and the mask intersection-over-union loss Mask-IoU Loss combination as the supervision signal to construct the pixel-guided loss function, which is specifically expressed as: Where L represents the pixel-guided loss function, Represents smooth L1 loss Huber Loss, Represents the Mask-IoU Loss, λ represents the trade-off parameter, k represents the threshold of the prediction deviation in the smooth L1 loss Huber Loss, Ω represents the set of all pixels in the real next frame image, q(Ω) is the function used to guide the most difficult area of ​​Huber Loss. Specifically, the most difficult area is the pixel area where the difference between adjacent frames ranks in the top 40%. Represents the next frame of image reconstructed, Indicates the real next frame image; Calculate the pixel-by-pixel similarity of the two frames, rank the similarities, select pixels with a set ranking ratio, calculate the smooth L1 loss Huber Loss, and use the Mask-IoULoss loss to optimize pixel-level changes; The inter-frame relationship training module is used to calculate the feature differences between adjacent frame images based on the pixel-guided loss function, perform inter-frame relationship training based on dataset B, minimize the calculation result of the pixel-guided loss function to train the backbone network ResNet-18, and reconstruct the next frame image based on the previous frame image. Specifically, it includes: Take two consecutive frame images I t-1 and I t As input, use the encoder ￡(I, θ) from I t-1 and I t Extract features from t-1 , X t ∈R H×W×C , feature X t-1 and feature X t The pairwise difference between them is used as the reconstruction feature to represent the spatial difference between the two frame images; The feature X t-1 Neutral and Feature X t The area around the pixel i at the same position in is defined as the response area Z of pixel i, based on the feature X t-1 With feature X t Generate affinity matrix for all pixels Specifically expressed as: Where t represents the index of the frame in the video, H and W represent the height and width of the frame respectively, c represents the number of channels, <·,·> represents the dot product operation of two vectors, i is the pixel point of the image feature of the previous frame, j is the pixel point of the image feature of the next frame in the Z region, Respectively represent the longitudinal slice vector features along the channel direction in the pixel position of the previous frame image feature and the pixel position of the next frame image feature in the Z area; The target segmentation module is used to obtain the segmentation result of the previous frame image, input the segmentation result of the previous frame image into the trained backbone network ResNet-18, and obtain the segmentation result of the next frame image.

5. The self-supervised video object segmentation system based on foreground object dense feature enhancement according to claim 4 is characterized in that The dense projection head uses a 1*1 convolutional layer.

6. The self-supervised video object segmentation system based on foreground object dense feature enhancement according to claim 4 is characterized in that Pre-training uses contrast loss, which is specifically expressed as: Among them, r·t + is a positive pixel sample pair, r·t - is a negative pixel pair, and τ is a temperature hyperparameter used to adjust the proportion of positive and negative sample learning.

Images