A semi-supervised video object segmentation method based on confidence gated spatio-temporal memory network

Abstract

The application discloses a kind of semi-supervised video target segmentation methods based on confidence gating spatio-temporal memory network, comprising: obtaining video segmentation related dataset and corresponding segmentation label;Encoder is constructed, and the information contained in video image is extracted;Confidence gating spatio-temporal memory reading module is constructed, and the noise in historical spatio-temporal information is filtered;Dilated convolution spatial pooling pyramid module is constructed, multi-scale target feature information is captured, and the feature recognition capability of model is improved;Segmentation decoder is constructed, and target appearance information is restored to high resolution, and target segmentation result is obtained.The method of the application can effectively reduce the influence on video target segmentation task caused by target occlusion or disappearance, further improve the recognition ability of small target or similar target local feature and the accuracy of video target segmentation.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a semi-supervised video target segmentation method based on a confidence gated spatio-temporal memory network. BACKGROUND

[0002] Video object segmentation (VOS) is a challenging task in computer vision, which has wide applications in automatic driving technology, video summary generation, human-computer interaction, etc. Semi-supervised video object segmentation is a method of only giving the prior information of the first frame of the video, and requiring to segment the target object in the remaining frames of the video. The target object in the video sequence usually has a large appearance change due to some complex factors such as deformation, occlusion, camera jitter, etc. At the same time, there may be high-speed moving objects in the video, which can easily cause prediction drift. Therefore, it is of great significance to study an efficient and accurate semi-supervised video object segmentation method for automatic driving, video summary and other application scenarios.

[0003] The existing semi-supervised video object segmentation method based on spatio-temporal memory network calculates the similarity between the current frame and the historical frame through global feature matching, and performs feature fusion on the current frame and all historical frames according to the similarity. When the target object disappears in the video sequence, this method will introduce a large amount of noise, which will reduce the segmentation accuracy. Moreover, it is not fine enough to identify small targets and very similar targets in the video sequence, and it is easy to produce false segmentation results. SUMMARY

[0004] The purpose of the present application is to overcome the shortcomings of the existing semi-supervised video object segmentation method based on historical spatio-temporal information, and to propose a semi-supervised video object segmentation method based on a confidence gated spatio-temporal memory network. The method reduces the influence of noise in the historical spatio-temporal information on the segmentation result by designing a confidence gated spatio-temporal memory reading module, and designs a dilated convolution pyramid module to increase the attention of the video object segmentation network to the local information of the target, and further improves the accuracy of the video object segmentation.

[0005] I. Technical principles

[0006] The semi-supervised video object segmentation method only provides the true value of object segmentation in the first frame, and the segmentation result of the current frame will be used as the object mask of the subsequent frame to provide reference for the segmentation of the subsequent frame. However, the prior art is difficult to solve the segmentation noise introduced by the disappearance or serious occlusion of the target, and the false segmentation caused by the too small target or too high similarity. Since the semi-supervised video object segmentation method is a time sequence method, the influence of such problems on the existing semi-supervised video object segmentation method will be further magnified. In view of the above problems, the application proposes a confidence gating spatiotemporal memory reading module, which makes the video object segmentation network focus on the video frames with higher importance rather than all the video frames, thereby reducing the influence of target occlusion or disappearance; and a dilated convolution pyramid module is proposed to increase the convolution receptive field and capture multi-scale target appearance information, further improving the recognition ability of the model to the local features of small targets or similar targets and the accuracy of video object segmentation.

[0007] II. According to the above principle, the application is realized by the following scheme:

[0008] A semi-supervised video object segmentation method based on a confidence gating spatiotemporal memory network, characterized by comprising the following steps:

[0009] (1) Obtain the data set and the segmentation label:

[0010] Obtain the semi-supervised video object segmentation data set and the corresponding segmentation label.

[0011] (2) Build a segmentation model: the model is composed of an encoder, a confidence gating spatiotemporal memory reading module, a dilated convolution pyramid module, and a decoder, and the specific construction process comprises the following steps:

[0012] (2-a) Build an encoder, including a query encoder and a memory encoder, both of which have the same structure and include a convolution module, three residual modules, and a feature embedding module.

[0013] The query encoder only encodes the query frame image information, and the current query frame image sequentially passes through the convolution module and the three residual modules of the query encoder to obtain the output feature maps x r1 , x r2 , and x r3 corresponding to the three residual modules; after the feature embedding module encodes x r3 , the feature pair (k Q , v Q ) of the current query frame image is obtained, wherein k Q and v Q are the key information and value information of the query frame image, respectively.

[0014] The memory encoder encodes the image and the corresponding mask, and the last frame image of the current query frame image is regarded as a memory frame image. The memory frame image and its mask pass through the convolution module and the three residual modules of the memory encoder in turn, and then are encoded by the feature embedding module to obtain the feature pair (k M , v M ) of the memory frame image, wherein k M and v M represent the key information and the value information of the memory frame image respectively.

[0015] The encoding results of the memory frame image and the images and their masks of the frames before the memory frame image jointly constitute a memory encoding library H wherein represents the jth feature pair in the memory encoding library H, j≤T, T is a positive integer, T represents the maximum number of feature pairs in the memory encoding library H, and 0<T<200; when the total number of the memory frame image and all the frames before the memory frame image is less than or equal to T, the memory encoding library H is jointly constituted by the encoding results of the memory frame image and the images and their masks of all the frames before the memory frame image; when the total number of the memory frame image and all the frames before the memory frame image is greater than T, the memory encoding library H is jointly constituted by the encoding results of the memory frame image and the images and their masks of the T-1 frames before the memory frame image.

[0016] (2-b) constructing a confidence gated spatio-temporal memory reading module: calculating the confidence between k Q obtained in step (2-a) and each key information in the memory encoding library H, j is a positive integer, and j≤T; the confidence gated mechanism selects the first N feature pairs with the strongest relevance to the current query frame image from the memory encoding library H according to the size of the confidence, N is a positive integer, N<T and 0<N<100; if the number of existing feature pairs in the memory encoding library H is less than N, a feature pair with a feature value of (0, 0) is supplemented to N; respectively constructing the key information set and the value information set corresponding to the N feature pairs, wherein represents the tth key information with the strongest relevance to the current query frame image, represents the tth value information with the strongest relevance to the current query frame image; calculating the weight value by using the elements in the key information set Keys, then performing weighted summation on the elements in the value information set Values by using the calculated weight value to obtain v sum , finally, performing feature splicing on v sum and v Q obtained in step (2-a) to obtain the comprehensive feature information R.

[0017] The calculation formula of the confidence gating mechanism weight calculation and weighted summation is respectively:

[0018]

[0019]

[0020] wherein, W t represents weight, exp(·) represents exponential operation with natural constant e as base.

[0021] (2-c) Constructing a dilated convolution pyramid module: the module includes four different convolution modules in parallel, which are a 1×1 size standard convolution module, a 3×3 dilated convolution module with an expansion rate of 6, a 3×3 dilated convolution block with an expansion rate of 12, and a 3×3 dilated convolution module with an expansion rate of 18; the four different convolution modules are used to perform multi-scale feature extraction on the comprehensive feature information R obtained in step (2-b), and the extracted results are pixel-by-pixel added to obtain enhanced features J.

[0022] (2-d) Constructing a decoder: performing twice 3×3 convolution and once up-sampling on the enhanced features J obtained in step (2-c) to obtain intermediate features f1; performing twice 3×3 convolution on the x r2 obtained in step (2-a) to obtain intermediate features pixel-by-pixel adding f1 and performing once up-sampling to obtain intermediate features f2; performing twice 3×3 convolution on the x r1 obtained in step (2-a) to obtain intermediate features pixel-by-pixel adding f2 and performing once 3×3 convolution and once up-sampling to obtain the prediction segmentation mask Y of the current query frame.

[0023] ​​(2-e) After the prediction segmentation mask of the current query frame image is obtained, the query frame image becomes a new memory frame image, and the next frame image of the query frame image becomes a new query frame image, the memory encoder constructed in step (2-a) is used to encode the new memory frame image, and the encoding result is saved to the memory code library H, and the memory code library is updated; the query encoder constructed in step (2-a) is used to encode the new query frame image, and the encoding feature pair of the new query frame image is obtained, and the confidence threshold gating spatiotemporal memory reading module constructed in step (2-b) is used to calculate the comprehensive feature information of the new query frame image; the multi-scale feature extraction of the comprehensive feature information is carried out by using the hollow convolution pyramid module constructed in step (2-c) to obtain the enhanced feature; the decoder constructed in step (2-d) is used to decode the enhanced feature, and the prediction segmentation mask of the new query frame image is obtained; after the prediction segmentation mask of the new query frame image is completed, the query frame image becomes the latest memory frame image, and the next frame image of the query frame image becomes the latest query frame image; thus, the cycle is repeated until the prediction segmentation mask of the last frame image of the video sequence is completed.

[0024] (3) Training the segmentation model:

[0025] The segmentation model constructed in step (2) is trained by using the data set obtained in step (1), cross entropy is used as the loss function of the segmentation model to obtain a loss value, and the ADAM optimizer is used to update the model parameters of the segmentation model until the loss value no longer decreases, and a trained segmentation model is obtained.

[0026] (4) Inference:

[0027] The test video data is obtained and input into the trained segmentation model in step (3), and the segmentation mask of each frame image of the test video data is obtained.

[0028] (5) Video target segmentation:

[0029] The segmentation results of each frame of the test video sequence in the inference stage are obtained, the original image and the segmentation mask are weighted and mixed, and a video stream is made to obtain the video target segmentation result.

[0030] In step (2-a), the constant T is preferably 150.

[0031] In step (2-b), the constant N is preferably 50.

[0032] Compared with the prior art, the present application has the following advantages:

[0033] (1) The confidence gating spatiotemporal memory reading module is proposed in the application, and the video target segmentation network added with the module will pay more attention to effective historical spatiotemporal information and reduce the influence of noise in invalid historical spatiotemporal information on segmentation quality. In addition, the module only normalizes the non-zero high-confidence weight, so as to greatly reduce the calculation amount and improve the reading speed, improve the calculation efficiency while ensuring the performance.

[0034] (2) The application proposes a dilated convolution pyramid module, which extracts and integrates multi-scale target feature information, so that the video target segmentation network pays more attention to global feature information and local feature information of the target, further enhances the recognition ability of small targets and similar targets and improves the accuracy of segmentation. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 The semi-supervised video target segmentation method based on the confidence gating spatiotemporal memory network in the embodiment of the application is a flowchart.

[0036] Figure 2 The overall framework and inference flowchart of the video target segmentation model constructed in the embodiment of the application are shown in the figure.

[0037] Figure 3 The confidence gating spatiotemporal memory reading module structure diagram in the segmentation model in the embodiment of the application is shown in the figure.

[0038] Figure 4 The dilated convolution pyramid module structure diagram in the segmentation model in the embodiment of the application is shown in the figure.

[0039] Figure 5 The decoder structure diagram in the segmentation model in the embodiment of the application is shown in the figure.

[0040] Figure 6 The comparison diagram of the segmentation results of the video sequence in the embodiment of the application and the segmentation results of other methods is shown in the figure. DETAILED DESCRIPTION

[0041] The specific embodiments of the application are described below:

[0042] Embodiment 1

[0043] Figure 1 The semi-supervised video target segmentation method based on the confidence gating spatiotemporal memory network constructed in the embodiment of the application is a flowchart, and the specific steps are as follows:

[0044] Step 1, obtain the data set and the segmentation label:

[0045] Obtain the semi-supervised video target segmentation data set and the corresponding segmentation label.

[0046] Step 2, constructing a segmentation model: the model is composed of an encoder, a confidence gated spatiotemporal memory reading module, a dilated convolution pyramid module, and a decoder. Figure 2 The overall framework of the video object segmentation model constructed in the embodiment of the application and the inference flowchart are shown, and the specific construction process includes the following steps:

[0047] (2-a) Constructing an encoder, including a query encoder and a memory encoder, both of which have the same structure and include a convolution module, three residual modules, and a feature embedding module.

[0048] The query encoder only encodes the current query frame image information. The current query frame image with a size of 3x480x864 is sequentially subjected to the convolution module and the three residual modules of the query encoder to obtain the output feature maps x r1 , x r2 , and x r3 of the three residual modules, which have sizes of 256x240x432, 512x120x216, and 1024x60x108, respectively; x r3 After encoding by the feature embedding module, the feature pair (k Q , v Q ) of the current query frame image is obtained, wherein k Q and v Q are the key information and the value information of the query frame image, respectively, and have sizes of 128x60x108 and 512x60x108, respectively.

[0049] The memory encoder encodes the image and the corresponding mask. The input information of the memory encoder has a size of 4x480x864. The previous frame image of the current query frame image is regarded as a memory frame image. The memory frame image and its mask are sequentially subjected to the convolution module and the three residual modules of the memory encoder, and then are encoded by the feature embedding module to obtain the feature pair (k M , v M ) of the memory frame image, wherein k M and v M represent the key information and the value information of the memory frame image, respectively, and have sizes of 128x60x108 and 512x60x108, respectively.

[0050] The encoding results of the memory frame image and the images and masks of the previous frames of the memory frame image jointly constitute a memory encoding library wherein represents the jth feature pair in the memory code library, j≤T, T represents the maximum number of feature pairs in the memory code library, T=150; when the total number of the memory frame image and all the frame images before it is less than or equal to T, then the memory code library H is composed of the memory frame image and the encoding results of the masks of all the frame images before the memory frame image; when the total number of the memory frame image and all the frame images before it is greater than T, then the memory code library H is composed of the memory frame image and the encoding results of the masks of T-1 frame images before the memory frame image.

[0051] (2-b) constructing a confidence-gated spatiotemporal memory reading module, Figure 3 Fig. 2 shows a structural diagram of a confidence-gated spatiotemporal memory reading module constructed in an embodiment of the present application, which respectively calculates the k Q and the confidence between each key information and the value information in the memory code library H j is a positive integer, and j≤T, the symbol represents the dot product operation, and the confidence-gated mechanism selects the first N feature pairs with the strongest relevance to the current query frame image from the memory code library H according to the size of the confidence, N<T, N=50; if the number of existing feature pairs in the memory code library H is less than N, then a feature pair with feature values (0, 0) is supplemented to N; respectively construct the key information set and the value information set corresponding to the N feature pairs , wherein represents the tth key information with the strongest relevance to the current query frame image, represents the tth value information with the strongest relevance to the current query frame image; calculate the weight value using the elements in the key information set Keys, and then weight-sum the elements in the value information set Values using the calculated weight value to obtain v sum , and finally perform feature splicing on v sum and v Q obtained in step (2-a) to obtain comprehensive feature information R with a size of 1024×60×108.

[0052]

[0053]

[0054]

[0055] , wherein W t represents the weight value, and exp(·) represents the exponential operation with the natural constant e as the base.

[0056] (2-c) constructing a dilated convolution pyramid module,Figure 4 Figure 2 shows the structure of the cavity convolution pyramid module constructed in the embodiment of the present application, which includes four different convolution modules in parallel, namely a standard convolution module with a size of 1x1, a 3x3 cavity convolution module with an expansion rate of 6, a 3x3 cavity convolution module with an expansion rate of 12, and a 3x3 cavity convolution module with an expansion rate of 18. The four different convolution modules are used to perform multi-scale feature extraction on the comprehensive feature information R obtained in step (2-b), and four feature maps with a size of 1024x60x108 are obtained. The extracted results are added pixel by pixel to obtain enhanced features J with a size of 1024x60x108.

[0057] (2-d) constructing a decoder, Figure 5 Figure 3 shows the structure of the decoder constructed in the embodiment of the present application. The decoder performs two convolutions and one upsampling on the enhanced features J with a size of 1024x60x108 obtained in step (2-c) to obtain intermediate features f1 with a size of 1x120x216; the x r2 performs two convolutions to obtain intermediate features f1 with a size of 1x120x216 and f1 are added pixel by pixel, and then one upsampling is performed to obtain intermediate features f2 with a size of 1x240x432; the x r1 performs two convolutions to obtain intermediate features f2 with a size of 1x240x432 and f2 are added pixel by pixel, and then one convolution and one upsampling are performed to obtain the prediction segmentation mask Y of the current query frame image, with a size of 1x480x864.

[0058] ​​(2-e) After the prediction segmentation mask of the current query frame image is obtained, the query frame image becomes a new memory frame image, and the next frame image of the query frame image becomes a new query frame image. The memory encoder constructed in step (2-a) is used to encode the new memory frame image, and the encoding result is saved to the memory code library H, and the memory code library is updated. The query encoder constructed in step (2-a) is used to encode the new query frame image, and the encoding feature pair of the new query frame image is obtained. The confidence threshold gating spatiotemporal memory reading module constructed in step (2-b) is used to calculate the comprehensive feature information of the new query frame image. The multi-scale feature extraction of the comprehensive feature information is performed by using the hollow convolution pyramid module constructed in step (2-c) to obtain the enhanced feature. The decoder constructed in step (2-d) is used to decode the enhanced feature, and the prediction segmentation mask of the new query frame image is obtained. After the prediction segmentation mask of the new query frame image is completed, the query frame image becomes the latest memory frame image, and the next frame image of the query frame image becomes the latest query frame image. The cycle is repeated until the prediction segmentation mask of the last frame image of the video sequence is completed.

[0059] Step 3, training the segmentation model:

[0060] The data set obtained in step (1) is used to train the segmentation model constructed in step (2). Cross entropy is used as the loss function of the entire model to obtain the loss value, and the ADAM algorithm is used to update the parameters in the segmentation model. The number of iterations is 240, the initial learning rate is 0.00001, and the learning rate is decayed for the last 133 iterations until the loss value no longer decreases. The trained segmentation model is obtained.

[0061] Step 4, reasoning:

[0062] The test video data is obtained and input into the trained segmentation model in step (3) to obtain the segmentation mask of each frame image of the test video data.

[0063] Step 5, video target segmentation:

[0064] The segmentation results of each frame image of the test video sequence in the reasoning stage are obtained, the original image and the segmentation mask are weighted and mixed to produce a video stream, and the video target segmentation result is obtained.

[0065] Example 2

[0066] Semi-supervised video object segmentation experiments were performed on the public datasets DAVIS-16 (Densely Annotated Video Segmentation 2016) and DAVIS-17 (Densely Annotated Video Segmentation 2017) using the method in Example 1. The DAVIS-16 dataset is a binary classification dataset, with two segmentation categories: foreground and background; the DAVIS-17 dataset is a multi-classification dataset, with multiple categories of foreground. The operating system of this experiment is Linux ubuntu 16.06 version, implemented based on the PyTorch 1.5.0 framework of CUDA 10.0 and cuDNN 7.6.0, and trained and tested using a personal computer equipped with Intel Xeon Bronze 3104 CPU (1.70 GHz) and NVIDIA GeForce RTX 2080Ti (11 GB) hardware.

[0067] In this example, region similarity, contour accuracy, inference speed, and overall evaluation are used as four indicators to compare RGMP, RANet, and FRTM, three advanced algorithm models. Region similarity, contour accuracy, and overall evaluation are used as three indicators to compare five segmentation networks and the method of the present application on the DAVIS-17 test set. The average result of region similarity in all samples is M J The calculation formula is as follows:

[0068]

[0069] In the formula, S represents the video sample segmentation result, G represents the corresponding label of the video sample, and ∪ and ∩ represent the set and intersection operations, respectively.

[0070] The average result of contour accuracy in all samples is M F The calculation formula is as follows:

[0071]

[0072] In the formula, P c represents the precision of the segmentation result and the label contour point, and R c represents the recall of the segmentation result and the label contour point.

[0073] The overall evaluation is the average of region similarity and contour accuracy, and the average result in all samples is M J&F The calculation formula is as follows:

[0074]

[0075] The comparison results are shown in Tables 1 and 2. It can be found that, compared with other methods, the present application can obtain more accurate segmentation results at a faster inference speed, and fully guarantees the balance of inference speed and segmentation accuracy.

[0076] Figure 6 The figure shows the segmentation results of the video sequence of the embodiment of the present application and the segmentation results of other methods. Among them Figure 6 (a) is an example of a video test frame original picture of a test data set, Figure 6 (b) is the segmentation label corresponding to the test frame original picture example, Figure 6 (c)-(f) are the segmentation results of the present application, FRTM, RANet and RGMP respectively. It can be seen that Figure 6 (e) method and Figure 6 (f) method has poor segmentation effect on video sequences with occlusion interaction between multiple target objects, and has misclassification and omission phenomenon, such as the person on the motorcycle in the third row of test examples and the person in the fifth row of test examples; ​ (d) method has rough segmentation of small target objects, such as the gun in the hand of the person in the fourth row of test examples, and the method has splitting phenomenon for part of the objects, such as the racing car in the second row of test examples. It can be seen that, compared with the above methods, the present application has better segmentation effect and can more effectively handle object occlusion, object size too small and other abnormal situations.

[0077] The above-described embodiments are only the preferred embodiments of the present application, and do not limit the scope of the present application. Any changes made according to the shape and principle of the present application should be covered within the protection scope of the present application.

[0078] Table 1

[0079]

[0080] Table 2

[0081]

Claims

1. A semi-supervised video object segmentation method based on confidence gated spatio-temporal memory network, characterized in that The method comprises the following steps: (1) obtaining a data set and a segmentation label: Obtain a semi-supervised video target segmentation data set and a corresponding segmentation label; (2) constructing a segmentation model: the model is composed of an encoder, a confidence gating spatiotemporal memory reading module, a dilated convolution pyramid module and a decoder, and the specific construction process comprises the following steps: (2-a) constructing an encoder, including a query encoder and a memory encoder, both of which have the same structure and comprise a convolution module, three residual modules and a feature embedding module; The query encoder only encodes the query frame image information, and the current query frame image sequentially passes through the convolution module and the three residual modules of the query encoder to obtain the output feature maps x r1 , x r2 , and x r3 corresponding to the three residual modules; and the feature embedding module encodes x r3 to obtain the feature pair (k Q , v Q ) of the current query frame image, wherein k Q and v Q are the key information and the value information of the query frame image, respectively. The memory encoder encodes the image and its corresponding mask. The previous frame of the current query frame is considered the memory frame image. This memory frame image and its mask are sequentially passed through the convolution module and three residual modules of the memory encoder, and then encoded by the feature embedding module to obtain the feature pairs (k) of the memory frame image. M v M ), where k M and v M These represent the key and value information of the memory frame image, respectively. The encoding results of the memory frame image and the images of the previous frames of the memory frame image and the masks thereof jointly constitute a memory encoding library wherein denotes the jth feature pair in the memory encoding library, j≤T, T is a positive integer, and T represents the maximum number of feature pairs in the memory encoding library; when the total number of the memory frame image and the images of all the previous frames thereof is less than or equal to T, the memory encoding library H is jointly constituted by the encoding results of the memory frame image and the images of all the previous frames thereof and the masks thereof; when the total number of the memory frame image and the images of all the previous frames thereof is greater than T, the memory encoding library H is jointly constituted by the encoding results of the memory frame image and the images of T-1 previous frames thereof and the masks thereof. (2-b) Constructing confidence-gated spatio-temporal memory reading module: calculate k Q the confidence between each key information in memory encoding library H j is a positive integer, and j≤T; the confidence-gated mechanism filters out the first N feature pairs with the strongest relevance to the current query frame image from the memory encoding library H according to the size of the confidence, N is a positive integer, N<T; if the number of existing feature pairs in the memory encoding library H is less than N, then a feature pair with feature values (0, 0) is supplemented to N; respectively construct the key information set corresponding to the N feature pairs and the value information set wherein represents the tth key information with the strongest relevance to the current query frame image, represents the tth value information with the strongest relevance to the current query frame image; calculate the weight value using the elements in the key information set Keys, then weight-sum the elements in the value information set Values using the calculated weight value to obtain v sum , finally, concatenate v sum with v Q obtained in step (2-a) to obtain the comprehensive feature information R;​ (2-c) constructing a dilated convolution pyramid module: the module comprises four different convolution modules in parallel, and the four different convolution modules are used to perform multi-scale feature extraction on the comprehensive feature information R obtained in step (2-b), and the extracted results are pixel-by-pixel added to obtain enhanced features J; (2-d) Constructing the decoder: The enhanced feature J obtained in step (2-c) is subjected to two convolutions and one upsampling to obtain the intermediate feature f1; the x obtained in step (2-a) is then processed. r2 Intermediate features are obtained by performing two convolutions. Will The intermediate feature f2 is obtained by adding f1 pixel by pixel and then upsampling it again; the intermediate feature f2 is obtained by adding x obtained from step (2-a) pixel by pixel. r1 Intermediate features are obtained by performing two convolutions. Will The predicted segmentation mask Y of the current query frame image is obtained by adding f2 pixel by pixel and then performing a convolution and an upsampling. (2-e) after obtaining the predicted segmentation mask of the current query frame image, the query frame image becomes a new memory frame image, and the next frame image of the query frame image becomes a new query frame image, the memory encoder constructed in step (2-a) is used to encode the new memory frame image, and the encoding result is saved to the memory encoding library H, and the memory encoding library is updated; the query encoder constructed in step (2-a) is used to encode the new query frame image to obtain the encoding feature pair of the new query frame image, the confidence gating spatiotemporal memory reading module constructed in step (2-b) is used to calculate the comprehensive feature information of the new query frame image; the dilated convolution pyramid module constructed in step (2-c) is used to perform multi-scale feature extraction on the comprehensive feature information to obtain enhanced features; the decoder constructed in step (2-d) is used to decode the enhanced features to obtain the predicted segmentation mask of the new query frame image; after the predicted segmentation mask of the new query frame image is completed, the query frame image becomes the latest memory frame image, and the next frame image of the query frame image becomes the latest query frame image; the cycle is repeated until the predicted segmentation mask of the last frame image of the video sequence is completed; (3) training the segmentation model: The data set obtained in step (1) is used to train the segmentation model constructed in step (2): cross entropy is used as the loss function of the segmentation model to obtain a loss value, and the Adam optimizer is used to update the model parameters of the segmentation model until the loss value no longer decreases, and a trained segmentation model is obtained; (4) reasoning: Obtain test video data and input it into the trained segmentation model in step (3) to obtain the segmentation mask of each frame image of the test video data; (5) video target segmentation: Obtain the segmentation mask of each frame image of the test video data in the reasoning stage, weight the original image and the segmentation mask, mix them to produce a video stream, and obtain the video target segmentation result.

2. The semi-supervised video object segmentation method based on the confidence-gated spatio-temporal memory network according to claim 1, wherein, The confidence in step (2-b) is calculated according to the following formula: wherein, denotes a dot product operation. 3.The method of claim 1, wherein, The confidence gating mechanism weight value in step (2-b) is calculated and weighted according to the following formulas: where W t denotes a weight, and exp(·) denotes an exponential operation with base of the natural constant e. 4.The method of claim 1, wherein, The cavity convolution pyramid module in the step (2-c) comprises four different convolution modules in parallel, which are a 1*1 size standard convolution module, a 3*3 cavity convolution module with an expansion rate of 6, a 3*3 cavity convolution block with an expansion rate of 12 and a 3*3 cavity convolution module with an expansion rate of 18.

5. The semi-supervised video object segmentation method based on the confidence-gated spatio-temporal memory network according to claim 1, wherein, The decoder in the step (2-d) is a decoder with a convolution kernel size of 3*3 in the convolution layer.

Images