Frame supervision video instance segmentation system based on time sequence feature fusion

Abstract

The invention discloses a frame supervision video instance segmentation system based on time sequence feature fusion, and the system comprises a feature extraction network which is used for extracting features # imgabs0 # and # imgabs1 # which cover time-space similarity information from a current frame picture and an adjacent frame picture of a video, t represents a current moment, and delta represents a time interval between two adjacent frame pictures; and the space-time prediction self-attention network is used for generating a memory feature Mt + delta according to the features # imgabs2 # and # imgabs3 #, and performing weighted fusion on the memory feature Mt + delta and the feature # imgabs4 # to obtain a fusion feature # imgabs5 # instance mask prediction network, and the space-time prediction self-attention network is used for performing instance mask prediction according to the feature # imgabs6 # and the fusion feature # imgabs7 #. An IFIM module is introduced into a feature extraction network, and a channel screening module and a space-time prediction self-attention module are combined, so that the video instance segmentation precision is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video instance segmentation, and in particular relates to a frame-supervised video instance segmentation system with temporal feature fusion. Background Art

[0002] Video instance segmentation based on deep learning is an emerging computer vision task in recent years. Its task content is to extend the image instance segmentation algorithm to the video field, and obtain the segmentation mask and detection results of each instance in the form of a time series by cross-frame association and tracking of the same instance in different frames of the entire video. Video instance segmentation methods are divided into two modes: full supervision and border-level supervision according to the different training data annotation methods and model training methods. Among them, fully supervised video instance segmentation requires pixel-level annotation of target instances on video frames to obtain accurate mask segmentation results. Its lengthy video sequence samples will cause a sharp increase in the cost of manual fine annotation. Border-supervised video instance segmentation only requires the target instance bounding box coordinates and category information for segmentation model supervision, which significantly reduces the complexity of manual annotation. It is a key core technology for intelligent assisted driving vehicles to achieve pixel-level perception of multiple target instances in complex dynamic scenes and quickly adapt to different task scenarios.

[0003] Bounding box supervised video instance segmentation is an emerging technology in the frontier field of computer vision image processing. It only requires instance bounding box coordinates and category information as true values ​​for network constraint learning to achieve pixel-level classification, segmentation and tracking, which greatly saves the cost of manual annotation and has broad application prospects. The essence of bounding box supervised video instance segmentation is a challenging extension of bounding box supervised image instance segmentation on video objects. Current bounding box supervised image instance segmentation algorithms usually use prior predictors such as GrabCut and MCG to generate class-agnostic target masks, or generate high-quality pseudo masks based on pixel association information based on methods such as pairwise affinity to achieve training supervision. Yan et al. proposed the STC-Seg algorithm, which combines the bounding box-level positioning information of the target instance and uses video depth estimation and optical flow field complementarity to generate high-quality pseudo labels to extract video spatiotemporal information, and first realizes bounding box supervised video instance segmentation. MaskFreeVIS (Mask-Free Video Instance Segmentation) proposed by Ke et al. introduces temporal TK-Loss based on Mask2Former (mask2former is an end-to-end detection and segmentation framework based on Transformer). It establishes the correspondence between adjacent frames of the video through frame-by-frame matching, and effectively identifies and tracks the continuity of the same target instance between different frames by using the rich temporal mask consistency constraints in the video, significantly improving the performance of bounding box supervised video instance segmentation. BoxVIS (Box Video Instance Segmentation, reference paper: Li M, Zhang L. BoxVIS: Video Instance Segmentation with Box Annotations. arXiv preprint arXiv:2303.14618, 2023; it uses bounding box annotations instead of mask annotations to train the model) proposed by Li et al. designs a STPA loss guided by the center of the bounding box, uses the bounding box center to guide the shift to generate temporal paired pixels, and uses feature space color similarity and patch correlation to calculate the pairwise affinity of adjacent frame pixels, effectively improving the average segmentation accuracy of the bounding box supervised video instance model.

[0004] Although the above models have improved the accuracy of video instance segmentation to a certain extent, they still face some unique challenges and technical problems due to the particularity of video instance segmentation:

[0005] 1. Object occlusion and disappearance: In a video, an object may be occluded by other objects or disappear from the video (for example, entering or leaving the field of view), which makes accurate tracking and segmentation difficult. The model needs to be able to handle occlusion and partial disappearance, and correctly identify and associate the object when it reappears.

[0006] 2. Dynamic background changes: The background in the video is often dynamic and may move or change, which makes the segmentation task more challenging. The model needs to distinguish which are changes in the background and which are the movement of the foreground object.

[0007] 3. Changes in the appearance of objects: Over time, the appearance of objects may change due to factors such as lighting, angle, deformation, etc. The model must be able to handle such changes to ensure correct identification and segmentation of objects.

[0008] It is precisely because of the above reasons that improving the accuracy of video instance segmentation has become a research hotspot in this field. Summary of the invention

[0009] In order to improve the segmentation accuracy, the present invention provides a frame-supervised video instance segmentation system with temporal feature fusion.

[0010] The technical solution adopted by the present invention is:

[0011] The present application discloses a temporal feature fusion frame supervision video instance segmentation system, comprising:

[0012] Feature extraction network, used to extract features covering spatiotemporal similarity information from the current frame and adjacent frames of the video and Among them, t represents the current time, δ represents the time interval between two adjacent frames;

[0013] Spatiotemporal prediction self-attention network is used to and Generate memory feature M t+δ , and the memory feature M t+δ With features Weighted fusion to obtain fusion features

[0014] Instance mask prediction network, used to predict the and fusion features Make instance mask prediction.

[0015] As an optional technical solution, the backbone network of the feature extraction network adopts the residual network ResNet50, which includes 4 block layers, each of which is followed by an IFIM module, the input of each IFIM module is the output of the previous block layer, and the output of each IFIM module is {(F 11 ,F 21 ),(F 12 ,F 22 ),(F 13 ,F23 ),(F 14 ,F 24 )}; For the first-level IFIM module, it includes: an interaction operation unit, an aggregation operation unit, an update operation unit and a fusion operation unit.

[0016] The interactive operation unit is used to convert the current frame feature F t With adjacent frame features F t+δ The feature F containing the video timing information is obtained by splicing, and the feature F is reshaped into a feature matrix S, and then the feature matrix S and its transposed matrix S are combined. T After multiplication, the spatial information weight is normalized to obtain the similarity weight matrix D; among which, the current frame feature F t It is the feature extracted by the first block layer through convolution operation on the current frame, and the adjacent frame feature F t+δ It is the feature extracted by the first-level block layer through convolution operation on adjacent frames.

[0017] The aggregation operation unit is used to multiply the obtained similarity weight matrix D with the feature matrix S and reshape it to obtain the aggregated feature map Z1.

[0018] The updating operation unit is used to obtain a weight mapping relationship matrix Q using the aggregate feature map Z1, and to multiply the weight mapping relationship matrix Q with the feature F containing the video timing information to obtain a timing feature map E.

[0019] The fusion operation unit is used to split the time series feature map E in the time dimension to obtain new feature maps F1 and F2, and to combine the feature maps F1 and F2 with the current frame feature map F1 and F2 respectively. t and adjacent frame features F t+δ Perform weighted fusion to obtain the feature map F covering the spatiotemporal similarity information 11 and F 21 .

[0020] The structures of the other IFIM modules are the same as those of the first IFIM module. The output characteristic graph of the first IFIM module is {F 11 ,F 21}, the second, third, and fourth level IFIM modules output feature maps {F 12 ,F 22}, {F 13 ,F 23}, {F 14 ,F 24}, the characteristics

[0021] As an optional technical solution, the interactive operation unit is further used to: t With adjacent frame features Ft+δ The feature F containing the video timing information is obtained by splicing. The calculation process is as follows:

[0022] F=stack(F t ,F t+δ );

[0023] The stack(·) function represents a concatenation operation in a new dimension; the feature F is reshaped into a matrix S as follows:

[0024] S = Reshape(F);

[0025] The Reshape(·) function represents the feature reshaping operation; the matrix S and its transposed matrix S T Multiply the semantic similarity information of the previous and next frames of the video. Finally, use the Softmax activation function to normalize the spatial information weights to obtain the similarity weight matrix D, as follows:

[0026]

[0027] in, Represents element-wise multiplication.

[0028] As an optional technical solution, the characteristic map Z1 is expressed as:

[0029]

[0030] As an optional technical solution, the weight mapping relationship matrix Q is expressed as:

[0031] Q = GAP(σ1(BN(Conv 1×1 (Z1))))

[0032] Among them, GAP(·) represents the global average pooling operation, σ1(·) represents the ReLU activation function, BN(·) represents the batch normalization operation, Conv k×k (·) represents a convolution operation with a kernel size of k×k.

[0033] As an optional technical solution, the feature map F 11 and F 21 Respectively expressed as:

[0034]

[0035] F1, F2 = split(E);

[0036] Among them, split(·) means splitting the five-dimensional time series feature into two four-dimensional features. Represents element-by-element addition.

[0037] As an optional technical solution, the spatiotemporal prediction self-attention network includes an initialization module, a first spatiotemporal prediction self-attention module and a second spatiotemporal prediction self-attention module, wherein the output of the first spatiotemporal prediction self-attention module serves as the input of the second spatiotemporal prediction self-attention module; the first spatiotemporal prediction self-attention module includes a spatial feature extraction unit and a SAM unit.

[0038] Among them, the initialization module is used to initialize the feature M t-δ , M t-δ represents the memory characteristics at time t-δ; represents the hidden state of the first spatiotemporal prediction self-attention module at time t-δ; C t-δ represents the original storage memory unit at time t-δ; The feature set with a batch size of 4, a channel number of 128, a height of 48, and a width of 80 is represented. The spatial feature extraction unit is used to extract the feature according to the initialization feature. C t-δ and the features input by the feature extraction network Extract the feature H at the current time t t and C t The SAM unit is used to calculate the t and initialization feature M t-δ Generate the feature M at the current time t t and

[0039] As an optional technical solution, the feature C t The calculation is as follows:

[0040] C t =ft t ⊙C t-δ +i t ⊙g t

[0041] Among them, ⊙ represents the matrix dot product, f t 、i t , g t They are:

[0042]

[0043] Among them, σ2(·) represents the Sigmoid activation function, tanh(·) represents the tanh activation function; f t represents the first forget gate, W hf is the weight of the first forget gate, b f is the bias of the first forget gate; i t represents the first input gate, w hi is the weight of the first input gate, b i represents the bias of the first input gate; g t represents the memory cell state of the first spatiotemporal prediction self-attention module, W hC is the weight of the memory cell state, b c is the bias of the memory cell state.

[0044] As an optional technical solution, the feature H t The calculation process is as follows:

[0045] H t =O t ⊙tanh(C t )

[0046]

[0047] Among them, O t is the first output gate, W ho is the weight of the first output gate, b o is the bias of the first output gate.

[0048] As an optional technical solution, the feature M t The calculation process is as follows:

[0049] M t =(1-i t ′)⊙M t-δ +i t ′⊙g t '

[0050] Among them, i t ′, g t ′ respectively represent as follows:

[0051] i t ′=σ2(W m；zi ×Z+W m；hi ×H t +b m；i )

[0052] g t ′=tanh(W m；zg ×Z+W m；hg ×H t +b m；g )

[0053] Among them, i t ' is the second input gate, W m；zi , W m；hi is the weight of the second input gate, bm；i is the bias of the second input gate; g t ′ represents the memory cell state of the SAM unit, W m；zg , W m；hg is the weight of the memory cell state, n m；g is the bias of the memory cell state.

[0054] As an optional technical solution, the feature The calculation is as follows:

[0055]

[0056] Among them, t ' is the second output gate, and the expression is as follows:

[0057] o t ′=σ2(W m；zo ×Z+W m；ho ×H t +b m；o )

[0058] Among them, W m；zo 、E m；ho is the weight of the second output gate, b m；o is the bias for the second output gate.

[0059] As an optional technical solution, the second spatiotemporal prediction self-attention module has the same structure as the first spatiotemporal prediction self-attention module; the second spatiotemporal prediction self-attention module is used to generate a feature C output by the first spatiotemporal prediction self-attention module. t , M t And the features extracted by the feature extraction network Generate memory feature M t+δ .

[0060] As an optional technical solution, the instance mask prediction network includes:

[0061] The first CSM module and the first mask branch use features Perform instance mask prediction and perform weighted fusion of the respective predicted instance masks;

[0062] The second CSM module and the second mask branch use features Perform instance mask prediction and perform weighted fusion of the respective predicted instance masks;

[0063] The second CSM module has the same structure as the first CSM module, and the second mask branch has the same structure as the first mask branch.

[0064] As an optional technical solution, the first CSM module includes an optimization unit, a channel screening unit and a splicing convolution unit.

[0065] The optimization unit is used to perform batch normalization on the feature M output by the feature extraction network and activate it through the ReLU function to obtain the optimized feature map M′. The calculation formula is as follows:

[0066] M′=σ1(BN(M))

[0067] Among them, M∈{F 11 ,F 12 ,F 13}, BN(·) represents batch normalization operation, and σ1(·) represents ReLU activation function. The channel screening unit takes the optimized feature map M′ as input and is used to perform channel screening to extract effective channel information G p , p=12,3,4,G p represents the effective channel information extracted by the pth feature selector. The concatenated convolution unit is used to p Perform concatenation and convolution operations to extract effective feature information U.

[0068] As an optional technical solution, the effective channel information G p The calculation process is as follows:

[0069] G p =fc 8 / 128 (fc 128 / 8 (G)

[0070] Among them, fc 128 / 8 represents the first fully connected layer, whose input dimension is 128 and output dimension is 8; fc 8 / 128 represents the second fully connected layer, whose input dimension is 8 and output dimension is 128; G is the global average pooling result, which is expressed as follows:

[0071]

[0072] As an optional technical solution, the effective feature information U is expressed as follows:

[0073]

[0074] in, It represents a convolution operation with a convolution kernel of k×k, n input channels, and m output channels. Weight represents the updated channel weight information. Y is the concatenation feature, which is expressed as follows:

[0075] Y = cat(N1, N2, N3, N4);

[0076]

[0077] As an optional technical solution, the first CSM module further includes a regularization loss unit for regularizing the effective channel information G p Take the mean and add it to the loss term as a regularization result for optimization. The regularized loss function is expressed as:

[0078] L loss,p =mean(G p )

[0079] Among them, mean(·) represents the mean operation.

[0080] The beneficial effects of the present invention are as follows: in this application, by introducing the IFIM module into the feature extraction network based on the residual network ResNet50, and combining the channel screening module and the spatiotemporal prediction self-attention module, the video instance segmentation accuracy is effectively improved; the experimental results of the traffic scene dataset extracted based on Youtube-VIS2019 show that the average segmentation accuracy of this application is improved by 6.2% to 39.2% compared with the border supervision benchmark network, which provides an effective algorithmic basis for realizing multi-target perception and instance-level scene understanding. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 It is an architecture diagram of a bounding box supervised video instance segmentation system with temporal feature fusion in an exemplary embodiment.

[0082] Figure 2 is an architecture diagram of a feature extraction network in an exemplary embodiment.

[0083] Figure 3 is a schematic diagram of an IFIM module in an exemplary embodiment.

[0084] Figure 4 2 is an architecture diagram of a spatiotemporal prediction self-attention network in an exemplary embodiment.

[0085] Figure 5 is an architecture diagram of an example mask prediction network in an exemplary embodiment.

[0086] Figure 6 This is a specific implementation architecture diagram of a spatiotemporal prediction self-attention network in an exemplary embodiment.

[0087] Figure 7 is a schematic diagram of a SAM unit in an exemplary embodiment.

[0088] Figure 8 It is an implementation architecture diagram of a channel screening module in an exemplary embodiment.

[0089] Fig. 9This is a general architecture diagram for performing video instance recognition and prediction using the video instance segmentation system of this embodiment. DETAILED DESCRIPTION

[0090] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components and method steps of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0091] Example

[0092] like Figure 1 As shown, this embodiment discloses a border-supervised video instance segmentation system with temporal feature fusion, including:

[0093] Feature extraction network 1, used to extract features covering spatiotemporal similarity information from the current frame and adjacent frames of the video and Among them, t represents the current time, δ represents the time interval between two adjacent frames;

[0094] Spatiotemporal prediction self-attention network 2 is used to and Generate memory feature M t+δ , and the memory feature M t+δ With features Weighted fusion to obtain fusion features

[0095] Instance mask prediction network 3 is used to predict the and fusion features Make instance mask prediction.

[0096] like Figure 2 As shown, as an optional implementation, the backbone network of the feature extraction network 1 adopts the residual network ResNet50, and the residual network ResNet50 includes 4 block layers, each of which is followed by an IFIM module, and the input of each IFIM module is the output of the previous block layer, and the output of each IFIM module is {(F 11 ,F 21 ),(F12 ,F 22 ),(F 13 ,F 23 ),(F 14 ,F 24 )};like Figure 3 As shown, the first-level IFIM module includes: an interaction operation unit, an aggregation operation unit, an update operation unit and a fusion operation unit.

[0097] The interactive operation unit is used to convert the current frame feature F t With adjacent frame features F t+δ The feature F containing the video timing information is obtained by splicing, and the feature F is reshaped into a feature matrix S, and then the feature matrix S and its transposed matrix S are combined. T After multiplication, the spatial information weight is normalized to obtain the similarity weight matrix D; among which, the current frame feature F t It is the feature extracted by the first block layer through convolution operation on the current frame, and the adjacent frame feature F t+δ It is the feature extracted by the first-level block layer through convolution operation on adjacent frames.

[0098] The aggregation operation unit is used to multiply the obtained similarity weight matrix D with the feature matrix S and reshape it to obtain the aggregated feature map Z1.

[0099] The updating operation unit is used to obtain a weight mapping relationship matrix Q using the aggregate feature map Z1, and to multiply the weight mapping relationship matrix Q with the feature F containing the video timing information to obtain a timing feature map E.

[0100] The fusion operation unit is used to split the time series feature map E in the time dimension to obtain new feature maps F1 and F2, and to combine the feature maps F1 and F2 with the current frame feature map F1 and F2 respectively. t and adjacent frame features F t+δ Perform weighted fusion to obtain the feature map F covering the spatiotemporal similarity information 11 and F 21 .

[0101] The structures of the other IFIM modules are the same as those of the first IFIM module. The output characteristic graph of the first IFIM module is {F 11 ,F 21}, the second, third, and fourth level IFIM modules output feature maps {F 12 ,F 22}, {F 13 ,F 23}, {F 14 ,F 24}, the outputs of the first, second and third level IFIM modules are used as the inputs of the second, third and fourth level block layers respectively.

[0102] As an optional implementation, the interactive operation unit is further used to: t With adjacent frame features F t+δ The feature F containing the video timing information is obtained by splicing. The calculation process is as follows:

[0103] F=stack(F t ,F t+δ );

[0104] The stack(·) function represents a concatenation operation in a new dimension; the feature F is reshaped into a matrix S as follows:

[0105] S = Reshape(F);

[0106] The Reshape(·) function represents the feature reshaping operation; the matrix S and its transposed matrix S T Multiply the semantic similarity information of the previous and next frames of the video. Finally, use the Softmax activation function to normalize the spatial information weights to obtain the similarity weight matrix D, as follows:

[0107]

[0108] in, Represents element-wise multiplication.

[0109] As an optional implementation, the characteristic graph Z1 is expressed as:

[0110]

[0111] As an optional implementation, the weight mapping relationship matrix Q is expressed as:

[0112] Q = GAP(σ1(BN(Conv 1×1 (Z1))))

[0113] Among them, GAP(·) represents the global average pooling operation, σ1(·) represents the ReLU activation function, BN(·) represents the batch normalization operation, Conv k×k (·) represents a convolution operation with a kernel size of k×k.

[0114] As an optional implementation, the feature map F 11 and F 21 Respectively expressed as:

[0115]

[0116] F1, F2 = split(E);

[0117] Among them, split(·) means splitting the five-dimensional time series feature into two four-dimensional features. Represents element-by-element addition.

[0118] As an optional implementation, Figure 4 As shown, the spatiotemporal prediction self-attention network includes an initialization module, a first spatiotemporal prediction self-attention module and a second spatiotemporal prediction self-attention module, wherein the output of the first spatiotemporal prediction self-attention module serves as the input of the second spatiotemporal prediction self-attention module; the first spatiotemporal prediction self-attention module includes a spatial feature extraction unit and a SAM unit.

[0119] Among them, the initialization module is used to initialize the feature M t-δ , M t-δ represents the memory characteristics at time t-δ; represents the hidden state in the first spatiotemporal prediction self-attention module at time t-δ; C t-δ represents the original storage memory unit at time t-δ; The feature set with a batch size of 4, a channel number of 128, a height of 48, and a width of 80 is represented. The spatial feature extraction unit is used to extract the feature according to the initialization feature. C t-δ and the features input by the feature extraction network Extract the feature H at the current time t t and C t The SAM unit is used to calculate the t and initialization feature M t-δ Generate the feature M at the current time t t and

[0120] As an optional technical solution, the feature C t The calculation is as follows:

[0121] C t =f t ⊙C t-δ +i t ⊙g t

[0122] Among them, ⊙ represents the matrix dot product, f t 、i t , g t They are:

[0123]

[0124] Among them, σ2(·) represents the Sigmoid activation function, tanh(·) represents the tanh activation function; f t represents the first forget gate, W hf is the weight of the first forget gate, b f is the bias of the first forget gate; i t represents the first input gate, w hi is the weight of the first input gate, b i represents the bias of the first input gate; g t represents the memory cell state of the first spatiotemporal prediction self-attention module, W hC is the weight of the memory cell state, b c is the bias of the memory cell state.

[0125] As an optional implementation, the feature H t The calculation process is as follows:

[0126] H t =O t ⊙tanh(C t )

[0127]

[0128] Among them, O t is the first output gate, W ho is the weight of the first output gate, b o is the bias of the first output gate.

[0129] As an optional implementation, the feature M t The calculation process is as follows:

[0130] M t =(1-i t ′)⊙M t-δ +i t ′⊙g t '

[0131] Among them, i t ′, g t ′ respectively represent as follows:

[0132] i t ′=σ2(W m；zi ×Z+W m；hi ×H t +b m；i )

[0133] g t ′=tanh(Wm；zg ×Z+W m；hg ×H t +b m；g )

[0134] Among them, i t ' is the second input gate, W m；zi , W m；hi is the weight of the second input gate, b m；i is the bias of the second input gate; g t ′ represents the memory cell state of the SAM unit, W m；zg , W m；hg is the weight of the memory cell state, b m；g is the bias of the memory cell state.

[0135] As an optional embodiment, the feature The calculation is as follows:

[0136]

[0137] Among them, t ' is the second output gate, and the expression is as follows:

[0138] o t ′=σ2(W m；zo ×Z+W m；ho ×H t +b m；o )

[0139] Among them, W m；zo , W m；ho is the weight of the second output gate, b m；o is the bias for the second output gate.

[0140] As an optional implementation, the second spatiotemporal prediction self-attention module has the same structure as the first spatiotemporal prediction self-attention module; the second spatiotemporal prediction self-attention module is used to generate a feature C output by the first spatiotemporal prediction self-attention module. t , M t And the features extracted by the feature extraction network Generate memory feature M t+δ .

[0141] As an optional implementation, Figure 5 As shown, the instance mask prediction network includes:

[0142] The first CSM module and the first mask branch use features Perform instance mask prediction and perform weighted fusion of the respective predicted instance masks;

[0143] The second CSM module and the second mask branch use features Perform instance mask prediction and perform weighted fusion of the respective predicted instance masks;

[0144] The second CSM module has the same structure as the first CSM module, and the second mask branch has the same structure as the first mask branch.

[0145] As an optional implementation, the first CSM module includes an optimization unit, a channel screening unit and a splicing convolution unit.

[0146] The optimization unit is used to perform batch normalization on the feature M output by the feature extraction network and activate it through the ReLU function to obtain the optimized feature map M′. The calculation formula is as follows:

[0147] M′=σ1(BN(M))

[0148] Among them, M∈{F 11 ,F 12 ,F 13}, BN(·) represents batch normalization operation, and σ1(·) represents ReLU activation function. The channel screening unit takes the optimized feature map M′ as input and is used to perform channel screening to extract effective channel information G p , p=12,3,4,G p represents the effective channel information extracted by the pth feature selector. The concatenated convolution unit is used to p Perform concatenation and convolution operations to extract effective feature information U.

[0149] As an optional implementation, the effective channel information G p The calculation process is as follows:

[0150] G p =fc 8 / 128 (fc 128 / 8 (G)

[0151] Among them, fc 128 / 8 represents the first fully connected layer, whose input dimension is 128 and output dimension is 8; fc 8 / 128 represents the second fully connected layer, whose input dimension is 8 and output dimension is 128; G is the global average pooling result, which is expressed as follows:

[0152]

[0153] As an optional implementation manner, the valid characteristic information U is expressed as follows:

[0154]

[0155] in, It represents a convolution operation with a convolution kernel of k×k, n input channels, and m output channels. Weight represents the updated channel weight information. Y is the concatenation feature, which is expressed as follows:

[0156] Y = cat(N1, N2, N3, N4);

[0157]

[0158] As an optional implementation, the first CSM module further includes a regularization loss unit for regularizing the effective channel information G p Take the mean and add it to the loss term as a regularization result for optimization. The regularized loss function is expressed as:

[0159] L loss,p =mean(G p )

[0160] Among them, mean(·) represents the mean operation.

[0161] In order to better understand the present application, each module in this embodiment is further described below.

[0162] 1. Inter-frame feature integration module (IFIM).

[0163] In this embodiment, an interframe feature integration module (IFIM) is embedded after the Block layer in the backbone network ResNet50, which helps to improve the accuracy and stability of model segmentation by capturing the temporal consistent correlation information between the current frame and the adjacent frames and realizing the selective aggregation of interframe temporal features according to their correlation strength. The backbone network (i.e., the feature extraction network mentioned above) after the IFIM module is embedded is as follows: Figure 2 shown.

[0164] The inter-frame feature integration module structure mainly includes four parts: interaction operation unit, aggregation operation unit, update operation unit, and fusion operation unit. Figure 3 The specific operation process is as follows:

[0165] Interaction unit: First, the current frame feature F after the Block layer in the backbone network is t and adjacent frame features F t+δ The feature F containing the video timing information is obtained by splicing along the third dimension, where: in represents a real number set, B represents the number of batches, C represents the number of channels, H represents height, W represents width, and T represents the third dimension. The concatenation obtains features containing video timing information, and then reshapes F into a matrix And its transposed matrix Multiply the previous and next frames to get the semantic similarity information of the video. Finally, use the Softmax activation function to normalize the spatial information weights to get the similarity weight matrix The calculation formula for interactive operation can be expressed as follows:

[0166] S=Reshape(F)

[0167]

[0168] Among them, Reshape(·) represents the feature reshaping operation, stack(·) represents the splicing operation in the new dimension, Represents element-wise multiplication.

[0169] Aggregation operation unit: Multiply the similarity weight matrix D and the feature map S to strengthen the feature information with high similarity between video frames, while weakening the irrelevant or low-similarity feature information to improve the expressiveness of the feature. Finally, reshape the feature to obtain the aggregated feature map The calculation formula for the aggregation operation is as follows:

[0170]

[0171] Update operation unit: First, the aggregated features are linearly combined through point-by-point convolution operations to promote the flow of channel information and enhance the correlation between different channels. Secondly, batch normalization (BN) and nonlinear ReLU activation function are introduced to transform each channel feature, suppress the expression of irrelevant information, and improve the generalization performance of the model. Then, the feature map after channel update is globally averaged pooled to extract global context information and establish the interdependence and weight mapping relationship between different channels. Finally, the obtained weight mapping relationship Q is multiplied by the feature map F containing the video timing information to obtain the feature map The update operation calculation formula is as follows:

[0172] Q = GAP(σ1(BN(Conv 1×1 (Z1))))

[0173]

[0174] Among them, GAP(·) represents the global average pooling operation, σ1(·) represents the ReLU activation function, BN(·) represents the batch normalization operation, Conv k×k(·) represents a convolution operation with a kernel size of k×k.

[0175] Fusion operation unit: Split the obtained time series feature map E in the time dimension to obtain a new feature map and In order to prevent the network performance from degrading, a residual structure is used to convert the input feature F t and F t+δ The feature map F covering the spatiotemporal similarity information is obtained by weighted fusion with F1 and F2 respectively. 11 and F 21 The calculation formula for the fusion operation is as follows:

[0176] F1, F2 = split(E)

[0177]

[0178] Among them, split(·) means splitting the five-dimensional time series feature into two four-dimensional features. Represents element-by-element addition.

[0179] 2. Spatiotemporal prediction self-attention network.

[0180] In order to reduce the deviation of instance feature transmission between frames and maintain the consistency and continuity of video object instances, this embodiment introduces a spatiotemporal prediction self-attention module to improve the encoding performance of the segmentation network for complex sequence dynamic spatiotemporal relationships. The overall architecture of the spatiotemporal prediction self-attention network is as follows: Figure 4 , including two spatiotemporal prediction self-attention modules (i.e., the first spatiotemporal prediction self-attention module and the second spatiotemporal prediction self-attention module), and its specific implementation method can adopt the SA-ConvLSTM (Self-Attention-ConvLSTM) model, such as Figure 6 shown.

[0181] First of all, Initialize to zero and compare with the feature At the same time, the SA-ConvLSTM module is input to extract spatial features with global and local dependencies. The specific calculation formula is as follows:

[0182]

[0183] C t =f t ⊙C t-δ +i t ⊙g t

[0184]

[0185] H t =O t⊙tanh(C t )

[0186] Among them, σ2(·) represents the Sigmoid activation function, tanh(·) represents the tanh activation function; f t represents the first forget gate, W hf is the weight of the first forget gate, b f is the bias of the first forget gate; i t represents the first input gate, W hi is the weight of the first input gate, b i represents the bias of the first input gate; g t represents the memory cell state of the first spatiotemporal prediction self-attention module, W hC is the weight of the memory cell state, b c is the bias of the memory cell state.

[0187] Secondly, considering that the relevant features of the past moments in the video sequence play an important role in promoting the instance prediction of the current moment, such as Figure 7 As shown in the figure, the SAM unit introduces a memory unit based on the self-attention mechanism to represent the global context information. t and the memory feature M of the previous moment t-δ The SAM module aggregates features to improve the prediction performance of the module at the current moment. Specifically, the SAM module defines multiple convolution kernels to aggregate H t With M t-δ Mapping to different feature spaces for weighted and similarity calculations to obtain features and The calculated result Z h and Z m After concatenation in the channel dimension, a convolution operation is performed and then combined with the original feature H t Merge channels to get the final aggregate features The calculation process is expressed as follows:

[0188]

[0189] Among them, SA(·) represents the self-attention mechanism; cat(·) represents concatenation in the channel dimension; W Z (·) represents the convolution operation.

[0190] In addition, the SAM module uses a gating mechanism in the memory update phase to combine the aggregated feature Z to update the memory feature M at the previous moment. t-δ Update to the current feature M t , the calculation formula of the memory update phase is as follows:

[0191] i t ′=σ2(W m；zi ×Z+W m；hi ×H t +b m；i )

[0192] g t ′=tanh(W m；zg ×Z+W m；hg ×H t +b m；g )

[0193] M t =(1-i t ′)⊙M t-δ +i t ′⊙g t '

[0194] Among them, σ2 represents the Sigmoid activation function; i t ' is the second input gate, W m；zi , W m；hi is the weight of the second input gate, b m；i is the bias of the second input gate; g t ′ represents the memory cell state of the SAM unit, W m；zg , W m；hg is the weight of the memory cell state, b m；g is the bias of the memory cell state.

[0195] Then, the second output gate and the updated memory feature M t The product between them is used as the output feature of the self-attention memory module The specific calculation formula for the output stage is as follows:

[0196] o t ′=σ2(W m；zo ×Z+W m；ho ×H t +b m；o )

[0197]

[0198] Among them, W m；zo , W m；ho is the weight of the second output gate, b m；o is the bias for the second output gate.

[0199] Finally, the global and local dependency space features C extracted at the current moment are t , M t and the characteristics at time t+δ Input the second spatiotemporal prediction self-attention module, and output the memory feature M of the second spatiotemporal prediction self-attention module t+δ and Weighted fusion is performed to enhance the network's ability to focus on target temporal information.

[0200] 3. Channel Screening Module (CSM).

[0201] The CrossVIS network fuses the multi-scale feature maps output from the feature extraction stage, and passes the fused features to the Mask Branch for mask prediction. After a series of 3×3 convolution operations, the Mask Branch finally generates a prototype instance mask with 8 channels. However, due to the different importance of information between different feature map channels to the segmentation task, the last 3×3 convolution operation of the Mask Branch branch directly reduces the number of feature map channels from 128 to 8, which may cause the loss of instance edge texture detail information and weakened representation ability in some feature spaces. Therefore, in response to this problem, the present embodiment designs a channel screening module (i.e., the first CSM module and the second CSM module mentioned above) to perform instance mask prediction on the fused feature map in parallel with the Mask Branch branch, and weightedly fuses the instance masks predicted by the CSM module and the MaskBranch branch to effectively make up for their respective shortcomings in instance prediction. The CSM module adaptively obtains the importance of feature channel information to the segmentation task, and selects effective feature channels accordingly to make up for the problem of partial semantic information loss and weakened feature representation ability caused by the reduction in the number of feature channels of the mask prediction branch, thereby improving the accuracy of the model in target positioning and edge contour segmentation.

[0202] An implementation framework of the channel screening module is as follows Figure 8 As shown, first, the feature map output from the feature extraction stage is Use batch normalization and activate through the ReLU function to enhance the expressiveness of features, accelerate network convergence, and finally obtain the optimized feature map The calculation formula is as follows:

[0203] M′=σ1(BN(M))

[0204] Among them, M∈{F 11 ,F 12 ,F 13}or BN(·) denotes the batch normalization operation, and σ1(·) denotes the ReLU activation function.

[0205] Secondly, the optimized feature map M′ is input into the feature selector for channel screening to select channel information that is highly relevant to the segmentation task. Specifically, M′ is first subjected to a global average pooling operation to extract global context information and obtain the global distribution and weight information of the feature channels. The global average pooling is calculated as follows:

[0206]

[0207] Then, the fully connected layer fc is used 128 / 8 and fc 8 / 128 Complete the cross-channel information interaction and information integration of features, complete the update of feature weights, and extract effective channel information. The specific operation is as follows:

[0208] G p =fc 8 / 128 (fc 128 / 8 (G)

[0209] Among them, fc 128 / 8 Represents the first fully connected layer (input dimension is 128, output dimension is 8). 8 / 128 represents the second fully connected layer (input dimension is 8, output dimension is 128). Then take the mean of the obtained channel information, and add the mean as the regularization result to the loss term for optimization, so that the model learns smaller parameter values, thereby achieving the regularization effect. The calculation formula is as follows:

[0210] L loss,p =mean(G p )

[0211] Among them, mean(·) represents the mean operation; p represents the p-th feature selector.

[0212] During the model training process, the network gradient back propagation is used to continuously learn, automatically adjust and update to adapt to the current segmentation task, and finally obtain weight = w1, w2, w3, ..., w c-1 ,w c . Output feature G of the fully connected layer p Multiplying with the original feature M′ helps the network learn the importance or weight of different features during training and select effective feature channels. The calculation formula is as follows:

[0213]

[0214] in, Represents element-wise multiplication.

[0215] Finally, in order to obtain the effective channel information extracted by each feature selector, the features output by each feature selector are concatenated in the channel dimension to obtain the feature The concatenated feature map is then convolved to extract feature information that is effective for segmentation. The calculation formula is as follows:

[0216] Y=cat(N1,N2,N3,N4)

[0217]

[0218] in, It represents a convolution operation with a convolution kernel of k×k, n input channels, and m output channels. weight represents the updated channel weight information, and cat(·) represents concatenation in the channel dimension.

[0219] The following further describes how to use the video instance segmentation system of this embodiment to perform video instance recognition and prediction. The overall architecture is as follows: Fig. 9 shown.

[0220] The overall framework of the model of the present invention mainly includes two stages: feature extraction and feature post-processing. In the feature extraction stage, the t and t+δ frame images in the same video are input into the ResNet50 embedded in the IFIM module for feature extraction to generate a multi-scale feature map ψ t and t+δ ,in:

[0221] Ψ t ={F 11 ,F 12 ,F 13 ,F 14}

[0222] Ψ t+δ ={F 21 ,F 22 ,F 23 ,F 24}

[0223] In the feature post-processing stage, the multi-scale feature map output from the backbone network feature extraction stage is first input into the controller head branch to generate an instance-specific dynamic filter. Then the backbone network features are fused with multi-scale features to obtain and Then it is input into the SA-ConvLSTM module to extract the global and local dependency spatial features of the video. We perform weighted fusion to effectively utilize the temporal context information inherent in the video to enhance the cross-frame instance representation of the video. and The input mask branch and channel filtering module are used to generate the predicted instance mask F. mask Features, and concatenate them with relative position information to generate instance mask feature maps and Finally, the generated mask feature map and dynamic filter are cross-input into the mask terminal to generate static and cross-dynamic masks.

[0224] Based on the above description, the advantages of this embodiment are further described below.

[0225] Video instance segmentation is an important topic in the field of computer vision. It involves segmenting and tracking each object instance in a video sequence. It is a key technology for intelligent traffic scene assisted driving vehicles to achieve multi-target perception and scene understanding. Although there have been many advances in the fields of image segmentation and target detection, video instance segmentation still faces some unique challenges and technical problems: for example, it is difficult to improve the segmentation accuracy due to object occlusion and disappearance, background dynamic changes, and changes in the appearance of objects.

[0226] In view of the problem that the pixel-level annotation cost of the training data of the fully supervised video instance segmentation network is too high, resulting in the inability of assisted driving vehicles to quickly adapt to complex and changing scenes, this embodiment proposes a border-supervised video instance segmentation system based on temporal feature fusion. First, a block-by-block fusion strategy of inter-frame features is proposed to capture the temporal consistent correlation information between the current frame and the adjacent frames, and selective aggregation of inter-frame temporal features is achieved according to the correlation strength, so as to overcome the problem of missing initial mask detail features of the instance inherent in the current orthogonal projection and color similarity loss. Secondly, a channel screening module is constructed to adaptively obtain the importance of feature channel information to the segmentation task, and select effective feature channels accordingly to make up for the problem of partial semantic information loss and weakened representation ability caused by the reduction of the number of feature channels of the mask prediction branch. Finally, the SA-ConvLSTM module is introduced to strengthen the memory of the spatiotemporal dependency features of the video to maintain the continuity of the instance target, effectively improving the encoding performance of the segmentation network for the dynamic spatiotemporal relationship of complex sequences. The experimental results based on the traffic scene dataset extracted from Youtube-VIS2019 show that the average segmentation accuracy of the method in this embodiment is improved by 6.2% to 39.2% compared with the bounding box supervision baseline network, providing an effective algorithmic basis for realizing multi-target perception and instance-level scene understanding.

[0227] The average segmentation accuracy of the frame-supervised video instance segmentation algorithm BVIS as a benchmark network is 33.0%. In this embodiment, firstly, the IFIM module is embedded in the ResNet50 feature extraction network to capture the temporal consistent association information between the current frame and the adjacent frames, and the inter-frame temporal features are selectively aggregated according to the association strength, which effectively overcomes the problem of missing instance initial mask detail features inherent in the current orthogonal projection and color similarity loss. The results show that after the IFIM module is embedded, the average segmentation accuracy reaches 36.6% while the average segmentation speed remains basically unchanged; secondly, the CSM module is designed to adaptively capture rich and complementary channel information for each group of features, effectively compensating for the problem of instance semantic information loss and weakened representation ability caused by the sudden drop in the number of mask prediction branch channels, further improving the average segmentation accuracy to 37.7%; finally, the SA-ConvLSTM module is introduced on the basis of the above two modules to successfully extract the spatial features of video global dependency and local dependency, so that the average segmentation accuracy finally reaches 39.2%, verifying the effectiveness of this embodiment.

[0228] In addition, when processing feature maps of different channels of an image, which may cause large differences in pixel color, color similarity loss can better calculate the probability difference between the foreground and background. When the color pixel calculation difference between the foreground and background is small, obtaining a closer pixel prediction probability value will increase the model calculation amount and the segmentation effect is not good. Therefore, this embodiment designs a channel screening module to perform instance mask prediction on the fused feature map, and weightedly fuses the instance masks predicted by the CSM module and the Mask Branch to effectively make up for the shortcomings of each instance prediction.

[0229] Moreover, in the inter-frame feature integration module (IFIM), the present embodiment adopts global average pooling for the result of channel update in the update operation unit. A similar operation is global maximum pooling, which is similar to global average pooling. The global maximum pooling layer uses the maximum value of each channel as the output of the channel. This method selects the most significant feature value, so it can better capture the significant features of the image in some cases. In this embodiment, the global average pooling layer averages the pixels of each channel of the entire feature map to obtain a single value, thereby converting a feature map into a feature vector. This operation can reduce the number of parameters and retain certain spatial information.

[0230] Finally, since this embodiment embeds the inter-frame feature integration module (IFIM module) after the Block layer in the backbone network ResNet50, due to the complexity of the segmentation network, the positions that can be embedded have more differences. The reason why this embodiment is embedded in the backbone network is that the backbone network is the first processing unit for the original image input, and the Block layer is mainly a low-kernel convolution information extraction process. It uses less processing operations such as high-kernel convolution and has a lower degree of feature loss. Therefore, the network can better utilize the time dimension information in the video, thereby improving the understanding and processing capabilities of dynamic scenes, while optimizing the use of computing resources.

[0231] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the embodiment of the present application can be embodied in the form of a software product, which can be stored in a readable medium or on a network, and includes several instructions to enable an electronic device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiment of the present application.

[0232] The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. All technical solutions that fall within the scope defined by the claims of the present invention fall within the protection scope of the present invention.

Claims

1. A temporal feature fusion bounding box supervised video instance segmentation system, characterized in that: include: Feature extraction network, used to extract features covering spatiotemporal similarity information from the current frame and adjacent frames of the video and Among them, t represents the current time, δ represents the time interval between two adjacent frames; Spatiotemporal prediction self-attention network is used to and Generate memory feature M t+δ , and the memory feature M t+δ With features Weighted fusion to obtain fusion features Instance mask prediction network, used to predict the and fusion features Make instance mask prediction.

2. The border-supervised video instance segmentation system with temporal feature fusion according to claim 1, characterized in that: The backbone network of the feature extraction network adopts the residual network ResNet50, which includes 4 block layers. Each block layer is followed by an IFIM module. The input of each IFIM module is the output of the previous block layer. The output of each IFIM module is {(F 11 ,F 21 ),(F 12 ,F 22 ),(F 13 ,F 23 ),(F 14 ,F 24 )};for The first level IFIM module includes: The interactive operation unit is used to convert the current frame feature F t With adjacent frame features F t+δ The feature F containing the video timing information is obtained by splicing, and the feature F is reshaped into a feature matrix S, and then the feature matrix S and its transposed matrix S are combined. T After multiplication, the spatial information weight is normalized to obtain the similarity weight matrix D; among which, the current frame feature F t is the feature extracted by the first-level block layer through convolution operation on the current frame, and the adjacent frame feature f t+δ The features extracted by the first-level block layer through convolution operation on adjacent frames; An aggregation operation unit, used for multiplying the obtained similarity weight matrix D with the feature matrix S and reshaping to obtain an aggregate feature map Z1; An updating operation unit, configured to obtain a weight mapping relationship matrix Q using the aggregate feature map Z1, and to multiply the weight mapping relationship matrix Q by the feature F containing the video timing information to obtain a timing feature map E; A fusion operation unit is used to split the time series feature map E in the time dimension to obtain new feature maps F1 and F2, and to merge the feature maps F1 and F2 with the current frame feature map F1 and F2 respectively. t and adjacent frame features F t+δ Perform weighted fusion to obtain the feature map F covering the spatiotemporal similarity information 11 and F 21 ; The features 3. The border-supervised video instance segmentation system with temporal feature fusion according to claim 2, characterized in that: The interactive operation unit is further used for: The current frame feature F t With adjacent frame features F t+δ The feature F containing the video timing information is obtained by splicing. The calculation process is as follows: F=stack(F t ,F t+δ ); The stack(·) function represents a concatenation operation in a new dimension; the feature F is reshaped into a matrix S as follows: S = Reshape(F); The Reshape(·) function represents the feature reshaping operation; the matrix S and its transposed matrix S T Multiply the semantic similarity information of the previous and next frames of the video. Finally, use the Softmax activation function to normalize the spatial information weights to obtain the similarity weight matrix D, as follows: in, Represents element-wise multiplication.

4. The border-supervised video instance segmentation system with temporal feature fusion according to claim 3, characterized in that: The characteristic map Z1 is expressed as:

5. The border-supervised video instance segmentation system with temporal feature fusion according to claim 2, characterized in that: The weight mapping relationship matrix Q is expressed as: Q=GAP(σ1(BN(Conv 1×1 (Z1)))) Among them, GAP(·) represents the global average pooling operation, σ1(·) represents the ReLU activation function, BN(·) represents the batch normalization operation, Conv k×k (·) represents a convolution operation with a kernel size of k×k.

6. The border-supervised video instance segmentation system with temporal feature fusion according to claim 2, characterized in that: Feature map F 11 and F 21 Respectively expressed as: F1, F2 = split(E); Among them, split(·) means splitting the five-dimensional time series feature into two four-dimensional features. Represents element-by-element addition.

7. The frame-supervised video instance segmentation system with temporal feature fusion according to claim 1, characterized in that: The spatiotemporal prediction self-attention network includes an initialization module, a first spatiotemporal prediction self-attention module and a second spatiotemporal prediction self-attention module, wherein the output of the first spatiotemporal prediction self-attention module is used as the input of the second spatiotemporal prediction self-attention module; the first spatiotemporal prediction self-attention module includes: Initialization module, used to initialize feature M t-δ , C t-δ ,in, M t-δ represents a new memory unit with self-attention mechanism at time t-δ; represents the hidden state of the first spatiotemporal prediction self-attention module at time t-δ; C t-δ represents the original storage memory unit at time t-δ; The feature set represents a batch of 4 feature dimensions, 128 channels, 48 ​​height, and 80 width; The spatial feature extraction unit is used to extract the spatial feature according to the initialization feature. C t-δ and the features input by the feature extraction network Extract the feature H at the current time t t and C t ; SAM unit is used to calculate the t and initialization feature M t-δ Generate the feature M at the current time t t and 8. The border-supervised video instance segmentation system with temporal feature fusion according to claim 7, characterized in that: The feature C t The calculation is as follows: C t =f t ⊙C t-δ +i t ⊙g t Among them, ⊙ represents the matrix dot product, f t 、i t , g t They are: Among them, σ2(·) represents the Sigmoid activation function, tanh(·) represents the tanh activation function; f t represents the first forget gate, W hf is the weight of the first forget gate, b f is the bias of the first forget gate; i t represents the first input gate, W hi is the weight of the first input gate, b i represents the bias of the first input gate; g t represents the memory cell state of the first spatiotemporal prediction self-attention module, W hC is the weight of the memory cell state, b c is the bias of the memory cell state.

9. The border-supervised video instance segmentation system with temporal feature fusion according to claim 8, characterized in that: The feature H t The calculation process is as follows: H t =O t ⊙tanh(C t ) Among them, O t is the first output gate, W ho is the weight of the first output gate, b o is the bias of the first output gate.

10. The border-supervised video instance segmentation system with temporal feature fusion according to claim 9, characterized in that: The feature M t The calculation process is as follows: M t =(1-i′ t )⊙M t-δ +i′ t ☉g′ t Among them, i′ t , g′ t They are respectively expressed as follows: i′ t =σ2(W m；zi ×Z+W m；hi ×H t +b m；i ) g′ t =tanh(W m；zg ×Z+W m；hg ×H t +b m；g ) Among them, i′ t is the second input gate, W m；zi , W m；hi is the weight of the second input gate, b m；i is the bias of the second input gate; g′ t Represents the memory cell state of the SAM unit, W m；zg , W m；hg is the weight of the memory cell state, b m；g is the bias of the memory cell state.

11. The border-supervised video instance segmentation system with temporal feature fusion according to claim 10, characterized in that: The features The calculation is as follows: Among them, O t ′ is the second output gate, and the expression is as follows: o′ t =σ2(W m；zo ×Z+W m；ho ×H t +b m；o ) Among them, W m；zo , W m；ho is the weight of the second output gate, b m；o is the bias for the second output gate.

12. The border-supervised video instance segmentation system with temporal feature fusion according to claim 11, characterized in that: The second spatiotemporal prediction self-attention module has the same structure as the first spatiotemporal prediction self-attention module; the second spatiotemporal prediction self-attention module is used to output the feature C of the first spatiotemporal prediction self-attention module according to the feature C of the first spatiotemporal prediction self-attention module. t , M t And the features extracted by the feature extraction network Generate memory feature M t+δ .

13. The border-supervised video instance segmentation system with temporal feature fusion according to claim 2, characterized in that: The instance mask prediction network includes: The first CSM module and the first mask branch use features Perform instance mask prediction and perform weighted fusion of the respective predicted instance masks; The second CSM module and the second mask branch respectively utilize features Perform instance mask prediction and perform weighted fusion of the respective predicted instance masks; The second CSM module has the same structure as the first CSM module, and the second mask branch has the same structure as the first mask branch.

14. The border-supervised video instance segmentation system with temporal feature fusion according to claim 13, characterized in that: The first CSM module comprises: The optimization unit is used to perform batch normalization on the features M output by the feature extraction network and activate them through the ReLU function to obtain the optimized feature map M ′ , the calculation formula is as follows: M′=σ1(BN(M)) Among them, M∈{F 11 , F 12 , F 13 }, BN(·) represents batch normalization operation, σ1(·) represents ReLU activation function; Channel screening unit, with optimized feature map M ′ As input, it is used to perform channel screening to extract effective channel information G p , p=12,3,4,G p Represents the effective channel information extracted by the pth feature selector; The convolution unit is used to convolute the effective channel information G p Perform concatenation and convolution operations to extract effective feature information U.

15. The border-supervised video instance segmentation system with temporal feature fusion according to claim 14, characterized in that: The effective channel information G p The calculation process is as follows: G p =fc 8 / 128 (fc 128 / 8 (G)) Among them, fc 128 / 8 represents the first fully connected layer, whose input dimension is 128 and output dimension is 8; fc 8 / 128 represents the second fully connected layer, whose input dimension is 8 and output dimension is 128; G is the global average pooling result, which is expressed as follows:

16. The border-supervised video instance segmentation system with temporal feature fusion according to claim 15, characterized in that: The effective characteristic information U is expressed as follows: in, It represents a convolution operation with a convolution kernel of k×k, n input channels, and m output channels. Weight represents the updated channel weight information. Y is the concatenation feature, which is expressed as follows: Y = cat(N1, N2, N3, N4); 17. The border-supervised video instance segmentation system with temporal feature fusion according to claim 15, characterized in that: The first CSM module also includes a regularization loss unit for regularizing the effective channel information G p Take the mean and add it to the loss term as a regularization result for optimization. The regularized loss function is expressed as: L loss,p =mean(G p ) Among them, mean(·) represents the mean operation.

Images