A video instance segmentation method, device and medium

Abstract

The application relates to a video instance segmentation method and device and a storage medium, which comprises the following steps: inputting video data into a deep learning model; extracting a plurality of layers of first feature maps of each video frame image based on the deep backbone network; performing information fusion processing on the extracted plurality of layers of first feature maps based on the feature pyramid network to obtain a plurality of layers of second feature maps after information fusion; processing the plurality of layers of second feature maps by using the RPN network to obtain a plurality of candidate regions; performing boundary box detection and mask prediction on each candidate region by using the boundary box detection branch and the mask prediction branch to obtain the instance segmentation result of each video frame image; and extracting the feature vectors of the instances in the boundary boxes of different video frame images by using the full convolution network based on the label prediction branch, and matching the instances according to the extracted feature vectors.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, and particularly relates to a video instance segmentation method, device and computer readable storage medium. BACKGROUND

[0002] The video instance segmentation task is a relatively new computer vision task proposed after the image target detection, semantic segmentation, instance segmentation and target tracking. The video instance segmentation task aims to identify and segment the instances of the video content of the video frame, and match the same instances between different video frames. The video instance segmentation task can achieve more explicit instance division than the image target tracking task.

[0003] After the video instance segmentation task is proposed, the mainstream algorithm model realizes the task in a two-stage manner. The first stage is to realize the instance segmentation of the in-frame image, and the second stage realizes the instance matching between frames.

[0004] However, the present inventors found in research that the second stage of the prior art needs to enhance the representation ability of the extracted instance image features when realizing the instance matching between frames, that is, there is room for improvement in the instance matching effect extracted by the prior art. Moreover, the result of the instance matching between frames in the second stage depends on the result of the instance segmentation of the in-frame image in the first stage. If the instance segmentation result in the first stage is more accurate, the matching result in the second stage can be better. SUMMARY

[0005] To solve the above problems, the present application provides a video instance segmentation method, device and computer readable storage medium, which can extract a feature vector that better represents instances and realize better instance matching between frames.

[0006] To achieve the above-mentioned purposes, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a video instance segmentation method, which comprises:

[0008] inputting video data into a deep learning model, wherein the deep learning model comprises a deep backbone network, a feature pyramid network, an RPN network, a bounding box detection branch, a mask prediction branch and a label prediction branch;

[0009] extracting a plurality of layers of first feature maps of each video frame image input based on the deep backbone network;

[0010] performing information fusion processing on the extracted plurality of layers of first feature maps based on the feature pyramid network to obtain a plurality of layers of second feature maps after information fusion;

[0011] processing the multi-layer second feature map by using the RPN network to obtain a plurality of candidate regions;

[0012] detecting a bounding box and predicting a mask for each of the candidate regions by using the bounding box detection branch and the mask prediction branch to obtain a result of instance segmentation of each of the video frame images;

[0013] extracting a feature vector of an instance within a bounding box of different video frame images by using a fully convolutional network based on the label prediction branch, and performing matching of the instance according to the extracted feature vector.

[0014] In an implementation manner of the present application, the method further comprises: pre-processing each video frame image of the input video data, and then inputting the deep learning model.

[0015] In an implementation manner of the present application, the extracting, based on the deep backbone network, of the multi-layer first feature map of each input video frame image comprises: extracting, based on a ResNet backbone residual network, the multi-layer first feature map of each input video frame image.

[0016] In an implementation manner of the present application, the information fusion processing of the extracted multi-layer first feature map based on the feature pyramid network comprises:

[0017] performing top-down information fusion processing on the extracted multi-layer first feature map, and performing information fusion of each layer of the first feature map and a layer of the first feature map above it.

[0018] In an implementation manner of the present application, the extracting, by using a fully convolutional network, of the feature vector of the instance within the bounding box of different video frame images comprises:

[0019] extracting, by using a fully convolutional recurrent network structure, the feature vector of the instance within the bounding box of different video frame images.

[0020] In an implementation manner of the present application, the extracting, by using a fully convolutional recurrent network structure, of the feature vector of the instance within the bounding box of different video frame images comprises:

[0021] inputting an image within the bounding box into a convolutional layer of a prediction level for feature extraction;

[0022] recursively inputting, again, a result of the feature extraction by the convolutional layer of the prediction level into the convolutional layer of the prediction level for feature extraction;

[0023] acquiring the feature vector of the instance within the bounding box by performing global average pooling on a result of the recursive input into the convolutional layer of the prediction level for feature extraction.

[0024] In an implementation form of the method, the method further comprises: pre-cyclically training the deep learning model until the deep learning model converges.

[0025] In an implementation form of the method, the cyclically training the deep learning model comprises:

[0026] The weights of the deep learning network are learned and updated by using a stochastic gradient descent optimization algorithm until the deep learning model converges.

[0027] In a second aspect, the present application provides a video instance segmentation device, the video instance segmentation device comprising a deep backbone network, a feature pyramid network, an RPN network, a bounding box detection branch, a mask prediction branch, and a label prediction branch.

[0028] The deep backbone network is configured to extract a plurality of layers of first feature maps of each input video frame image.

[0029] The feature pyramid network is configured to perform information fusion processing on the extracted plurality of layers of first feature maps to obtain a plurality of layers of second feature maps after information fusion.

[0030] The RPN network is configured to process the plurality of layers of second feature maps to obtain a plurality of candidate regions.

[0031] The bounding box detection branch and the mask prediction branch are configured to perform bounding box detection and mask prediction on each of the candidate regions to obtain the results of instance segmentation of each of the video frame images.

[0032] The label prediction branch is configured to extract feature vectors of instances within the bounding boxes of different video frame images by using a full convolutional network, and perform matching of the instances according to the extracted feature vectors.

[0033] In a third aspect, the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program running to control a device where the computer readable storage medium is located to perform the video instance segmentation method of the first aspect.

[0034] The application has the following advantages: in the application, video data is input into a deep learning model, a plurality of first feature maps of each frame of video image are extracted based on a deep backbone, information fusion processing is performed on the extracted first feature maps based on a feature pyramid network, and a second feature map is obtained. Since the second feature map is obtained through information fusion processing, the semantic information of high-level features is taken into account, and the low-level feature positioning accuracy is high. Further, the RPN network is used to generate a candidate region, and the bounding box detection branch and the mask prediction branch are used for processing, so that a higher-quality instance segmentation result can be obtained. Meanwhile, based on the label prediction branch, the full convolution network is used to extract the feature vectors of the instances in the bounding box of different video frame images, and the instances are matched according to the extracted feature vectors, so that the feature vectors with better representation of instances can be extracted, and better instance matching between frames can be realized. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of a video instance segmentation method provided by an embodiment of the application;

[0036] Figure 2 is an architecture and data flow diagram of a deep learning model related to an embodiment of the application;

[0037] Figure 3 is a structure diagram of an RPN network related to an embodiment of the application;

[0038] Figure 4 is a full convolution network structure diagram of a label prediction branch in an embodiment of the application;

[0039] Figure 5 is a functional block diagram of a video instance segmentation device provided by an embodiment of the application;

[0040] Figure 6 is a structure diagram of a computer device related to an embodiment of the application. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme of the embodiments of the application will be described clearly and completely below with reference to the drawings of the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the described embodiments of the application, all other embodiments obtained by those skilled in the art belong to the scope of protection of the application.

[0042] The two-stage video instance segmentation method of the prior art has the problems that the representation capability of the extracted instance image features needs to be enhanced when the second stage implements instance matching between frames, and the result of the instance matching between frames in the second stage depends on the result of the instance segmentation of the image in the first stage, and the accuracy of the result of the instance segmentation in the first stage is not high. A video instance segmentation method, device and medium are provided, which include: inputting video data into a deep learning model, the deep learning model including a deep backbone network, a feature pyramid network, an RPN network, a bounding box detection branch, a mask prediction branch and a label prediction branch; extracting a plurality of layers of first feature maps of each video frame image input based on the deep backbone network; performing information fusion processing on the extracted plurality of layers of first feature maps based on the feature pyramid network to obtain a plurality of layers of second feature maps after information fusion; processing the plurality of layers of second feature maps using the RPN network to obtain a plurality of candidate regions; performing bounding box detection and mask prediction on each candidate region using the bounding box detection branch and the mask prediction branch to obtain the result of instance segmentation of each video frame image; and extracting a feature vector of an instance in a bounding box of different video frame images based on the label prediction branch using a full convolutional network, and performing instance matching according to the extracted feature vector. The technical scheme of the present application can extract a feature vector that better represents an instance and achieve better instance matching between frames.

[0043] Reference is made to Figure 1 In an embodiment of the present application, a video instance segmentation method is provided. The video instance segmentation method in the embodiment of the present application can extract a feature vector that better represents an instance and achieve better instance matching between frames.

[0044] The video instance segmentation method in the embodiment of the present application specifically includes the following steps:

[0045] S11, input video data into a deep learning model.

[0046] The deep learning model includes a deep backbone network, a feature pyramid network, an RPN network, a bounding box detection branch, a mask prediction branch and a label prediction branch.

[0047] S12, extract a plurality of layers of first feature maps of each video frame image input based on the deep backbone network.

[0048] S13, perform information fusion processing on the extracted plurality of layers of first feature maps based on the feature pyramid network to obtain a plurality of layers of second feature maps after information fusion.

[0049] S14, process the plurality of layers of second feature maps using the RPN network to obtain a plurality of candidate regions.

[0050] S15, performing bounding box detection and mask prediction on each candidate region by using the bounding box detection branch and the mask prediction branch to obtain the instance segmentation result of each video frame image.

[0051] S16, based on the label prediction branch, extracting the feature vector of the instance in the bounding box of different video frame images by using the fully convolutional network, and performing instance matching according to the extracted feature vector.

[0052] The video instance segmentation method in the embodiments of the present application inputs video data into a deep learning model, extracts a plurality of first feature maps of each frame of video image based on a deep backbone, and performs information fusion processing on the extracted first feature maps based on a feature pyramid network to obtain a second feature map. Since the second feature map is obtained after information fusion processing, it takes into account both the semantic information of high-level features and the advantage of high positioning accuracy of low-level features. Further, the RPN network is used to generate candidate regions, and the bounding box detection branch and the mask prediction branch are used for processing, so that a higher quality instance segmentation result can be obtained. Meanwhile, based on the label prediction branch, the fully convolutional network is used to extract the feature vector of the instance in the bounding box of different video frame images, and the instances are matched according to the extracted feature vector, so that a feature vector with better representation of the instance can be extracted, and better instance matching between frames can be achieved.

[0053] In some more detailed embodiments of the present application, the above processes S11-S16 are further described in more detail in combination with more drawings to further describe the technical solutions of the present application and explain the advantages thereof.

[0054] The instance segmentation method of the present application comprises:

[0055] S11, inputting video data into a deep learning model.

[0056] Specifically, the specific architecture of the deep learning model involved in the embodiments of the present application can refer to Figure 2 .

[0057] In Figure 2 , the deep learning model shown comprises a deep backbone network 201, a feature pyramid network 202, an RPN network 203, a bounding box detection branch 204, a mask prediction branch 205, and a label prediction branch 206.

[0058] In some embodiments of the present application, the input video data can be, but is not limited to, a video image of RGB three color channels collected by an image collection device.

[0059] Further, in the embodiments of the present application, the input image in the form of RGB three color channels can be pre-processed, such as size scaling, numerical normalization, and then input into the deep learning model for processing.

[0060] S12, based on the deep backbone network, a plurality of first feature maps of each video frame image input are extracted.

[0061] Specifically, in the embodiments of the present application, the video deep learning model is arranged according to the order of the video frame data. According to the order of the video frame, the multi-layer feature map is extracted.

[0062] As Figure 2 In the embodiments of the present application, the video frame data is input into the ResNet backbone residual network for multi-stage feature extraction, and different levels of image information are extracted at each stage. The backbone network includes four stages, and four feature maps with different sizes and orders are obtained by taking the feature maps output at each stage. Specifically, the feature map can be represented as x i,＝1,,, , where i represents the stage to which the feature map belongs, and a larger i value means that the feature map is generated by a deeper stage.

[0063] S13, based on the feature pyramid network, the information fusion processing of the extracted multi-layer first feature map is performed to obtain a multi-layer second feature map after information fusion.

[0064] In the embodiments of the present application, a processing method of information fusion is provided. Specifically, the extracted multi-layer first feature map is processed by top-down information fusion, and each layer of the first feature map is fused with the first feature map above it.

[0065] Taking the four-stage feature map as input, the feature pyramid first performs top-down feature fusion: the highest layer x4 is unchanged, and each layer of feature is fused by the input x i of the current layer and the input x i+1 of the higher layer, and the fusion is performed by interpolation size scaling and corresponding position addition, and the fusion relationship is represented as The result of top-down feature fusion is:

[0066]

[0067] In order to further improve the effect of information fusion and make the fused feature map have better representation effect, the present application can further provide a further information fusion processing scheme to improve the traditional pyramid network.

[0068] In the embodiments of the present application, the pyramid network of the present application performs top-down information fusion processing on the extracted multi-layer first feature maps, and fuses each layer of the first feature maps with all the first feature maps above it.

[0069] More specifically, the specific steps of the dense fusion pyramid network in the information fusion processing can include:

[0070] Based on each layer of the first feature maps, the weighted coefficients of each layer of the first feature maps are calculated; and

[0071] Based on the weighted coefficients corresponding to each first feature map, each layer of the first feature maps and all the first feature maps above it are weighted and superimposed to obtain the second feature map corresponding to each layer of the first feature maps after information fusion.

[0072] More specifically, the calculation method of information fusion can be:

[0073] Based on each layer of the first feature maps, the weighted coefficients of each layer of the first feature maps are calculated; and

[0074] Based on the weighted coefficients corresponding to each first feature map, each layer of the first feature maps and all the first feature maps above it are weighted and superimposed to obtain the second feature map corresponding to each layer of the first feature maps after information fusion.

[0075] For example:

[0076] According to each layer of the first feature maps, the average value corresponding to each layer of the first feature maps is calculated;

[0077] According to the average value corresponding to each layer of the first feature maps, the function value of the sigmoid function is calculated as the first coefficient, which is specifically represented as:

[0078] sa i = sigmoid (average (x i )) (2)

[0079] Where x is the first feature map, i = 1, 2, … N (N is the number of layers of the feature map, which can be 4, etc.).

[0080] According to the first coefficient of each layer of the first feature maps, the weighted coefficients of each layer of the first feature maps are calculated based on the softmax function, and then weighted and superimposed based on the weighted coefficients, which is specifically represented as:

[0081]

[0082] Where X i is the fused second feature map, i = 1, 2, … N.

[0083] S14, processing the multi-layer second feature map by using the RPN network to obtain a plurality of candidate regions.

[0084] Specifically, the architecture of the RPN (Region Proposal Network) is as shown in Figure 3 The RPN network takes a feature map as input, generates anchor boxes of different sizes and different dimensions at different positions of the feature map at different stages, extracts features by using operations such as convolution, activation, and pooling, classifies each anchor box as foreground or background and calculates the class confidence, filters out non-foreground anchor boxes according to a confidence threshold, and performs target class prediction and bounding box prediction on the remaining foreground anchor boxes to obtain target class information and bounding box position information of the anchor boxes.

[0085] In the embodiment of the present application, the RPN generates a plurality of regions of interest (ROIs) based on the multi-layer second feature map.

[0086] S15, performing bounding box detection and mask prediction on each of the candidate regions by using the bounding box detection branch and the mask prediction branch to obtain the result of instance segmentation.

[0087] Specifically, the regions of interest (ROIs) are mapped to the second feature map by using a region of interest alignment strategy (RoIAlign), and subsequent instance prediction and mask prediction are performed on the obtained regions.

[0088] In the embodiment of the present application, the instance class, instance bounding box, and instance mask are predicted for each region, and the final result of instance segmentation is output.

[0089] S16, based on the label prediction branch, extracting a feature vector of an instance within a bounding box of a different video frame image by using a fully convolutional network, and performing matching of the instance according to the extracted feature vector.

[0090] Specifically, in the traditional video instance segmentation tracker, the feature vector extraction of the instance within the bounding box is implemented by using a fully connected layer, and the representation ability of the extracted feature vector needs to be enhanced.

[0091] In the embodiment of the present application, the feature vector extraction of the instance within the bounding box of a different video frame image is implemented by using a fully convolutional network.

[0092] In the embodiment of the present application, the architecture of the fully convolutional network and the feature extraction manner are as shown in Figure 4 The data processing manner includes:

[0093] inputting the image within the bounding box into the convolutional layer of the prediction level to perform feature extraction;

[0094] The result of the feature extraction of the convolutional layer of the prediction level is recursively input again into the convolutional layer of the prediction level for feature extraction.

[0095] The result of the feature extraction of the convolutional layer of the prediction level is recursively input again into the convolutional layer of the prediction level for feature extraction.

[0096] Specifically in Figure 4 In the specific embodiment, the instance image in the input bounding box is subjected to twice 3x3 convolution operation for preliminary feature extraction, and the extracted preliminary features are introduced into the two convolutional layers of the previous step in a recursive structure for feature re-extraction to obtain the extracted feature map. Finally, the global average pooling operation is performed on the extracted feature map to generate a feature vector of a specified dimension for subsequent instance label assignment processing.

[0097] In the embodiment of the present application, the instances between frames are matched by id, so as to achieve the purpose of video instance segmentation. In the instance id matching, the id assignment problem of a new instance is modeled as a multi-classification problem: assuming that N instances have been detected, the new instance is either one of the N instances or a new instance, and thus the classification problem of assigning a label n to a candidate box i can be defined as follows:

[0098]

[0099] In the embodiment of the present application, in order to make the deep learning model achieve better results, the deep learning model is trained in a loop until the deep learning model converges. For example, the random gradient descent optimization algorithm is used to learn and update the weights of the deep learning network until the deep learning model converges.

[0100] The technical solution of the present application is described below in a specific model, and the advantages of the present application are illustrated with specific numerical values.

[0101] Specifically, the network architecture of the model can refer to Figure 2 .

[0102] The processing steps of the model of the present application include:

[0103] Step 1: initialize the weights of the model of the present application.

[0104] Step 2: input the video frame data into the backbone residual network of ResNet, fully perform deep feature extraction through the operation of the deep network, and obtain multi-scale first feature maps at different depths by extracting feature maps at different stages of the deep backbone network;

[0105] Step 3: The multi-scale first feature map is input into a feature pyramid network to perform a multi-scale information fusion operation, further strengthen high and low order feature fusion and feature re-extraction, and obtain a multi-scale second feature map with more information.

[0106] The specific information fusion processing process can refer to the foregoing embodiments, which will not be described here.

[0107] Step 4: The second feature map extracted in step 3 is subjected to RPN to generate a candidate region proposal, the features are selected and the RPN parameters are adjusted according to the feature maps at different stages, and the region of interest ROIs are output.

[0108] Step 5: The region of interest ROIs generated in step 4 are aligned to the second feature map generated in step 3 through a region of interest alignment strategy RoIAlign, and the obtained regions are subjected to subsequent instance prediction and mask prediction.

[0109] Step 6: The candidate regions divided by RoIAlign are taken as the input of the head network, and the instance class, instance bounding box, instance mask and instance label of each region are predicted.

[0110] Step 7: The current batch of data is learned, a stochastic gradient descent optimization algorithm is adopted to learn and update the network weight, and the process returns to step 2 until the model training converges.

[0111] In the embodiments of the present application, the effect of the present application can be shown through experiments on the YOUTUBE-VIS 2021 dataset.

[0112] Firstly, the deep learning model achieves good convergence effect, wherein the classification loss, regression loss, mask loss and tracking loss of the algorithm converge significantly, and the total loss composed of the classification loss, regression loss, mask loss and tracking loss converges significantly.

[0113] Meanwhile, under the same experimental conditions, the present application is compared with the existing MaskTrack RCNN, the model effect is tested on the YOUTUBE-VIS 2021 dataset, and the verification is performed on the YOUTUBE-VIS verification set, and the comparison results of the existing model and the traditional model are shown in the following table:

[0114]

[0115]

[0116] Table 1

[0117] In Table 1, it can be seen that the model of the present application realizes the improvement of algorithm performance compared with the existing model.

[0118] In another aspect of the present application, a video instance segmentation device is also correspondingly provided.

[0119] Referring to Figure 5 In an embodiment of the present application, a video instance segmentation device 600 is illustrated.

[0120] The instance segmentation device 600 in the embodiment of the present application comprises a deep backbone network 601, a feature pyramid network 602, an RPN network 603, a bounding box detection branch 604, a mask prediction branch 605 and a label prediction branch 606.

[0121] The deep backbone network 601 is configured to extract a plurality of layers of first feature maps of each input video frame image;

[0122] The feature pyramid network 602 is configured to perform information fusion processing on the extracted plurality of layers of first feature maps to obtain a plurality of layers of second feature maps after information fusion;

[0123] The RPN network 603 is configured to process the plurality of layers of second feature maps to obtain a plurality of candidate regions;

[0124] The bounding box detection branch 604 and the mask prediction branch 605 are configured to perform bounding box detection and mask prediction on each of the candidate regions to obtain the results of instance segmentation of each of the video frame images;

[0125] The label prediction branch 606 is configured to extract feature vectors of instances within the bounding boxes of different video frame images by using a full convolutional network, and perform matching of the instances according to the extracted feature vectors.

[0126] The device of the present application will be described in more detail in the embodiments of the present application.

[0127] Specifically, the specific architecture of the deep learning model involved in the embodiments of the present application can refer to Figure 2 .

[0128] In Figure 2 , the deep learning model illustrated comprises a deep backbone network 201, a feature pyramid network 202, an RPN network 203, a bounding box detection branch 204, a mask prediction branch 205 and a label prediction branch 206.

[0129] In some embodiments of the present application, the input video data can be, but is not limited to, video images of RGB three color channels collected by an image collection device.

[0130] Further, in the embodiments of the present application, the input image in the form of RGB three color channels can be pre-processed, such as size scaling, numerical normalization, and then input into the deep learning model for processing.

[0131] The deep backbone network 601 is used to extract a plurality of layers of first feature maps of each video frame image input.

[0132] Specifically, in the embodiments of the present application, the video deep learning model is arranged according to the order of the video frame data. According to the order of the video frame, a plurality of layers of feature maps are extracted.

[0133] As Figure 2 In the embodiments of the present application, the video frame data is input into the ResNet backbone residual network for multi-stage feature extraction, and different levels of image information are extracted at each stage. The backbone network includes four stages, and four feature maps with different sizes and orders are obtained by taking the output feature maps of each stage. Specifically, the feature map can be represented as x i,＝1,,, , where i represents the stage to which the feature map belongs, and a larger i value means that the feature map is generated by a deeper stage.

[0134] The feature pyramid network 602 is used to perform information fusion processing on the extracted plurality of layers of first feature maps, to obtain a plurality of layers of second feature maps after information fusion.

[0135] In the embodiments of the present application, a processing method of information fusion is provided, and specifically, the extracted plurality of layers of first feature maps are processed by top-down information fusion, and each layer of first feature maps is fused with the layer of first feature maps above it.

[0136] Taking the four-stage feature map as input, the feature pyramid first performs top-down feature fusion: the highest layer x4 is unchanged, and each layer of feature is fused by the input x i of the current layer and the input x i+1 of the layer above it by interpolation size scaling and corresponding position addition, where the fusion relationship is represented as x , and the result of top-down feature fusion is:

[0137]

[0138] In order to further improve the effect of information fusion and make the fused feature map have better representation effect, the present application can further provide a further information fusion processing scheme to improve the traditional pyramid network.

[0139] In the embodiments of the present application, the pyramid network of the present application performs top-down information fusion processing on the extracted multi-layer first feature maps, and fuses each layer of the first feature maps with all the first feature maps above it.

[0140] More specifically, the specific steps of the dense fusion pyramid network in the information fusion processing can include:

[0141] based on each layer of the first feature maps, calculating a weighting coefficient of each layer of the first feature maps; and

[0142] based on the weighting coefficient corresponding to each first feature map, weighting and superimposing each layer of the first feature maps with all the first feature maps above it to obtain a second feature map corresponding to each layer of the first feature maps after information fusion.

[0143] More specifically, the calculation method of information fusion can be:

[0144] based on each layer of the first feature maps, calculating a weighting coefficient of each layer of the first feature maps; and

[0145] based on the weighting coefficient corresponding to each first feature map, weighting and superimposing each layer of the first feature maps with all the first feature maps above it to obtain a second feature map corresponding to each layer of the first feature maps after information fusion.

[0146] For example:

[0147] According to each layer of the first feature maps, calculating a mean value corresponding to each layer of the first feature maps;

[0148] According to the mean value corresponding to each layer of the first feature maps, calculating a function value of a sigmoid function as a first coefficient, which is specifically represented as:

[0149] sa i = sigmoid (aveerage (x i )) (2)

[0150] wherein, x is the first feature map, i = 1, 2, … N (N is the number of layers of the feature map, which can be 4, etc.).

[0151] According to the first coefficient of each layer of the first feature maps, calculating a weighting coefficient of each layer of the first feature maps based on a softmax function, and then weighting and superimposing based on the weighting coefficient, which is specifically represented as:

[0152]

[0153] wherein, X i is the fused second feature map, i = 1, 2, … N.

[0154] The RPN network 603 is configured to process the multi-layer second feature map to obtain a plurality of candidate regions.

[0155] Specifically, the architecture of the RPN (Region Proposal Network) is as shown in Figure 3 The RPN network takes a feature map as input, generates anchor boxes of different sizes and different dimensions at different positions of the feature map at different stages, extracts features through convolution, activation, and pooling, classifies each anchor box as foreground or background and calculates the class confidence, filters out non-foreground anchor boxes according to a confidence threshold, and performs target class prediction and bounding box prediction on the remaining foreground anchor boxes to obtain target class information and bounding box position information of the anchor boxes.

[0156] In the embodiment of the present application, the RPN generates a plurality of regions of interest (ROIs) based on the multi-layer second feature map.

[0157] The bounding box detection branch 604 and the mask prediction branch 605 are configured to perform bounding box detection and mask prediction on each of the candidate regions to obtain the results of instance segmentation of each of the video frame images.

[0158] Specifically, the regions of interest (ROIs) are mapped to the second feature map through a region of interest alignment strategy RoIAlign, and the obtained regions are subjected to subsequent instance prediction and mask prediction.

[0159] In the embodiment of the present application, the prediction of instance class, instance bounding box, and instance mask is performed on each region, and the final results of instance segmentation are output.

[0160] The label prediction branch 606 is configured to extract feature vectors of instances within the bounding boxes of different video frame images using a fully convolutional network, and perform matching of the instances according to the extracted feature vectors.

[0161] Specifically, in the traditional video instance segmentation tracker, the extraction of feature vectors of instances within the bounding boxes is achieved through a fully connected layer, and the representation ability of the extracted feature vectors needs to be enhanced.

[0162] In the embodiment of the present application, the feature vectors of instances within the bounding boxes of different video frame images are extracted through a fully convolutional network.

[0163] In the embodiment of the present application, the architecture of the fully convolutional network and the feature extraction method are as shown in Figure 4 The data processing method includes:

[0164] inputting the image within the bounding box into the convolutional layer of the prediction level for feature extraction;

[0165] The result of the feature extraction of the convolutional layer of the prediction level is recursively input again into the convolutional layer of the prediction level for feature extraction.

[0166] The result of the feature extraction of the convolutional layer of the prediction level is recursively input again into the convolutional layer of the prediction level for feature extraction.

[0167] Specifically in Figure 4 In the example, the instance image in the input bounding box is subjected to twice 3x3 convolution operation for preliminary feature extraction, the extracted preliminary features are introduced into the two convolutional layers of the previous step in a recursive structure for feature re-extraction to obtain the extracted feature map, and finally the global average pooling operation is performed on the extracted feature map to generate a feature vector of a specified dimension for subsequent instance label assignment processing.

[0168] In the embodiment of the present application, the instances between frames are matched in id, so as to achieve the purpose of video instance segmentation. In the instance id matching, the id assignment problem of a new instance is modeled as a multi-classification problem: assuming that N instances have been detected, the new instance is either one of the N instances or a new instance, and thus the classification problem of assigning a label n to a candidate box i can be defined as follows:

[0169]

[0170] In the embodiment of the present application, in order to make the deep learning model achieve better results, the deep learning model is trained in a loop until the deep learning model converges. For example, the weights of the deep learning network are learned and updated by using a stochastic gradient descent optimization algorithm until the deep learning model converges.

[0171] The embodiment of the present application further provides a computer readable storage medium, the storage medium comprising a stored program, wherein when the program is running, the device where the storage medium is located performs the instance segmentation method described above. The specific implementation process is not repeated here.

[0172] The embodiment of the present application further provides a computer device 500, as shown in Figure 6 The computer device 500 of the embodiment comprises a processor 501, a memory 502, and a computer program 503 stored in the memory and executable on the processor 501, and the processor 501 implements the video instance segmentation method in the embodiment when executing the computer program 503. To avoid repetition, details are not repeated here. Alternatively, the computer program is executed by the processor 501 to implement the functions of each model / unit in the video instance segmentation apparatus in the embodiment, and to avoid repetition, details are not repeated here.

[0173] The computer device 500 can be a desktop computer, a notebook computer, a palm computer, a server computer, a cloud server computer, and the like. The computer device can include, but is not limited to, a processor 501, a memory 502. Those skilled in the art can understand that Figure 6 The computer device 500 is only an example and does not constitute a limitation on the computer device 500, and can include more or fewer components than shown, or combine certain components, or include different components, for example, the computer device can also include an input / output device, a network access device, a bus, and the like.

[0174] The processor 501 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0175] The memory 502 can be an internal storage unit of the computer device 500, such as a hard disk or a memory of the computer device 500. The memory 502 can also be an external storage device of the computer device 500, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the memory 502 can include both the internal storage unit and the external storage device of the computer device 500. The memory 502 is used to store computer programs and other programs and data required by the computer device. The memory 502 can also be used to temporarily store data that has been output or will be output.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the system, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0177] In several embodiments provided by the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the embodiments of the device described above are merely schematic, and the division of the units is merely a logical function division. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0178] The integrated unit implemented in the form of a software functional unit can be stored in a computer readable storage medium. The software functional unit stored in the storage medium includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (Processor) to perform some of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk.

[0179] The above merely describes the preferred embodiments of the present application and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method of video instance segmentation, the method comprising: The method comprises: inputting video data into a deep learning model, the deep learning model comprising a deep backbone network, a feature gold tower network, an RPN network, a bounding box detection branch, a mask prediction branch, and a label prediction branch; extracting a plurality of layers of first feature maps of each video frame image input based on the deep backbone network; performing information fusion processing on the extracted plurality of layers of first feature maps based on the feature gold tower network to obtain a plurality of layers of second feature maps after information fusion; processing the plurality of layers of second feature maps using the RPN network to obtain a plurality of candidate regions; performing bounding box detection and mask prediction on each of the candidate regions using the bounding box detection branch and the mask prediction branch to obtain the results of instance segmentation of each video frame image; based on the label prediction branch, extracting feature vectors of instances within the bounding boxes of different video frame images using a fully convolutional network, and performing matching of the instances according to the extracted feature vectors; wherein the extracting of the feature vectors of the instances within the bounding boxes of different video frame images using the fully convolutional recurrent network structure comprises: inputting the image within the bounding box into a convolutional layer of a prediction level for feature extraction; recursively inputting the result of the feature extraction by the convolutional layer of the prediction level into the convolutional layer of the prediction level again for feature extraction; obtaining the feature vectors of the instances within the bounding box through global average pooling of the result of the recursive input into the convolutional layer of the prediction level for feature extraction.

2. The video instance segmentation method of claim 1, wherein, The method further comprises: pre-processing each video frame image of the input video data before inputting the deep learning model.

3. The video instance segmentation method of claim 2, wherein, The extracting of the plurality of layers of first feature maps of each video frame image input based on the deep backbone network comprises: extracting the plurality of layers of first feature maps of each video frame image input based on a ResNet backbone residual network.

4. The video instance segmentation method of claim 3, wherein, The information fusion processing on the extracted plurality of layers of first feature maps based on the feature gold tower network comprises: performing top-down information fusion processing on the extracted plurality of layers of first feature maps, and performing information fusion of each layer of first feature maps with the layer of first feature maps above it.

5. The video instance segmentation method of claim 1, wherein, The extracting of the feature vectors of the instances within the bounding boxes of different video frame images using the fully convolutional network comprises: extracting the feature vectors of the instances within the bounding boxes of different video frame images using a fully convolutional recurrent network structure.

6. The video instance segmentation method of claim 1, wherein, The method further comprises: pre-training the deep learning model in a loop until the deep learning model converges.

7. The video instance segmentation method of claim 6, wherein, The training of the deep learning model in a loop comprises: learning and updating the weights of the deep learning network using a stochastic gradient descent optimization algorithm until the deep learning model converges.

8. A video instance segmentation apparatus, characterized by, The video instance segmentation device comprises a deep backbone network, a feature gold tower network, an RPN network, a bounding box detection branch, a mask prediction branch, and a label prediction branch; the deep backbone network is configured to extract a plurality of layers of first feature maps of each video frame image input; the feature gold tower network is configured to perform information fusion processing on the extracted plurality of layers of first feature maps to obtain a plurality of layers of second feature maps after information fusion; The RPN network is configured to process the multi-layer second feature map to obtain a plurality of candidate regions. The bounding box detection branch and the mask prediction branch are configured to perform bounding box detection and mask prediction on each of the candidate regions to obtain a result of instance segmentation of each of the video frame images. The label prediction branch is configured to extract a feature vector of an instance in a bounding box of different video frame images by using a fully convolutional network, and perform matching of the instance according to the extracted feature vector. The extracting of the feature vector of the instance in the bounding box of the different video frame images by using the fully convolutional recurrent network structure comprises: inputting an image in the bounding box into a convolutional layer of a prediction level to perform feature extraction; recursively inputting a result of the feature extraction by using the convolutional layer of the prediction level into the convolutional layer of the prediction level to perform feature extraction again; obtaining the feature vector of the instance in the bounding box by using global average pooling on a result of the feature extraction by recursively inputting into the convolutional layer of the prediction level.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program controls a device where the computer readable storage medium is located to execute the video instance segmentation method in any one of claims 1 to 7 when running.

Images