Training methods for video object segmentation models, video object segmentation methods and devices

Abstract

This disclosure relates to a training method, a video object segmentation method, and an apparatus for a video object segmentation model. The training method includes: acquiring video samples, wherein the video samples include a first image frame and a second image frame carrying a target mask; inputting the first image frame and the target mask of the first image frame into a memory encoding network for encoding processing to obtain a first key-value pair feature; inputting the second image frame into a query encoding network for encoding processing to obtain a second key-value pair feature, wherein both the memory encoding network and the query encoding network include a multi-branch structure for extracting key features and a multi-branch structure for extracting value features; determining a predicted target mask of the second image frame based on the first key-value pair feature and the second key-value pair feature, combined with a decoding network; and adjusting the parameters of the memory encoding network, the query encoding network, and the decoding network according to the target mask of the second image frame and the predicted target mask to train a target video object segmentation model.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and in particular to a training method for a video object segmentation model, a video object segmentation method, and an apparatus. Background Technology

[0002] Video Object Segmentation (VOS) is a fundamental capability for video scene understanding and video editing, with broad application prospects in areas such as intelligent short video editing, special effects production, and short video creation. VOS technology refers to predicting pixel-level segmentation masks of a target object in subsequent image frames, given a target object mask in the initial image frames of a video sequence. With the development of deep learning technology, deep neural networks have been applied to VOS. Utilizing the high-level semantic features extracted by deep networks, target objects and backgrounds can be more accurately distinguished in complex scenes, thus greatly improving the performance of object segmentation. Deep learning-based VOS technology has therefore become one of the mainstream technologies.

[0003] Existing deep learning-based VOS technology first extracts depth features from image frames, then uses two parallel 3×3 convolutional branches to generate key and value features for subsequent segmentation. However, the key and value features obtained in this way have weak feature representation capabilities and lack good generalization performance for diverse scenarios, thus affecting the accuracy of video object segmentation algorithms. Summary of the Invention

[0004] This disclosure provides a training method for a video object segmentation model, a video object segmentation method, and an apparatus, to at least address the problem of how to improve the accuracy of video object segmentation algorithms in related technologies.

[0005] According to a first aspect of this disclosure, a training method for a video object segmentation model is provided. The video object segmentation model includes a memory encoding network, a query encoding network, and a decoding network. The training method includes: acquiring video samples, wherein the video samples include a first image frame and a second image frame carrying a target mask; inputting the first image frame and the target mask of the first image frame into the memory encoding network for encoding processing to obtain a first key-value pair feature, wherein the memory encoding network includes a first key feature extraction network and a first value feature extraction network, both of which are multi-branch structures; inputting the second image frame into the query encoding network for encoding processing to obtain a second key-value pair feature, wherein the query encoding network includes a second key feature extraction network and a second value feature extraction network, both of which are multi-branch structures; determining a predicted target mask of the second image frame based on the first key-value pair feature and the second key-value pair feature, combined with the decoding network; and adjusting the parameters of the memory encoding network, the query encoding network, and the decoding network according to the target mask of the second image frame and the predicted target mask to train a target video object segmentation model.

[0006] Optionally, after training, the first key feature extraction network, the first value feature extraction network, the second key feature extraction network, and the second value feature extraction network in the video object segmentation model are respectively adjusted into a first target network, a second target network, a third target network, and a fourth target network. After training to obtain the target video object segmentation model, the training method further includes: converting the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model into single-branch structure first transformation network, second transformation network, third transformation network, and fourth transformation network, respectively, to obtain a transformed video object segmentation model.

[0007] Optionally, the structure of the first target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the second target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the third target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; and the structure of the fourth target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches.

[0008] Optionally, the first target network, the second target network, the third target network, and the fourth target network each include at least two convolutional branches with different kernel sizes and at least one identity mapping branch.

[0009] Optionally, the step of converting the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model into single-branch structured first transformation network, second transformation network, third transformation network, and fourth transformation network respectively includes: determining any one of the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model as the current target network; converting at least one identity mapping branch in the current target network into a unit convolution branch with an identity matrix as the convolution kernel; performing transformation processing on each convolution branch in the current target network by filling the convolution kernel matrix with 0 and increasing the convolution kernel size to a preset size to obtain the transformed convolution branch corresponding to each convolution branch in the current target network; and performing calculation processing on the convolution kernel parameters of each transformed convolution branch in the current target network according to the linear operation characteristics of convolution and the structure of the current target network to obtain a single-branch structured network, which serves as the transformation network corresponding to the current target network.

[0010] Optionally, determining the estimated target mask of the second image frame based on the first key-value pair features and the second key-value pair features, in conjunction with the decoding network, includes: determining the mask features of the target mask of the second image frame based on the first key-value pair features and the second key-value pair features; inputting the mask features into the decoding network for decoding processing to obtain the estimated target mask of the second image frame.

[0011] According to a second aspect of this disclosure, a video object segmentation method is provided, the video object segmentation method comprising: acquiring a first target image frame and a second target image frame of a video to be segmented, and a target mask of the first target image frame; inputting the first target image frame, the second target image frame and the target mask of the first target image frame into a target video object segmentation model to obtain a predicted target mask of the second target image frame, wherein the target video object segmentation model is trained by a training method according to an embodiment of this disclosure.

[0012] According to a third aspect of this disclosure, a training apparatus for a video object segmentation model is provided. The video object segmentation model includes a memory encoding network, a query encoding network, and a decoding network. The training apparatus includes: an acquisition unit configured to acquire video samples, wherein the video samples include a first image frame and a second image frame carrying a target mask; a first encoding unit configured to input the first image frame and its target mask into the memory encoding network for encoding processing to obtain first key-value pair features, wherein the memory encoding network includes a first key feature extraction network and a first value feature extraction network, both of which are multi-branch structures; and a second encoding unit. A unit is configured to input the second image frame into the query coding network for encoding processing to obtain a second key-value pair feature, wherein the query coding network includes a second key feature extraction network and a second value feature extraction network, both of which are multi-branch structures; a determination unit is configured to determine a predicted target mask of the second image frame based on the first key-value pair feature and the second key-value pair feature, combined with the decoding network; and an adjustment unit is configured to adjust the parameters of the memory coding network, the query coding network, and the decoding network according to the target mask of the second image frame and the predicted target mask, in order to train a target video object segmentation model.

[0013] Optionally, after training, the first key feature extraction network, the first value feature extraction network, the second key feature extraction network, and the second value feature extraction network in the video object segmentation model are respectively adjusted into a first target network, a second target network, a third target network, and a fourth target network. The training device further includes a conversion unit, configured to convert the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model into single-branch structure first conversion network, second conversion network, third conversion network, and fourth conversion network, respectively, after training to obtain a converted video object segmentation model.

[0014] Optionally, the structure of the first target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the second target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the third target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; and the structure of the fourth target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches.

[0015] Optionally, the first target network, the second target network, the third target network, and the fourth target network each include at least two convolutional branches with different kernel sizes and at least one identity mapping branch.

[0016] Optionally, the conversion unit is further configured to: determine any one of the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model as the current target network; convert at least one identity mapping branch in the current target network into a unit convolution branch with an identity matrix as the convolution kernel; perform conversion processing on each convolution branch in the current target network by filling the convolution kernel matrix with 0 and increasing the convolution kernel size to a preset size, to obtain the converted convolution branch corresponding to each convolution branch in the current target network; and perform calculation processing on the convolution kernel parameters of each converted convolution branch in the current target network according to the linear operation characteristics of convolution and the structure of the current target network to obtain a single-branch structure network, which serves as the conversion network corresponding to the current target network.

[0017] Optionally, the determining unit is further configured to perform a process of determining the mask features of the target mask of the second image frame based on the first key-value pair features and the second key-value pair features; inputting the mask features into the decoding network for decoding processing to obtain the estimated target mask of the second image frame.

[0018] According to a fourth aspect of this disclosure, a video object segmentation apparatus is provided, the video object segmentation apparatus comprising: an acquisition unit configured to acquire a first target image frame and a second target image frame of a video to be segmented, and a target mask of the first target image frame; and a segmentation unit configured to input the first target image frame, the second target image frame, and the target mask of the first target image frame into a target video object segmentation model to obtain a predicted target mask of the second target image frame, wherein the target video object segmentation model is trained by a training method according to an embodiment of this disclosure.

[0019] According to a fifth aspect of this disclosure, an electronic device is provided, the electronic device comprising: at least one processor; at least one memory storing computer-executable instructions, wherein, when executed by the at least one processor, the computer-executable instructions cause the at least one processor to perform a training method or a video object segmentation method according to the present disclosure.

[0020] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by at least one processor, causes the at least one processor to perform a training method or a video object segmentation method according to the video object segmentation model of this disclosure.

[0021] According to a seventh aspect of this disclosure, a computer program product is provided, including computer instructions that, when executed by at least one processor, implement a training method or a video object segmentation method for a video object segmentation model according to this disclosure.

[0022] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0023] The training method, video object segmentation method, and apparatus for the video object segmentation model according to embodiments of the present disclosure enhance the representation ability of the corresponding networks by employing a multi-branch structure in the memory coding network and the query coding network to extract key-value pair features, thereby improving the model's ability to distinguish target objects and backgrounds in images, and thus improving the robustness and generalization of the algorithm, and improving the accuracy of the video object segmentation algorithm.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0026] Figure 1 A schematic diagram illustrating the structure of a video object segmentation model according to an exemplary embodiment of the present disclosure is shown.

[0027] Figure 2 This diagram illustrates the structure of the memory encoding network and the query encoding network.

[0028] Figure 3 A flowchart illustrating a training method for a video object segmentation model according to an exemplary embodiment of the present disclosure;

[0029] Figure 4 A schematic diagram illustrating the structure of a memory encoding network and a query encoding network to be trained according to an exemplary embodiment of the present disclosure is shown.

[0030] Figure 5 A flowchart illustrating a video object segmentation method according to an exemplary embodiment of the present disclosure is provided.

[0031] Figure 6 A block diagram illustrating a training apparatus for a video object segmentation model according to an exemplary embodiment of the present disclosure;

[0032] Figure 7 A block diagram illustrating a video object segmentation apparatus according to an exemplary embodiment of the present disclosure is shown.

[0033] Figure 8 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0035] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0036] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0037] Video object segmentation (VOS) is a fundamental capability for video scene understanding and video editing, with broad application prospects in areas such as intelligent short video editing, special effects production, and short video creation. VOS technology refers to predicting pixel-level segmentation masks of a target object in subsequent image frames, given a target object mask in the initial image frames of a video sequence. With the development of deep learning technology, deep neural networks have been applied to VOS. Utilizing the high-level semantic features extracted by deep networks, target objects and backgrounds can be more accurately distinguished in complex scenes, thus greatly improving the performance of object segmentation. Deep learning-based VOS technology has therefore become one of the mainstream technologies.

[0038] Existing deep learning-based VOS technology applies video object segmentation models to segment objects within the video to be segmented. (Refer to...) Figure 1 Video object segmentation models include memory encoders (corresponding to...). Figure 1 Enc M ), Query Encoder (corresponding to) Figure 1 Enc Q ), Space-time Memory (sometimes also called Space-time Memory Read) and Decoder (corresponding to Figure 1 The video to be segmented exists as a video stream containing multiple sequentially arranged image frames. Previously segmented image frames and their corresponding target object masks are stored in an external memory bank.

[0039] When predicting the target mask for the current image frame, firstly, multiple image frames and their target masks are selected from the aforementioned external repository. The selection of image frames can be varied; for example, the first image frame of the video stream and the image frame preceding the current image frame can be fixed, and then every few frames in between can be selected. Alternatively, the user can specify each frame individually. The selected image frames are denoted as memory image frames. Except for the target mask of the first image frame of the video stream, which is given, the target masks for other image frames are predicted. These memory image frames with target masks are input into a memory coding network to obtain key and value features corresponding to each memory image frame. These key and value features are concatenated by a concatenation layer to form a key feature map composed of the key features of the multiple memory image frames, and a value feature map composed of the value features of the multiple memory image frames. The key and value feature maps together constitute key-value pair features. The key features are used for addressing, while the value features store more detailed information used to generate the target mask. Simultaneously, the current image frame is input into the query coding network to obtain the key features and value features of the current image frame, forming the key-value pair features of the current image frame. It should be understood that the key-value pair features output by the memory coding network have the same dimensions as the key-value pair features output by the query coding network. The difference is that the former has an additional time dimension to represent the time of each memory image frame.

[0040] Then, in the spatiotemporal memory network, the matrix composed of the key features of multiple memory image frames output by the memory encoding network and the key features of the current image frame output by the query encoding network are multiplied by an inner product to obtain a similarity map between the current image frame and multiple memory image frames. This is equivalent to using a spatiotemporal attention mechanism to assign weights to the value features of different time and space regions. This similarity map is multiplied by the matrix composed of the value features of multiple memory image frames output by the memory encoding network as an intermediate result. This intermediate result is then concatenated with the value features of the current image frame output by the query encoding network to obtain the mask features required to predict the target mask of the current image frame.

[0041] Finally, the decoding network uses the mask features to predict the target mask of the current image frame, thus obtaining the predicted target mask for the current image frame. Specifically, features extracted from the intermediate layers of the query encoding network can be connected to the decoding network in the form of skip connections to supplement detailed information.

[0042] It should be noted that when applying this model for video object segmentation, the estimated target mask is used as the segmentation result. During model training, a loss value is calculated based on the estimated target mask and the target mask of the current image frame given in the sample data. This loss value is then used to adjust the model's parameters, thus achieving model training.

[0043] The structures of the memory encoding network and the query encoding network are as follows: Figure 2 As shown, it includes a deep learning backbone network and two parallel single-branch structures. Figure 2 The deep learning backbone network in this algorithm is ResNet50, consisting of one convolutional layer (conv1) and three residual layers (res2, res3, res4). The numbers on the left of each layer represent the number of channels, while the numbers on the right represent the resolution of the output features, which is reduced to 1 / 2, 1 / 4, etc. As the feature resolution decreases, the receptive field increases, thus enabling the extraction of higher semantic information. The single-branch structure is specifically a 3×3 convolutional branch (conv 3×3), with the numbers on both sides having the same meaning as before. When extracting key-value pair features, the deep learning backbone network first extracts deep features, then two parallel single-branch structures are created, each using a 3x3 convolutional layer to generate its corresponding key and value features. However, because only a single 3x3 convolutional layer is used to generate key and value features, the resulting features have weak representation capabilities and lack good generalization performance across diverse scenarios, thus affecting the overall performance of the video object segmentation algorithm.

[0044] The training method, video object segmentation method, and apparatus for a video object segmentation model according to exemplary embodiments of the present disclosure enhance the representational ability of the corresponding networks by employing a multi-branch structure in the memory coding network and the query coding network to extract key-value pair features, thereby improving the model's ability to distinguish target objects and backgrounds in images, and further improving the robustness and generalization of the algorithm, and enhancing the overall performance of the video object segmentation algorithm.

[0045] Below, we will refer to Figures 1 to 8 The present disclosure provides a detailed description of a method for training a video object segmentation model, a method for segmenting video objects, a training apparatus for a video object segmentation model, and a video object segmentation apparatus, all based on exemplary embodiments of the present disclosure.

[0046] Figure 3 A flowchart illustrating a training method for a video object segmentation model according to an exemplary embodiment of the present disclosure is provided. It should be understood that the training method for the video object segmentation model according to an exemplary embodiment of the present disclosure can be implemented in terminal devices such as smartphones, tablets, and personal computers (PCs), or in devices such as servers.

[0047] Reference Figure 3 In step 301, a video sample is acquired, wherein the video sample includes a first image frame and a second image frame carrying a target mask. The first image frame corresponds to the memory image frame mentioned above, and the second image frame corresponds to the current image frame mentioned above.

[0048] In step 302, the first image frame and the target mask of the first image frame are input into the memory coding network for encoding processing to obtain the first key-value pair features. The memory coding network includes a first key feature extraction network and a first value feature extraction network, both of which are multi-branch structures.

[0049] In step 303, the second image frame is input into the query coding network for encoding processing to obtain the second key-value pair features. The query coding network includes a second key feature extraction network and a second value feature extraction network, both of which are multi-branch structures.

[0050] In step 304, based on the first key-value pair features and the second key-value pair features, and in conjunction with the decoding network, the estimated target mask for the second image frame is determined.

[0051] Optionally, step 304 includes: determining the mask features of the target mask of the second image frame based on the first key-value pair features and the second key-value pair features; inputting the mask features into a decoding network for decoding processing to obtain the estimated target mask of the second image frame. The step of determining the mask features can be implemented using the spatiotemporal memory network described above, and its specific implementation process will not be elaborated here. By first determining the mask features based on the two key-value pair features, and then decoding to obtain the estimated target mask of the second image frame, the same data format can be used for calculation, ensuring feasibility and the accuracy of the obtained estimated target mask.

[0052] In step 305, the parameters of the memory coding network, query coding network, and decoding network are adjusted based on the target mask and the estimated target mask of the second image frame to train the target video object segmentation model.

[0053] Optionally, step 305 includes: determining a loss value based on the target mask and the estimated target mask of the second image frame; and adjusting the parameters of the memory encoding network, the query encoding network, and the decoding network based on the loss value to train a target video object segmentation model. The loss function used to calculate the loss value and the method for adjusting the network parameters can be common loss functions and parameter optimization algorithms in video object segmentation algorithms, and this disclosure does not limit them.

[0054] For steps 302 and 303, by employing the aforementioned multi-branch structure in the memory coding network and query coding network to extract key-value pair features, the representational ability of the corresponding networks can be enhanced, the ability of the model trained in step 305 to distinguish between target objects and backgrounds in the image can be improved, thereby enhancing the robustness and generalization of the algorithm and improving the overall performance of the video object segmentation algorithm.

[0055] It should be noted that the deep learning backbone networks of the memory encoding network and the query encoding network, in addition to employing Figure 2 In addition to ResNet50, other basic networks such as MobilenetV2 / V3 can also be used, and this disclosure does not impose any restrictions on them.

[0056] The memory encoding network and the query encoding network will be introduced in detail next.

[0057] Optionally, the first key feature extraction network, the first value feature extraction network, the second key feature extraction network, and the second value feature extraction network in the video object segmentation model, after training, are respectively adjusted to become the first target network, the second target network, the third target network, and the fourth target network. That is, the first target network, the second target network, the third target network, and the fourth target network are obtained by parameter tuning of the first key feature extraction network, the first value feature extraction network, the second key feature extraction network, and the second value feature extraction network, respectively. After step 305, the training method according to the exemplary embodiment of this disclosure further includes: converting the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model into a first transformation network, a second transformation network, a third transformation network, and a fourth transformation network with single-branch structures, respectively, to obtain a transformed video object segmentation model. By converting the multi-branch structure in the trained model into a single-branch structure, the multi-branch structure exists only during model training, simplifying the model structure used during inference (i.e., applying the model). This decouples the training and inference architectures. A locally multi-branch model is trained, and after training, this locally multi-branch model is equivalently converted into a locally single-branch model, which is then deployed and run during inference. This conversion enhances the model's representational ability without increasing computation time or memory usage due to the multi-branch structure, resulting in a more efficient algorithm and better memory conservation.

[0058] Optionally, the structure of the first target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the second target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the third target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; and the structure of the fourth target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches. In other words, the first target network, the second target network, the third target network, and the fourth target network, as multi-branch structures, can adopt a multi-branch structure composed of parallel single branches (i.e., a parallel network including at least two single branches), a multi-branch structure composed of serial single branches (i.e., a serial network including at least two single branches), or a composite multi-branch structure that simultaneously possesses parallel and serial single branches (i.e., a hybrid network including at least three single branches). This enhances the design flexibility of the multi-branch structures of the memory encoding network and the query encoding network, and meets different representation capability requirements.

[0059] Optionally, the first, second, third, and fourth target networks each include at least two convolutional branches with different kernel sizes and at least one identity mapping branch. The convolutional kernel size refers to the height and width of the convolutional kernel, i.e., the number of rows and columns in the kernel matrix. The height and width of the convolutional kernel are equal, so the kernel size is usually represented by n×n, where n represents the height and width of the convolutional kernel, for example, 3×3. An identity mapping is a mapping from a set to itself, where the input and output of the mapping are equal. By employing at least two convolutional branches with different kernel sizes and at least one identity mapping branch to construct a multi-branch structure, richer information can be extracted compared to the single-branch structures of existing memory-encoding and query-encoding networks, thereby improving the model's representational ability while balancing computational load control and representational capability enhancement during training. It should be understood that, in the same target network, these convolutional branches and identity mapping branches can be arranged in parallel, sequentially, or in a mixed manner; this disclosure does not impose any restrictions on this.

[0060] Optionally, the first target network, second target network, third target network, and fourth target network in the target video object segmentation model are respectively converted into a first transformation network, second transformation network, third transformation network, and fourth transformation network with a single branch structure, including the following steps:

[0061] First, any one of the first, second, third, and fourth target networks in the target video object segmentation model is designated as the current target network. It should be understood that for the first, second, third, and fourth target networks, single-branch structure transformations can be performed on whichever network is selected as needed; for example, a single-branch structure transformation can be performed on each network. Since each transformation targets one network, it is designated as the current target network. If at least two networks need to be transformed, they can be transformed sequentially, i.e., each network is designated as the current target network individually, or they can be transformed simultaneously, i.e., each network is designated as the current target network, and their respective transformation steps are executed in parallel. This disclosure does not impose any restrictions on this approach.

[0062] Then, at least one identity mapping branch in the current target network is converted into an identity convolution branch with the identity matrix as the convolution kernel. This step realizes the conversion from identity mapping branches to convolution branches, ensuring that each single branch in the current target network is a convolution branch, thus ensuring the consistency of branch structure type.

[0063] Next, by padding the convolution kernel matrix with zeros to increase the kernel size to a preset size, the various convolutional branches in the current target network are transformed, resulting in transformed convolutional branches corresponding to each convolutional branch in the current target network. Because a uniform preset size is used, the kernel size of each convolutional branch is consistent, ensuring that the linear operation characteristics of convolution can be utilized to further complete the network transformation. It should be understood that the unit convolutional branch obtained in the previous step will also undergo the above transformation process. By converting the identity mapping branch to a unit convolutional branch in the previous step, all branches can be transformed together in this step, making the entire transformation process from a multi-branch structure to a single-branch structure more rationally arranged. This avoids repeatedly performing this transformation step for different branches, improving transformation efficiency. It should be noted that to ensure that convolutional branches can be transformed by padding the convolution kernel matrix, the preset size must be greater than or equal to the maximum convolution kernel size of the current target network, where the maximum convolution kernel size is the largest among the convolution kernel sizes of all convolutional branches in the current target network. It should be understood that when the preset size is equal to the maximum kernel size, the convolution branch with the maximum kernel size can be directly used as its corresponding transformation convolution branch, which means that there is no need to actually perform the operation of filling the convolution kernel matrix.

[0064] Finally, based on the linear operation characteristics of convolution and the structure of the current target network, the convolution kernel parameters of each transformation convolution branch in the current target network are processed to obtain a single-branch network, which serves as the transformation network corresponding to the current target network. Since there is a linear relationship between the convolution kernel parameters, input, and output of the convolution branches, and the output of a parallel network is equal to the sum of the outputs of its individual branches, and the output of a preceding branch in a serial network is mapped to the input of a subsequent branch, the convolution kernel parameters of each single branch in a multi-branch network can be linearly processed to obtain a single-branch network with the same output as the multi-branch network, achieving an equivalent transformation.

[0065] Figure 4 A schematic diagram of the structure of a memory encoding network and a query encoding network to be trained according to a specific embodiment of the present disclosure is shown.

[0066] Reference Figure 4 In this specific embodiment, each multi-branch structure used in training includes three parallel single branches: a 3×3 convolution branch, a 1×1 convolution branch, and an identity mapping branch. In existing memory coding networks and query coding networks, the branches used to generate key features and value features are often 3×3 convolution branches. Therefore, the multi-branch structure in this specific embodiment is equivalent to adding parallel 1×1 convolution branches and identity mapping branches to the existing structure.

[0067] Corresponding to this multi-branch structure, the specific steps for converting the first target network, the second target network, the third target network, and the fourth target network into single-branch structures include: for the current target network, converting the identity mapping branch into a 1×1 convolution branch with the identity matrix as the convolution kernel; by filling the convolution kernel matrix with 0, converting all existing 1×1 convolution branches (including the original 1×1 convolution branches and the 1×1 convolution branches obtained after converting the identity mapping branch) into 3×3 convolution branches; and adding the corresponding convolution kernel parameters of the existing 3×3 convolution branches (including the original 3×3 convolution branches, the 3×3 convolution branches obtained after converting the original 1×1 convolution branches, and the 3×3 convolution branches obtained after converting the identity mapping branch) to obtain the single-branch structure converted from the multi-branch structure, i.e., the converted network corresponding to the current target network.

[0068] Specifically, the linear operation property of convolution is additivity. Assuming the kernel parameters of three 3×3 convolution branches are w1, w2, and w3, then the following holds:

[0069] conv(x,w1)+conv(x,w2)+conv(x,w3)=conv(x,w1+w2+w3).

[0070] In other words, if the kernel parameters of these three convolutional branches are added together to obtain another convolutional branch, then the sum of the outputs of the first three convolutional branches (specifically, the sum of the outputs of each channel) is equal to the output of the second convolutional branch obtained later.

[0071] Based on this, the 1×1 convolution branch can be regarded as a special 3×3 convolution branch with many zeros in the convolution kernel, while the identity mapping branch is a special 1×1 convolution branch with the identity matrix as the convolution kernel, which can also be regarded as a special 3×3 convolution branch.

[0072] It should be understood that, for the current target network being a serial network, the methods for obtaining the unit convolutional branch and transforming the convolutional branch remain unchanged. Subsequently, the linear operation properties can be used to convert the kernel parameters of multiple single-branch convolutions into equivalent kernel parameters of a single convolutional branch. As an example, assuming the kernel parameters of two 3×3 convolutional branches are a1, b1 and a2, b2 respectively, then the following holds:

[0073] conv(conv(x,a1,b1),a2,b2)=conv(x,a2*a1,a2*b1+b2).

[0074] Where conv(x,a1,b1)=a1*x+b1, so conv(conv(x,a1,b1),a2,b2)=a2*(a1*x+b1)+b2=a2*a1*x+a2*b1+b2, the equivalent single convolutional branch obtained by the transformation has convolution kernel parameters a2*a1 and a2*b1+b2.

[0075] If the current target network is a serial network consisting of multiple single branches, or if the current target network is a hybrid network, the same conversion can be performed to transform the multi-branch structure composed of multiple single branches into an equivalent single-branch structure. These are not listed one by one here.

[0076] Figure 5 A flowchart illustrating a video object segmentation method according to an exemplary embodiment of the present disclosure is provided. It should be understood that the video object segmentation method according to an exemplary embodiment of the present disclosure can be implemented in terminal devices such as smartphones, tablets, and personal computers (PCs), or in devices such as servers.

[0077] Reference Figure 5 In step 501, the first target image frame and the second target image frame of the video to be segmented, as well as the target mask of the first target image frame, are obtained.

[0078] In step 502, the first target image frame, the second target image frame, and the target mask of the first target image frame are input into the target video object segmentation model to obtain the estimated target mask of the second target image frame. The target video object segmentation model is trained using a training method for a video object segmentation model according to an exemplary embodiment of this disclosure, and therefore possesses all the beneficial technical effects of that training method, which will not be elaborated further here.

[0079] Figure 6 A block diagram of a training apparatus for a video object segmentation model according to an exemplary embodiment of the present disclosure is shown. The video object segmentation model includes a memory encoding network, a query encoding network, and a decoding network. It should be understood that the training apparatus for the video object segmentation model according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smartphone, tablet computer, or personal computer (PC) in a software, hardware, or software-hardware combination manner, or can be implemented in a device such as a server.

[0080] Reference Figure 6 The training device 600 for the video object segmentation model includes an acquisition unit 601, a first encoding unit 602, a second encoding unit 603, a determination unit 604, and an adjustment unit 605.

[0081] The acquisition unit 601 can acquire video samples, wherein the video samples include a first image frame and a second image frame carrying a target mask.

[0082] The first encoding unit 602 can input the first image frame and the target mask of the first image frame into the memory encoding network for encoding processing to obtain the first key-value pair features. The memory encoding network includes a first key feature extraction network and a first value feature extraction network, both of which are multi-branch structures.

[0083] The second encoding unit 603 can input the second image frame into the query encoding network for encoding processing to obtain the second key-value pair features. The query encoding network includes a second key feature extraction network and a second value feature extraction network, both of which are multi-branch structures.

[0084] The determining unit 604 can determine the estimated target mask of the second image frame based on the first key-value pair features and the second key-value pair features, combined with the decoding network.

[0085] Optionally, the determining unit 604 may also determine the mask features of the target mask of the second image frame based on the first key-value pair features and the second key-value pair features; input the mask features into the decoding network for decoding processing to obtain the estimated target mask of the second image frame.

[0086] The adjustment unit 605 can adjust the parameters of the memory coding network, query coding network, and decoding network according to the target mask and the estimated target mask of the second image frame, so as to train a target video object segmentation model.

[0087] Optionally, after training, the first key feature extraction network, the first value feature extraction network, the second key feature extraction network, and the second value feature extraction network in the video object segmentation model are adjusted to become the first target network, the second target network, the third target network, and the fourth target network, respectively. The training device 600 also includes a conversion unit (not shown in the figure), which can convert the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model into single-branch structure first conversion network, second conversion network, third conversion network, and fourth conversion network, respectively, after training to obtain the converted video object segmentation model.

[0088] Optionally, the structure of the first target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the second target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; the structure of the third target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; and the structure of the fourth target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches.

[0089] Optionally, the first target network, the second target network, the third target network, and the fourth target network each include at least two convolutional branches with different kernel sizes and at least one identity mapping branch.

[0090] Optionally, the conversion unit can also determine any one of the first target network, second target network, third target network, and fourth target network in the target video object segmentation model as the current target network; convert at least one identity mapping branch in the current target network into a unit convolution branch with an identity matrix as the convolution kernel; perform conversion processing on each convolution branch in the current target network by filling the convolution kernel matrix with 0 and increasing the convolution kernel size to a preset size, to obtain the converted convolution branches corresponding to each convolution branch in the current target network; and perform calculation processing on the convolution kernel parameters of each converted convolution branch in the current target network according to the linear operation characteristics of convolution and the structure of the current target network to obtain a single-branch structure network, which serves as the conversion network corresponding to the current target network.

[0091] Figure 7 A block diagram of a video object segmentation apparatus according to an exemplary embodiment of the present disclosure is shown. It should be understood that the video object segmentation apparatus according to an exemplary embodiment of the present disclosure can be implemented in a terminal device such as a smartphone, tablet computer, or personal computer (PC) in a software, hardware, or software-hardware combination manner, or in a device such as a server.

[0092] Reference Figure 7 The video object segmentation device 700 includes an acquisition unit 701 and a segmentation unit 702.

[0093] The acquisition unit 701 can acquire the first target image frame and the second target image frame of the video to be segmented, as well as the target mask of the first target image frame.

[0094] The segmentation unit 702 can input the first target image frame, the second target image frame, and the target mask of the first target image frame into the target video object segmentation model to obtain the estimated target mask of the second target image frame. The target video object segmentation model is trained using a video segmentation model training method according to an exemplary embodiment of this disclosure.

[0095] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0096] Figure 8 This is a block diagram of an electronic device according to exemplary embodiments of the present disclosure.

[0097] Reference Figure 8 The electronic device 800 includes at least one memory 801 and at least one processor 802. The at least one memory 801 stores a set of computer-executable instructions. When the set of computer-executable instructions is executed by the at least one processor 802, a training method or a video object segmentation method of a video object segmentation model according to an exemplary embodiment of the present disclosure is executed.

[0098] As an example, electronic device 800 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 800 is not necessarily a single electronic device, but may be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 800 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0099] In electronic device 800, processor 802 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0100] The processor 802 can execute instructions or code stored in the memory 801, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0101] The memory 801 may be integrated with the processor 802, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 801 may include a separate device, such as an external disk drive, a storage array, or other storage device that can be used by any database system. The memory 801 and the processor 802 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 802 to read files stored in the memory.

[0102] In addition, the electronic device 800 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 800 can be interconnected via a bus and / or network.

[0103] According to exemplary embodiments of the present disclosure, a computer-readable storage medium may also be provided, which, when executed by at least one processor, causes the at least one processor to perform a training method or a video object segmentation method for a video object segmentation model according to exemplary embodiments of the present disclosure. Examples of computer-readable storage media herein include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0104] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, the computer program product including computer instructions, which, when executed by at least one processor, cause at least one processor to perform a training method for a video object segmentation model or a video object segmentation method according to exemplary embodiments of the present disclosure.

[0105] The training method, video object segmentation method, and apparatus for a video object segmentation model according to exemplary embodiments of the present disclosure enhance the representational ability of the corresponding networks by employing a multi-branch structure in the memory coding network and the query coding network to extract key-value pair features, thereby improving the model's ability to distinguish target objects and backgrounds in images, and further improving the robustness and generalization of the algorithm, and increasing the accuracy of the video object segmentation algorithm.

[0106] Furthermore, by converting the aforementioned multi-branch structure in the trained model into a single-branch structure, the multi-branch structure exists only during model training, simplifying the model structure used during inference. This decouples the training and inference architectures. A locally multi-branch model is trained, and after training, this locally multi-branch model is equivalently converted into a locally single-branch model, which is then deployed and run during inference. This conversion enhances the model's representational ability without increasing computation time or memory usage due to the multi-branch structure, resulting in a more efficient algorithm and better memory conservation.

[0107] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0108] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a video object segmentation model, characterized in that, The video object segmentation model includes a memory encoding network, a query encoding network, and a decoding network, and the training method includes: Acquire video samples, wherein the video samples include a first image frame and a second image frame carrying a target mask; The first image frame and the target mask of the first image frame are input into the memory coding network for encoding processing to obtain the first key-value pair features. The memory coding network includes a first deep learning backbone network for extracting the depth features of the first image frame, and a first key feature extraction network and a first value feature extraction network connected in parallel after the first deep learning backbone network. Both the first key feature extraction network and the first value feature extraction network are multi-branch structures. The second image frame is input into the query coding network for encoding processing to obtain the second key-value pair features. The query coding network includes a second deep learning backbone network for extracting the depth features of the second image frame, and a second key feature extraction network and a second value feature extraction network connected in parallel after the second deep learning backbone network. Both the second key feature extraction network and the second value feature extraction network are multi-branch structures. Based on the first key-value pair features and the second key-value pair features, and in conjunction with the decoding network, the estimated target mask of the second image frame is determined; Based on the target mask of the second image frame and the estimated target mask, the parameters of the memory coding network, the query coding network, and the decoding network are adjusted to train a target video object segmentation model.

2. The training method as described in claim 1, characterized in that, After training, the first key feature extraction network, the first value feature extraction network, the second key feature extraction network, and the second value feature extraction network in the video object segmentation model are respectively adjusted to become a first target network, a second target network, a third target network, and a fourth target network. The training method further includes, after training the target video object segmentation model, the following steps: The first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model are respectively converted into a first transformation network, a second transformation network, a third transformation network, and a fourth transformation network with a single branch structure to obtain the transformed video object segmentation model.

3. The training method as described in claim 2, characterized in that, The structure of the first target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; The structure of the second target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; The structure of the third target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches; The structure of the fourth target network is one of the following: a parallel network including at least two single branches, a serial network including at least two single branches, or a hybrid network including at least three single branches.

4. The training method as described in claim 2, characterized in that, The first target network, the second target network, the third target network, and the fourth target network each include at least two convolutional branches with different kernel sizes and at least one identity mapping branch.

5. The training method as described in claim 4, characterized in that, The step of converting the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model into single-branch transformation networks includes: Any one of the first target network, the second target network, the third target network, and the fourth target network in the target video object segmentation model is determined as the current target network; All at least one identity mapping branch in the current target network are converted into identity convolution branches with the identity matrix as the convolution kernel; By filling the matrix of convolution kernels with 0s and increasing the size of the convolution kernels to a preset size, the convolution branches in the current target network are transformed to obtain the transformed convolution branches corresponding to each convolution branch in the current target network. Based on the linear operation characteristics of convolution and the structure of the current target network, the convolution kernel parameters of each of the transformation convolution branches in the current target network are processed to obtain a single-branch network, which serves as the transformation network corresponding to the current target network.

6. The training method as described in claim 1, characterized in that, The step of determining the estimated target mask of the second image frame based on the first key-value pair features and the second key-value pair features, combined with the decoding network, includes: Based on the first key-value pair features and the second key-value pair features, the mask features of the target mask of the second image frame are determined; The mask features are input into the decoding network for decoding to obtain the estimated target mask of the second image frame.

7. A video object segmentation method, characterized in that, The video object segmentation method includes: Obtain the first target image frame and the second target image frame of the video to be segmented, as well as the target mask of the first target image frame; The first target image frame, the second target image frame, and the target mask of the first target image frame are input into the target video object segmentation model to obtain the estimated target mask of the second target image frame. The target video object segmentation model is obtained by training the training method described in any one of claims 1 to 6.

8. A training device for a video object segmentation model, characterized in that, The video object segmentation model includes a memory coding network, a query coding network, and a decoding network, and the training device includes: The acquisition unit is configured to acquire video samples, wherein the video samples include a first image frame and a second image frame carrying a target mask; The first encoding unit is configured to perform encoding processing by inputting the first image frame and the target mask of the first image frame into the memory encoding network to obtain the first key-value pair features. The memory encoding network includes a first deep learning backbone network for extracting the depth features of the first image frame, and a first key feature extraction network and a first value feature extraction network connected in parallel after the first deep learning backbone network. Both the first key feature extraction network and the first value feature extraction network are multi-branch structures. The second encoding unit is configured to perform encoding processing on the second image frame input into the query encoding network to obtain the second key-value pair features. The query encoding network includes a second deep learning backbone network for extracting the depth features of the second image frame, and a second key feature extraction network and a second value feature extraction network connected in parallel after the second deep learning backbone network. Both the second key feature extraction network and the second value feature extraction network are multi-branch structures. The determining unit is configured to perform the task of determining the estimated target mask of the second image frame based on the first key-value pair features and the second key-value pair features, in conjunction with the decoding network; The adjustment unit is configured to perform adjustments to the parameters of the memory coding network, the query coding network, and the decoding network based on the target mask of the second image frame and the estimated target mask, in order to train a target video object segmentation model.

9. A video object segmentation device, characterized in that, The video object segmentation device includes: The acquisition unit is configured to acquire a first target image frame and a second target image frame of the video to be segmented, as well as a target mask of the first target image frame; The segmentation unit is configured to perform the following steps: inputting the first target image frame, the second target image frame, and the target mask of the first target image frame into the target video object segmentation model, to obtain the estimated target mask of the second target image frame. The target video object segmentation model is obtained by training the training method described in any one of claims 1 to 6.

10. An electronic device, characterized in that, include: At least one processor; At least one memory that stores computer-executable instructions. When the computer-executable instructions are executed by the at least one processor, the at least one processor causes the at least one processor to perform the training method of the video object segmentation model as described in any one of claims 1 to 6 or the video object segmentation method as described in claim 7.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor causes the at least one processor to perform the training method of the video object segmentation model as claimed in any one of claims 1 to 6 or the video object segmentation method as claimed in claim 7.

Images