Video object segmentation method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a video object segmentation method and device, electronic equipment and storage medium. The method comprises: obtaining a to-be-predicted video frame, and extracting a multi-scale image feature corresponding to the to-be-predicted video frame; performing inverse sub-pixel transformation on a first mask image to obtain at least two target mask images at different scales; fusing image features corresponding to each target mask image and video image features at at least two different scales in the multi-scale image features to obtain fused image features; generating region segmentation features corresponding to the first mask image according to the fused image features, and generating region segmentation features corresponding to the to-be-predicted video frame according to the multi-scale image features; and predicting a segmentation region corresponding to a target object in the to-be-predicted video frame according to the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the to-be-predicted video frame. The present disclosure can improve the effect of video object segmentation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a video object segmentation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Video object segmentation (VOS) is a fundamental technology for video scene understanding and video editing. This technology has broad application prospects in areas such as intelligent short video editing, special effects production, and short video creation.

[0003] In the process of segmenting objects in videos, related technologies often involve performing multiple convolution operations on a mask image and using the results of these multiple convolution operations to segment objects in the video frame to be predicted. However, information loss often occurs during the multiple convolution operations on the mask image, making it impossible to accurately segment the target objects in the video frame and hindering the improvement of video object segmentation results.

[0004] Therefore, the related technologies suffer from poor video object segmentation results. Summary of the Invention

[0005] This disclosure provides a video object segmentation method, apparatus, electronic device, and storage medium to at least solve the problem of poor video object segmentation results in related technologies. The technical solution of this disclosure is as follows:

[0006] According to a first aspect of the present disclosure, a video object segmentation method is provided, comprising:

[0007] Obtain the video frame to be predicted and extract the multi-scale image features corresponding to the video frame to be predicted;

[0008] An inverse subpixel transform is performed on the first mask image to obtain target mask images at at least two different scales; the first mask image is used to characterize the segmented region corresponding to the target object in the target video frame, and the target video frame and the video frame to be predicted belong to the same video.

[0009] By fusing the image features corresponding to each of the target mask images with video image features at least two different scales from the multi-scale image features, a fused image feature is obtained;

[0010] Based on the fused image features, region segmentation features corresponding to the first mask image are generated, and based on the multi-scale image features, region segmentation features corresponding to the video frame to be predicted are generated.

[0011] Based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted, the segmentation region corresponding to the target object in the video frame to be predicted is predicted.

[0012] In one possible implementation, extracting the multi-scale image features corresponding to the video frame to be predicted includes: performing N feature extraction processes on the video frame to be predicted to obtain N video image features at different scales, which are used as the multi-scale image features; wherein, the first video image feature is the result of performing feature extraction processing on the video frame to be predicted; the Nth video image feature is the result of performing feature extraction processing on the (N-1)th video image feature; and N is greater than or equal to 2.

[0013] In one possible implementation, when the N video image features at different scales are arranged in descending order of scale, the step of performing at least one inverse subpixel transformation on the first mask image to obtain at least two target mask images at different scales includes: performing N-1 inverse subpixel transformations on the first mask image to obtain N-1 transformed mask images at different scales, and using the first mask image and the N-1 transformed mask images as N target mask images; wherein the first target mask image is the first mask image, and the Nth target mask image is the N-1th transformed mask image.

[0014] In one possible implementation, fusing the image features corresponding to each of the target mask images with video image features at at least two different scales among the multi-scale image features to obtain fused image features includes: performing N fusion operations sequentially based on the image features corresponding to each of the target mask images and the N different scale video image features to obtain N feature fusion results, and using the Nth feature fusion result as the fused image feature; wherein, the first feature fusion result is the result obtained by fusing the image features corresponding to the first target mask image with the first video image feature; the Nth feature fusion result is the result obtained by fusing the (N-1)th feature fusion result, the image features corresponding to the Nth target mask image, and the Nth video image feature.

[0015] In one possible implementation, fusing the image features corresponding to each of the target mask images with video image features at at least two different scales among the multi-scale image features to obtain fused image features includes: performing a convolution operation on the first target mask image to obtain an image convolution result, and determining the activation result corresponding to the image convolution result through a preset activation function as the image feature corresponding to the first target mask image; the image feature corresponding to the first target mask image corresponds to a first resolution; the first resolution is the resolution corresponding to the first video image feature; and performing a fusion operation on the image feature corresponding to the first target mask image and the first video image feature to obtain the first feature fusion result.

[0016] In one possible implementation, fusing the image features corresponding to each of the target mask images with video image features at at least two different scales among the multi-scale image features to obtain fused image features includes: performing a convolution operation on the (N-1)th feature fusion result to obtain the convolution result of the feature fusion result, and determining the activation result corresponding to the convolution result of the feature fusion result through a preset activation function, as the (N-1)th fused feature; the (N-1)th fused feature corresponds to a second resolution; the second resolution is the resolution corresponding to the Nth video image feature; extracting the image features corresponding to the Nth target mask image; the image features corresponding to the Nth target mask image correspond to the second resolution; and performing a fusion operation on the (N-1)th fused feature, the image features corresponding to the Nth target mask image, and the Nth video image feature to obtain the Nth feature fusion result.

[0017] In one possible implementation, the region segmentation feature is a key-value pair feature, which includes mask addressing information and mask generation information. The step of predicting the segmented region corresponding to the target object in the video frame to be predicted based on the region segmentation feature corresponding to the first mask image and the region segmentation feature corresponding to the video frame to be predicted includes: adjusting the mask generation information corresponding to the first mask image based on the similarity between the mask addressing information corresponding to the first mask image and the mask addressing information corresponding to the video frame to be predicted, to obtain adjusted mask generation information; decoding the fusion information between the adjusted mask generation information and the mask generation information corresponding to the video frame to be predicted to obtain a second mask image; the second mask image is used to characterize the segmented region corresponding to the target object in the video frame to be predicted.

[0018] In one possible implementation, generating the region segmentation features corresponding to the video frame to be predicted based on the multi-scale image features includes: performing a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolutional features with a first number of channels, which serve as mask addressing information corresponding to the video frame to be predicted; performing a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolutional features with a second number of channels, which serve as mask generation information corresponding to the video frame to be predicted; wherein the first number of channels is less than the second number of channels.

[0019] In one possible implementation, generating region segmentation features corresponding to the first mask image based on the fused image features includes: performing a convolution operation on the fused image features to obtain a convolution operation result, and determining a convolution feature activation result corresponding to the convolution operation result through a preset activation function; performing a convolution operation on the convolution feature activation result to obtain a convolution feature with a third number of channels, which serves as mask addressing information corresponding to the first mask image; and performing a convolution operation on the convolution feature activation result to obtain a convolution feature with a fourth number of channels, which serves as mask generation information corresponding to the first mask image; wherein the third number of channels is less than the fourth number of channels.

[0020] According to a second aspect of the present disclosure, a video object segmentation apparatus is provided, comprising:

[0021] The acquisition unit is configured to acquire the video frame to be predicted and extract the multi-scale image features corresponding to the video frame to be predicted.

[0022] The transformation unit is configured to perform inverse subpixel transformation on the first mask image to obtain target mask images at at least two different scales; the first mask image is used to characterize the segmented region corresponding to the target object in the target video frame, and the target video frame and the video frame to be predicted belong to the same video.

[0023] The fusion unit is configured to perform the fusion of image features corresponding to each of the target mask images with video image features at at least two different scales among the multi-scale image features to obtain fused image features;

[0024] The generation unit is configured to perform the following operations: generating region segmentation features corresponding to the first mask image based on the fused image features, and generating region segmentation features corresponding to the video frame to be predicted based on the multi-scale image features.

[0025] The segmentation unit is configured to perform the task of predicting the segmented region corresponding to the target object in the video frame to be predicted, based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted.

[0026] In one possible implementation, the acquisition unit is specifically configured to perform N feature extraction processes on the video frame to be predicted to obtain N video image features at different scales, which are used as the multi-scale image features; wherein the first video image feature is the result of feature extraction processing on the video frame to be predicted; the Nth video image feature is the result of feature extraction processing on the (N-1)th video image feature; and N is greater than or equal to 2.

[0027] In one possible implementation, when the N video image features of different scales are arranged in descending order of scale, the transformation unit is specifically configured to perform N-1 inverse subpixel transformations on the first mask image to obtain N-1 transformed mask images of different scales, and use the first mask image and the N-1 transformed mask images as N target mask images; wherein the first target mask image is the first mask image, and the Nth target mask image is the N-1th transformed mask image.

[0028] In one possible implementation, the fusion unit is specifically configured to perform N fusion operations sequentially based on the image features corresponding to each of the target mask images and the N video image features of different scales, to obtain N feature fusion results, and to use the Nth feature fusion result as the fused image feature; wherein, the first feature fusion result is the result obtained by fusing the image features corresponding to the first target mask image with the first video image feature; and the Nth feature fusion result is the result obtained by fusing the (N-1)th feature fusion result, the image features corresponding to the Nth target mask image, and the Nth video image feature.

[0029] In one possible implementation, the fusion unit is specifically configured to perform a convolution operation on the first target mask image to obtain an image convolution result, and determine the activation result corresponding to the image convolution result through a preset activation function as the image feature corresponding to the first target mask image; the image feature corresponding to the first target mask image corresponds to a first resolution; the first resolution is the resolution corresponding to the first video image feature; and the image feature corresponding to the first target mask image and the first video image feature are fused to obtain the first feature fusion result.

[0030] In one possible implementation, the fusion unit is specifically configured to perform a convolution operation on the (N-1)th feature fusion result to obtain the convolution result of the feature fusion result, and determine the activation result corresponding to the convolution result of the feature fusion result through a preset activation function, as the (N-1)th fusion feature; the (N-1)th fusion feature corresponds to a second resolution; the second resolution is the resolution corresponding to the Nth video image feature; extract the image features corresponding to the Nth target mask image; the image features corresponding to the Nth target mask image correspond to the second resolution; and perform a fusion operation on the (N-1)th fusion feature, the image features corresponding to the Nth target mask image, and the Nth video image feature to obtain the Nth feature fusion result.

[0031] In one possible implementation, the region segmentation feature is a key-value pair feature, which includes mask addressing information and mask generation information. Specifically, the segmentation unit is configured to perform the following operations: adjust the mask generation information corresponding to the first mask image based on the similarity between the mask addressing information corresponding to the first mask image and the mask addressing information corresponding to the video frame to be predicted, to obtain adjusted mask generation information; decode the fusion information between the adjusted mask generation information and the mask generation information corresponding to the video frame to be predicted to obtain a second mask image; the second mask image is used to characterize the segmented region corresponding to the target object in the video frame to be predicted.

[0032] In one possible implementation, the generation unit is specifically configured to perform a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolutional features with a first number of channels, which serve as mask addressing information corresponding to the video frame to be predicted; and to perform a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolutional features with a second number of channels, which serve as mask generation information corresponding to the video frame to be predicted; wherein the first number of channels is less than the second number of channels.

[0033] In one possible implementation, the generation unit is specifically configured to perform a convolution operation on the fused image features to obtain a convolution operation result, and determine the convolution feature activation result corresponding to the convolution operation result through a preset activation function; perform a convolution operation on the convolution feature activation result to obtain a convolution feature with a third number of channels, which serves as the mask addressing information corresponding to the first mask image; and perform a convolution operation on the convolution feature activation result to obtain a convolution feature with a fourth number of channels, which serves as the mask generation information corresponding to the first mask image; wherein the third number of channels is less than the fourth number of channels.

[0034] According to a third aspect of the present disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the video object segmentation method as described in the first aspect or any possible implementation thereof.

[0035] According to a fourth aspect of the present disclosure, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the video object segmentation method as described in the first aspect or any possible implementation thereof.

[0036] According to a fifth aspect of the present disclosure, a computer program product is provided, the program product including a computer program stored in a readable storage medium, wherein at least one processor of a device reads from the readable storage medium and executes the computer program, causing the device to perform the video object segmentation method described in any possible implementation of the first aspect.

[0037] The technical solution provided by the embodiments of this disclosure brings at least the following beneficial effects: By acquiring the video frame to be predicted and extracting the multi-scale image features corresponding to the video frame to be predicted; then, by performing inverse subpixel transformation on the first mask image, at least two target mask images at different scales are obtained, the first mask image being used to characterize the segmented region corresponding to the target object in the target video frame, the target video frame and the video frame to be predicted both belonging to the same video; and by fusing the image features corresponding to each target mask image with the video image features at least two different scales in the multi-scale image features to obtain fused image features; then, by generating the region segmentation features corresponding to the first mask image based on the fused image features, and by generating the region segmentation features corresponding to the video frame to be predicted based on the multi-scale image features; and then, by generating the region segmentation features corresponding to the first mask image based on the multi-scale image features; and finally, by generating the region segmentation features corresponding to the first mask image based on the multi-scale image features. The domain segmentation features and the region segmentation features corresponding to the video frame to be predicted are used to predict the segmentation region corresponding to the target object in the video frame to be predicted. In this way, by performing multi-scale inverse subpixel transformation on the first mask image, multiple target mask images with resolution scales matching the multi-scale image features corresponding to the video frame to be predicted can be obtained without losing image information. By fusing multiple target mask images with the multi-scale image features corresponding to the video frame to be predicted, the positional correlation information in the mask images can be effectively mined and integrated to accurately segment the target object in the video frame to be predicted. This effectively avoids the information loss caused by multiple convolution operations on the mask image required by related technologies, and effectively achieves accurate segmentation of the target object in the video frame, improving the video object segmentation effect.

[0038] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0040] Figure 1 This is a flowchart illustrating a video object segmentation method according to an exemplary embodiment.

[0041] Figure 2 This is a flowchart illustrating another video object segmentation method according to an exemplary embodiment.

[0042] Figure 3 This is a schematic diagram illustrating an inverse subpixel transformation according to an exemplary embodiment.

[0043] Figure 4 This is a schematic diagram illustrating a subpixel according to an exemplary embodiment.

[0044] Figure 5 This is a model framework diagram illustrating a video object segmentation method according to an exemplary embodiment.

[0045] Figure 6 This is a block diagram illustrating a video object segmentation apparatus according to an exemplary embodiment.

[0046] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0047] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0048] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0049] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0050] Figure 1 This is a flowchart illustrating a video object segmentation method according to an exemplary embodiment, such as... Figure 1 As shown, this video object segmentation method is used in electronic devices and includes the following steps.

[0051] In step S102, the video frame to be predicted is obtained, and the multi-scale image features corresponding to the video frame to be predicted are extracted.

[0052] Among them, the video frame to be predicted can refer to the video frame from which the segmented region of the target object needs to be predicted.

[0053] The target object can be an object, a person, or other object in a video frame.

[0054] Multi-scale image features can refer to image features that include at least two scales.

[0055] In practice, when it is necessary to determine the image region where the target object is located in the video frame to be predicted, the video frame to be predicted can be input to an electronic device. After the electronic device acquires the video frame to be predicted, it extracts the multi-scale image features corresponding to the video frame to be predicted. Among them, the multi-scale image features include video image features at at least two different scales.

[0056] Specifically, the electronic device can input the video frame to be predicted into a pre-trained multi-scale feature extraction model. This model extracts image features at least two scales, resulting in the multi-scale image features corresponding to the video frame to be predicted. In practical applications, the multi-scale feature extraction model can employ ResNet50 (a convolutional neural network), which the electronic device uses to extract the multi-scale image features corresponding to the video frame to be predicted.

[0057] In step S102, the first mask image is subjected to inverse subpixel transformation to obtain target mask images at at least two different scales.

[0058] The first mask image is used to characterize the segmented region corresponding to the target object in the target video frame, and the target video frame and the video frame to be predicted belong to the same video.

[0059] In practice, the electronic device can identify target video frames that belong to the same video as the video frame to be predicted, and these target video frames have a corresponding mask image in advance. This mask image is used to characterize the segmented region corresponding to the target object in the target video frame. In practical applications, the target video frame can be a memory frame.

[0060] After the electronic device acquires the mask image corresponding to the target video frame, i.e. the first mask image, the electronic device performs inverse subpixel transformation on the first mask image to obtain target mask images at at least two different scales.

[0061] Inverse subpixel transformation refers to the inverse process of subpixel transformation, which involves splitting and rearranging units on a feature into multi-channel features; where r can be a scaling factor. For ease of understanding by those skilled in the art, Figure 2 An example diagram of inverse subpixel transformation is provided. Please see [link / reference]. Figure 2 The inverse subpixel transformation can rearrange a high-resolution image or feature of size rH×rW×C into a low-resolution image or feature of size H×W×C.

[0062] In this context, subpixels refer to pixels that exist between two actual physical pixels. During camera imaging, the acquired image data is discretized. Due to the limitations of the image sensor itself, each pixel on the imaging surface only represents the nearby color. For example, if there is a 4.5µm gap between pixels on two sensory elements, macroscopically they are connected, but microscopically there are countless tiny particles between them. These pixels existing between two actual physical pixels are called "subpixels."

[0063] It should be noted that subpixels do exist, but they are simply not detected by smaller sensors; therefore, they can only be approximated in software. For the ease of understanding of those skilled in the art, Figure 3 An example is provided to illustrate a sub-pixel point; such as Figure 3 As shown, the rectangular area enclosed by every four rectangular points represents the actual pixel on the original component, while the circular points represent sub-pixel points.

[0064] In step S104, the image features corresponding to each target mask image are fused with video image features at at least two different scales from the multi-scale image features to obtain fused image features.

[0065] In a specific implementation, the electronic device acquires each target mask image of the first mask image, and then the electronic device can acquire the image features corresponding to each target mask image. Specifically, the electronic device can input each target mask image into a convolutional neural network, and extract the image features corresponding to each target mask image through the convolutional neural network. The electronic device fuses the image features corresponding to each target mask image with video image features at least two different scales in the multi-scale image features to obtain fused image features.

[0066] It should be noted that the following text will explain in detail the steps of fusing the image features corresponding to each target mask image and the video image features at least two different scales in the multi-scale image features to obtain the fused image features, which will not be elaborated on here.

[0067] In step S106, region segmentation features corresponding to the first mask image are generated based on the fused image features, and region segmentation features corresponding to the video frame to be predicted are generated based on the multi-scale image features.

[0068] In this context, region segmentation features refer to features used to determine the segmented regions of a target object within a video frame. In practical applications, region segmentation features can refer to key-value pair features. These key-value pair features include key information and value information; the key information is used for addressing, while the value information is used to generate more detailed information in the mask image.

[0069] In practical implementation, the electronic device can perform a preset convolution operation on the fused image features to obtain convolution features that meet preset conditions, which are then used as the region segmentation features corresponding to the first mask image. The electronic device can also perform preset convolution operations on multi-scale image features to obtain convolution features that meet preset conditions, which are then used as the region segmentation features corresponding to the video frame to be predicted.

[0070] In step S108, the segmentation region corresponding to the target object in the video frame to be predicted is predicted based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted.

[0071] In practice, the electronic device can predict the segmented region of the target object in the video frame to be predicted based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted.

[0072] Specifically, the electronic device can adjust the region segmentation features corresponding to the video frame to be predicted based on the similarity between the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted, thereby obtaining the adjusted region segmentation features. The electronic device then inputs the adjusted region segmentation features into a pre-trained decoding network, and decodes the adjusted region segmentation features through the pre-trained decoding network to obtain a mask image mask used to represent the segmentation region corresponding to the target object in the video frame to be predicted, thereby realizing the prediction of the segmentation region corresponding to the target object in the video frame to be predicted.

[0073] In the aforementioned video object segmentation method, the following steps are taken: First, a video frame to be predicted is acquired, and multi-scale image features corresponding to that video frame are extracted. Then, an inverse subpixel transformation is performed on a first mask image to obtain at least two target mask images at different scales. This first mask image is used to characterize the segmented region corresponding to the target object in the target video frame. Both the target video frame and the video frame to be predicted belong to the same video. Next, the image features corresponding to each target mask image are fused with the video image features at least two different scales from the multi-scale image features to obtain fused image features. Then, based on the fused image features, region segmentation features corresponding to the first mask image are generated, and based on the multi-scale image features, region segmentation features corresponding to the video frame to be predicted are generated. Finally, based on the region segmentation features corresponding to the first mask image and... The region segmentation features corresponding to the video frame to be predicted are used to predict the segmentation region corresponding to the target object in the video frame to be predicted. Thus, by performing multi-scale inverse subpixel transformation on the first mask image, multiple target mask images with resolution scales matching the multi-scale image features corresponding to the video frame to be predicted can be obtained without losing image information. By fusing multiple target mask images with the multi-scale image features corresponding to the video frame to be predicted, the positional association information in the mask images can be effectively mined and integrated to accurately segment the target object in the video frame to be predicted. This effectively avoids the information loss that occurs when multiple convolution operations are performed on the mask image in related technologies, and effectively achieves accurate segmentation of the target object in the video frame, improving the video object segmentation effect.

[0074] In one exemplary embodiment, extracting multi-scale image features corresponding to the video frame to be predicted includes: performing feature extraction processing on the video frame to be predicted N times to obtain N video image features of different scales, which are used as multi-scale image features.

[0075] The first video image feature is the result of feature extraction processing on the video frame to be predicted; the Nth video image feature is the result of feature extraction processing on the (N-1)th video image feature; N is greater than or equal to 2.

[0076] In specific implementation, during the process of extracting multi-scale image features corresponding to the video frame to be predicted, the electronic device can perform feature extraction processing N times on the video frame to be predicted, obtaining N video image features of different scales. Specifically, the electronic device can perform feature extraction processing on the video frame to be predicted to obtain the first video image feature; the electronic device can perform feature extraction processing on the first video image feature to obtain the second video image feature; and so on, until the electronic device performs feature extraction processing on the (N-1)th video image feature to obtain the Nth video image feature; the electronic device uses the first to Nth video image features as multi-scale image features. The resolution scale of the (N-1)th video image feature is larger than that of the Nth video image feature; and the resolution of the (N-1)th video image feature can be twice the resolution of the Nth video image feature.

[0077] For example, assuming N equals 4, the electronic device can input the video frame to be predicted into a pre-trained multi-scale feature extraction model. This multi-scale feature extraction model may include convolutional layers, a first residual network layer, a second residual network layer, a third residual network layer, and a fourth residual network layer. The electronic device can input the video frame to be predicted into the convolutional layer to obtain a first video image feature at 1 / 2 resolution of the video frame to be predicted. The electronic device can input the first video image feature into the first residual network layer to obtain a second video image feature at 1 / 4 resolution of the video frame to be predicted. The electronic device can input the second video image feature into the second residual network layer to obtain a third video image feature at 1 / 8 resolution of the video frame to be predicted. The electronic device can input the third video image feature into the third residual network layer to obtain a fourth video image feature at 1 / 16 resolution of the video frame to be predicted. The electronic device uses the above-mentioned first, second, third, and fourth video image features as the multi-scale image features corresponding to the video frame to be predicted.

[0078] The technical solution of this embodiment obtains N video image features at different scales by performing feature extraction processing on the video frame to be predicted N times, thereby effectively extracting the multi-scale image features corresponding to the video frame to be predicted at different resolution scales.

[0079] In an exemplary embodiment, when N video image features of different scales are arranged in descending order of scale, at least one inverse subpixel transformation is performed on the first mask image to obtain at least two target mask images at different scales. This includes: performing N-1 inverse subpixel transformations on the first mask image to obtain N-1 transformed mask images at different scales, and using the first mask image and the N-1 transformed mask images as N target mask images.

[0080] In this diagram, the first target mask image is the first mask image, and the Nth target mask image is the (N-1)th transformed mask image. In practical applications, the scaling factor corresponding to the Nth target mask image can be twice the scaling factor corresponding to the (N-1)th target mask image.

[0081] In specific implementation, when the video image features of N different scales are arranged in descending order of scale, the electronic device can perform N-1 inverse subpixel transformations on the first mask image to obtain at least two target mask images at different scales during the process of performing at least one inverse subpixel transformation on the first mask image to obtain N-1 transformed mask images at different scales, and use the first mask image and the N-1 transformed mask images as N target mask images.

[0082] For example, suppose an electronic device performs feature extraction processing at four different scales on the video frame to be predicted. The electronic device can then perform three inverse subpixel transformations on the first mask image to obtain three transformed mask images at different scales. Specifically, the electronic device can perform an inverse subpixel transformation on the first mask image by a first scaling factor R1 to obtain the first transformed mask image; the electronic device can perform an inverse subpixel transformation on the first mask image by a second scaling factor R2 to obtain the second transformed mask image; and the electronic device can perform an inverse subpixel transformation on the first mask image by a third scaling factor R3 to obtain the third transformed mask image. The electronic device uses the first mask image, the first transformed mask image, the second transformed mask image, and the third transformed mask image as the target mask image.

[0083] In this embodiment, when N video image features of different scales are arranged in descending order of scale, N-1 inverse subpixel transformations are performed on the first mask image to obtain N-1 transformed mask images of different scales. The first mask image and the N-1 transformed mask images are then used as N target mask images, so that the number of target mask images matches the number of multi-scale image features.

[0084] In an exemplary embodiment, fusing the image features corresponding to each target mask image with video image features at least two different scales among the multi-scale image features to obtain fused image features includes: performing N fusion operations sequentially based on the image features corresponding to each target mask image and N different scale video image features to obtain N feature fusion results, and using the Nth feature fusion result as the fused image feature.

[0085] The first feature fusion result is obtained by fusing the image features corresponding to the first target mask image with the first video image features; the Nth feature fusion result is obtained by fusing the (N-1)th feature fusion result, the image features corresponding to the Nth target mask image, and the Nth video image features.

[0086] In the specific implementation, during the process of fusing the image features corresponding to each target mask image with video image features at least two different scales in the multi-scale image features to obtain the fused image features, the electronic device can sequentially perform N fusion operations based on the image features corresponding to each target mask image and N different scale video image features to obtain N feature fusion results, and use the Nth feature fusion result as the fused image feature.

[0087] Specifically, the electronic device can fuse the image features corresponding to the first target mask image with the first video image features to obtain the first feature fusion result; the electronic device can then fuse the first feature fusion result, the image features corresponding to the second target mask image, and the second video image features to obtain the second feature fusion result; and so on, until the electronic device fuses the (N-1)th feature fusion result, the image features corresponding to the Nth target mask image, and the Nth video image features to obtain the Nth feature fusion result. The electronic device uses this Nth feature fusion result as the fused image feature.

[0088] The technical solution of this embodiment, by performing N fusion operations sequentially based on the image features corresponding to each target mask image and N video image features at different scales, obtains N feature fusion results. This can achieve the fusion of video image features at different scales with target mask images at different scales, enabling the video object prediction process to fully extract the positional association information in the mask, such as semantic and shape information, and achieve accurate segmentation of target objects in the video frame to be predicted, thereby improving the video object segmentation effect.

[0089] In an exemplary embodiment, fusing image features corresponding to each target mask image with video image features at at least two different scales in multi-scale image features to obtain fused image features includes: performing a convolution operation on the first target mask image to obtain an image convolution result, and determining the activation result corresponding to the image convolution result through a preset activation function as the image feature corresponding to the first target mask image; and performing a fusion operation on the image feature corresponding to the first target mask image and the first video image feature to obtain a first feature fusion result.

[0090] The image features corresponding to the first target mask image correspond to the first resolution; the first resolution is the resolution corresponding to the first video image feature.

[0091] In specific implementation, during the process of fusing the image features corresponding to each target mask image with video image features at least two different scales in the multi-scale image features to obtain fused image features, the electronic device can perform a convolution operation on the first target mask image to obtain the image convolution result, and determine the activation result corresponding to the image convolution result through a preset activation function, which is used as the image feature corresponding to the first target mask image; and perform a fusion operation on the image feature corresponding to the first target mask image and the first video image feature to obtain the first feature fusion result.

[0092] Continuing the previous example, the electronic device can input the first mask image into a convolutional neural network. The parameters of this convolutional neural network are a 7×7 kernel and a stride of 2. The electronic device performs a convolution operation on the first mask image through this convolutional neural network to obtain the image convolution result at half the resolution of the first mask image. The activation result corresponding to the image convolution result is determined by a preset activation function PReLU, which is used as the image feature corresponding to the first target mask image. The electronic device then performs a fusion operation on the image feature corresponding to the first target mask image and the first video image feature to obtain the first feature fusion result.

[0093] The technical solution of this embodiment can fuse video image features at different scales with target mask images at different scales, so that the video object prediction process can fully explore the positional correlation information in the mask, achieve accurate segmentation of target objects in the video frame to be predicted, and improve the video object segmentation effect.

[0094] In an exemplary embodiment, fusing image features corresponding to each target mask image with video image features at least two different scales from multi-scale image features to obtain fused image features includes: performing a convolution operation on the (N-1)th feature fusion result to obtain a convolution result of the feature fusion result, and determining the activation result corresponding to the convolution result of the feature fusion result through a preset activation function as the (N-1)th fused feature; extracting image features corresponding to the Nth target mask image; the image features corresponding to the Nth target mask image correspond to a second resolution; and performing a fusion operation on the (N-1)th fused feature, the image features corresponding to the Nth target mask image, and the Nth video image features to obtain the Nth feature fusion result.

[0095] The (N-1)th fusion feature corresponds to the second resolution; the second resolution is the resolution corresponding to the Nth video image feature.

[0096] In specific implementation, the electronic device fuses the image features corresponding to each target mask image with video image features at least two different scales in the multi-scale image features to obtain fused image features. During this process, the electronic device can perform a convolution operation on the (N-1)th feature fusion result to obtain the convolution result of the feature fusion result, and determine the activation result corresponding to the convolution result of the feature fusion result through a preset activation function, which is used as the (N-1)th fused feature. The electronic device can then perform a fusion operation on the (N-1)th fused feature, the image features corresponding to the Nth target mask image, and the Nth video image features to obtain the Nth feature fusion result.

[0097] Continuing with the previous example, after the electronic device obtains the first feature fusion result, it can input the first feature fusion result into a convolutional neural network. The parameters of this convolutional neural network are a 3×3 kernel and a stride of 2. The electronic device performs a convolution operation on the first feature fusion result through this convolutional neural network to obtain the convolution result of the feature fusion result at 1 / 4 resolution of the first mask image. The activation result corresponding to the convolution result of the feature fusion result is determined by a preset activation function PReLU, which is used as the first fusion feature. The electronic device then performs a fusion operation on the first fusion feature, the image feature corresponding to the second target mask image, and the second video image feature to obtain the second feature fusion result.

[0098] The technical solution of this embodiment fuses multiple target mask images with multi-scale image features corresponding to the video frame to be predicted, thereby effectively mining and integrating the positional association information in the mask images to accurately segment the target objects in the video frame to be predicted. This effectively avoids the information loss that occurs when related technologies require multiple convolution operations on the mask images, thus effectively achieving accurate segmentation of target objects in the video frame and improving the video object segmentation effect.

[0099] In an exemplary embodiment, predicting the segmented region corresponding to the target object in the video frame to be predicted based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted includes: adjusting the mask generation information corresponding to the first mask image based on the similarity between the mask addressing information corresponding to the first mask image and the mask addressing information corresponding to the video frame to be predicted, to obtain adjusted mask generation information; and performing a decoding operation on the fusion information between the adjusted mask generation information and the mask generation information corresponding to the video frame to be predicted, to obtain a second mask image.

[0100] The second mask image is used to characterize the segmented region corresponding to the target object in the video frame to be predicted.

[0101] Among them, the region segmentation features are key-value pair features, which include mask addressing information and mask generation information. In practical applications, mask addressing information can also be named Key, and mask generation information can also be named Value.

[0102] In specific implementation, when the electronic device predicts the segmented region corresponding to the target object in the video frame to be predicted based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted, the electronic device can adjust the mask generation information corresponding to the first mask image based on the similarity between the mask addressing information corresponding to the first mask image and the mask addressing information corresponding to the video frame to be predicted, and obtain the adjusted mask generation information; the electronic device can perform a decoding operation on the fusion information between the adjusted mask generation information and the mask generation information corresponding to the video frame to be predicted, and obtain the second mask image.

[0103] Specifically, the electronic device can perform an inner product operation between the key information Key corresponding to the first mask image and the key information Key corresponding to the video frame to be predicted to calculate the similarity, thereby obtaining a similarity map that represents the similarity between the two key information Keys; thus, a spatiotemporal attention mechanism can be realized, which assigns weights to values ​​in different times and regions.

[0104] The electronic device then multiplies the similarity map with the value information corresponding to the first mask image to obtain the adjusted mask generation information (i.e., the Space-time Memory Read result); the electronic device concatenates the space-time memory read result with the value information corresponding to the video frame to be predicted, and sends the resulting fusion information to the pre-trained decoder; the electronic device can use the pre-trained decoder to decode the space-time memory read result, thereby restoring the second mask image used to characterize the segmentation region corresponding to the target object in the video frame to be predicted, and thus accurately predicting the segmentation region corresponding to the target object in the video frame to be predicted.

[0105] In an exemplary embodiment, generating region segmentation features corresponding to a video frame to be predicted based on multi-scale image features includes: performing a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolution features with a first number of channels, which serve as mask addressing information corresponding to the video frame to be predicted; and performing a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolution features with a second number of channels, which serve as mask generation information corresponding to the video frame to be predicted.

[0106] The number of channels in the first channel is less than the number of channels in the second channel.

[0107] In specific implementation, during the process of generating region segmentation features corresponding to the video frame to be predicted based on multi-scale image features, the electronic device can perform convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolution features with a first number of channels, which are used as mask addressing information corresponding to the video frame to be predicted; the electronic device can also perform convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolution features with a second number of channels, which are used as mask generation information corresponding to the video frame to be predicted.

[0108] Continuing with the previous example, the electronic device inputs the third video image features into the fourth residual network layer to obtain the fourth video image features at 1 / 16 resolution of the video frame to be predicted. The electronic device can then use a 3×3 convolution kernel to perform a convolution operation on the fourth video image features to obtain a convolution feature with 128 channels, which serves as the mask addressing information Key at 1 / 16 resolution of the video frame to be predicted. The electronic device then uses a 3×3 convolution kernel to perform a convolution operation on the fourth video image features to obtain a convolution feature with 512 channels, which serves as the mask generation information Value at 1 / 16 resolution of the video frame to be predicted.

[0109] The technical solution of this embodiment performs a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolutional features with a first number of channels, which serve as mask addressing information for the video frame to be predicted. Then, it performs a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolutional features with a second number of channels, which serve as mask generation information for the video frame to be predicted. This can effectively extract the region segmentation features corresponding to the video frame to be predicted from the multi-scale image features, which is convenient for the electronic device to perform video object segmentation in the future.

[0110] In an exemplary embodiment, generating region segmentation features corresponding to a first mask image based on fused image features includes: performing a convolution operation on the fused image features to obtain a convolution operation result, and determining a convolution feature activation result corresponding to the convolution operation result through a preset activation function; performing a convolution operation on the convolution feature activation result to obtain a convolution feature with a third number of channels, which serves as mask addressing information corresponding to the first mask image; and performing a convolution operation on the convolution feature activation result to obtain a convolution feature with a fourth number of channels, which serves as mask generation information corresponding to the first mask image.

[0111] The number of channels in the third channel is less than the number of channels in the fourth channel.

[0112] In specific implementation, during the process of generating region segmentation features corresponding to the first mask image based on the fused image features, the electronic device can perform convolution operations on the fused image features to obtain the convolution operation result, and determine the convolution feature activation result corresponding to the convolution operation result through a preset activation function; perform convolution operations on the convolution feature activation result to obtain convolution features with a third number of channels, which serve as the mask addressing information corresponding to the first mask image; and perform convolution operations on the convolution feature activation result to obtain convolution features with a fourth number of channels, which serve as the mask generation information corresponding to the first mask image.

[0113] For example, an electronic device can input fused image features into a convolutional neural network (CNN) with parameters of a 3×3 kernel and a stride of 2. The electronic device performs a convolution operation on the fused image features through the CNN to obtain the convolution operation result, and determines the corresponding convolution feature activation result through a preset activation function PReLU. The electronic device then uses a 3×3 kernel to perform a convolution operation on the convolution feature activation result to obtain a convolution feature with 128 channels, which serves as the mask addressing information Key at 1 / 16 resolution of the first mask image. The electronic device then uses a 3×3 kernel to perform a convolution operation on the convolution feature activation result to obtain a convolution feature with 512 channels, which serves as the mask generation information Value at 1 / 16 resolution of the first mask image.

[0114] The technical solution of this embodiment obtains the convolution operation result by performing a convolution operation on the fused image features, and determines the convolution feature activation result corresponding to the convolution operation result by using a preset activation function; and performs a convolution operation on the convolution feature activation result to effectively extract the region segmentation features corresponding to the first mask image based on the fused image features.

[0115] Figure 4 This is a flowchart illustrating another video object segmentation method according to an exemplary embodiment, such as... Figure 4 As shown, this method is used in electronic devices and includes the following steps.

[0116] In step S402, the video frame to be predicted is acquired, and the video frame to be predicted is subjected to N feature extraction processes to obtain N video image features at different scales, which are used as multi-scale image features.

[0117] In step S404, a first mask image is obtained; the first mask image is used to characterize the segmented region corresponding to the target object in the target video frame, and the target video frame and the video frame to be predicted belong to the same video.

[0118] In step S406, the first mask image is subjected to N-1 inverse subpixel transformations to obtain N-1 transformed mask images of different scales, and the first mask image and the N-1 transformed mask images are used as N target mask images.

[0119] In step S408, based on the image features corresponding to each target mask image and N video image features at different scales, N fusion operations are performed sequentially to obtain N feature fusion results, and the Nth feature fusion result is used as the fused image feature.

[0120] In step S410, region segmentation features corresponding to the first mask image are generated based on the fused image features.

[0121] In step S412, region segmentation features corresponding to the video frame to be predicted are generated based on multi-scale image features; the region segmentation features are key-value pair features, which include mask addressing information and mask generation information.

[0122] In step S414, the mask generation information corresponding to the first mask image is adjusted according to the similarity between the mask addressing information corresponding to the first mask image and the mask addressing information corresponding to the video frame to be predicted, so as to obtain the adjusted mask generation information.

[0123] In step S416, the fusion information between the adjusted mask generation information and the mask generation information corresponding to the video frame to be predicted is decoded to obtain the second mask image; the second mask image is used to characterize the segmented region corresponding to the target object in the video frame to be predicted.

[0124] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a video object segmentation method described above, and will not be repeated here.

[0125] For the convenience of those skilled in the art, Figure 5 An example of a model framework diagram for a video object segmentation method is provided; such as Figure 5 As shown, the video object segmentation model includes a memory encoder (Enc). M ) and Query Encoder (Enc Q The memory encoder includes a first inverse subpixel convolution module, a second inverse subpixel convolution module, a third inverse subpixel convolution module, a first convolution module, a second convolution module, a third convolution module, a fourth convolution module, a fifth convolution module, a first convolution kernel, and a second convolution kernel; the query encoder includes a multi-scale feature extraction model, a third convolution kernel, and a fourth convolution kernel; the multi-scale feature extraction model includes a convolutional layer, a first residual network layer, a second residual network layer, and a third residual network layer.

[0126] Specifically, when it is necessary to determine the image region where the target object is located in the video frame to be predicted, the electronic device acquires the video frame to be predicted and determines the target video frame that belongs to the same video as the video frame to be predicted, and the target video frame has a corresponding mask image in advance, namely the first mask image.

[0127] For the query encoder, the electronic device inputs the video frame to be predicted into a multi-scale feature extraction model. Specifically, the electronic device can input the video frame to be predicted into a convolutional layer to obtain a first video image feature with 64 channels at half the resolution of the video frame; the electronic device can input the first video image feature into a first residual network layer to obtain a second video image feature with 256 channels at 1 / 4 the resolution of the video frame; the electronic device can input the second video image feature into a second residual network layer to obtain a third video image feature with 512 channels at 1 / 6 the resolution of the video frame; the electronic device then inputs the third video image feature into a fourth residual network layer. The network layer obtains a fourth video image feature with 1024 channels at 1 / 16 resolution of the video frame to be predicted. The electronic device inputs the fourth video image feature into the third convolution kernel, and performs a convolution operation on the fourth video image feature using a 3×3 convolution kernel to obtain a convolution feature with 128 channels, which serves as the mask addressing information Key at 1 / 16 resolution of the video frame to be predicted. The electronic device then inputs the fourth video image feature into the fourth convolution kernel, and performs a convolution operation on the fourth video image feature using a 3×3 convolution kernel to obtain a convolution feature with 512 channels, which serves as the mask generation information Value at 1 / 16 resolution of the video frame to be predicted.

[0128] For the memory encoder, the electronic device performs an inverse subpixel transformation on the first mask image by scaling it by a factor r = 4 using a first inverse subpixel convolution module to obtain a first transformed mask image. Then, it performs convolution, batch normalization (BN), and activation operations on the first transformed mask image using the same first inverse subpixel convolution module to obtain the corresponding image features. Similarly, it performs the same inverse subpixel transformation on the first mask image by scaling it by a factor r = 4 using a second inverse subpixel convolution module to obtain a second transformed mask image. The same process is repeated for each transformation.

[0129] The electronic device can input a first mask image into a first convolution module. The parameters of the first convolution module are a 7×7 kernel and a stride of 2. The electronic device performs a convolution operation on the first mask image through the first convolution module to obtain an image convolution result with 64 channels at half the resolution of the first mask image. The activation result corresponding to the image convolution result is determined by a preset activation function PReLU, which is used as the image feature corresponding to the first mask image. The electronic device performs a fusion operation on the image feature corresponding to the first mask image and the first video image feature to obtain a first feature fusion result.

[0130] Next, the electronic device inputs the first feature fusion result into the second convolution module. The parameters of the second convolution module are a 3×3 kernel and a stride of 2. The electronic device performs a convolution operation on the first feature fusion result through the second convolution module to obtain a convolution result with 256 channels at 1 / 4 resolution of the first mask image. The activation result corresponding to the convolution result of the feature fusion result is determined by a preset activation function such as PReLU (Parametric Rectified Linear Unit), which is used as the first fused feature. The electronic device then performs a fusion operation on the first fused feature, the image features corresponding to the first transformed mask image, and the second video image features to obtain the second feature fusion result.

[0131] Next, the electronic device inputs the second feature fusion result into the third convolution module. The parameters of the third convolution module are a 3×3 kernel and a stride of 2. The electronic device performs a convolution operation on the second feature fusion result through the third convolution module to obtain a convolution result with 512 channels at 1 / 8 resolution of the first mask image. The activation result corresponding to the convolution result of the feature fusion result is determined by a preset activation function PReLU, which is used as the second fusion feature. The electronic device then performs a fusion operation on the second fusion feature, the image feature corresponding to the second transformed mask image, and the third video image feature to obtain the third feature fusion result.

[0132] Next, the electronic device inputs the third feature fusion result into the fourth convolution module. The parameters of the fourth convolution module are a 3×3 kernel and a stride of 2. The electronic device performs a convolution operation on the third feature fusion result through the fourth convolution module to obtain a convolution result with 1024 channels at 1 / 16 resolution of the first mask image. The activation result corresponding to the convolution result is determined by a preset activation function PReLU and used as the third fusion feature. The electronic device then performs a fusion operation on the third fusion feature, the image feature corresponding to the third transformed mask image, and the fourth video image feature to obtain the fourth feature fusion result.

[0133] Next, the electronic device inputs the fourth feature fusion result into the first convolution kernel. The first convolution kernel uses a 3×3 convolution kernel to perform a convolution operation on the fourth feature fusion result, resulting in a convolution feature with 128 channels, which serves as the mask addressing information Key at 1 / 16 resolution of the first mask image. The electronic device then inputs the fourth feature fusion result into the second convolution kernel. The second convolution kernel uses a 3×3 convolution kernel to perform a convolution operation on the fourth feature fusion result, resulting in a convolution feature with 512 channels, which serves as the mask generation information Value at 1 / 16 resolution of the first mask image.

[0134] Finally, the electronic device uses the mask addressing information Key and mask generation information Value corresponding to the first mask image, as well as the mask addressing information Key and mask generation information Value corresponding to the video frame to be predicted, to predict the segmentation region corresponding to the target object in the video frame to be predicted.

[0135] It should be understood that, although Figure 1 and Figure 4 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 and Figure 4At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0136] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0137] Figure 6 This is a block diagram illustrating a video object segmentation apparatus according to an exemplary embodiment. (Refer to...) Figure 6 The device includes:

[0138] The acquisition unit 610 is configured to acquire the video frame to be predicted and extract the multi-scale image features corresponding to the video frame to be predicted.

[0139] Transformation unit 620 is configured to perform inverse subpixel transformation on a first mask image to obtain target mask images at at least two different scales; the first mask image is used to characterize the segmented region corresponding to the target object in the target video frame, and the target video frame and the video frame to be predicted belong to the same video.

[0140] The fusion unit 630 is configured to perform fusion of image features corresponding to each of the target mask images with video image features at least two different scales among the multi-scale image features to obtain fused image features;

[0141] The generation unit 640 is configured to perform the following operations: generating region segmentation features corresponding to the first mask image based on the fused image features; and generating region segmentation features corresponding to the video frame to be predicted based on the multi-scale image features.

[0142] The segmentation unit 650 is configured to perform the operation of predicting the segmentation region corresponding to the target object in the video frame to be predicted based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted.

[0143] In an exemplary embodiment, the acquisition unit 610 is specifically configured to perform N feature extraction processes on the video frame to be predicted to obtain N video image features at different scales, which are used as the multi-scale image features; wherein, the first video image feature is the result of feature extraction processing on the video frame to be predicted; the Nth video image feature is the result of feature extraction processing on the (N-1)th video image feature; and N is greater than or equal to 2.

[0144] In an exemplary embodiment, when the N video image features of different scales are arranged in descending order of scale, the transformation unit 620 is specifically configured to perform N-1 inverse subpixel transformations on the first mask image to obtain N-1 transformed mask images of different scales, and to use the first mask image and the N-1 transformed mask images as N target mask images; wherein the first target mask image is the first mask image, and the Nth target mask image is the N-1th transformed mask image.

[0145] In an exemplary embodiment, the fusion unit 630 is specifically configured to perform N fusion operations sequentially based on the image features corresponding to each of the target mask images and the N video image features of different scales, to obtain N feature fusion results, and to use the Nth feature fusion result as the fused image feature; wherein, the first feature fusion result is the result obtained by fusing the image features corresponding to the first target mask image with the first video image feature; the Nth feature fusion result is the result obtained by fusing the (N-1)th feature fusion result, the image features corresponding to the Nth target mask image, and the Nth video image feature.

[0146] In an exemplary embodiment, the fusion unit 630 is specifically configured to perform a convolution operation on the first target mask image to obtain an image convolution result, and determine the activation result corresponding to the image convolution result through a preset activation function as an image feature corresponding to the first target mask image; the image feature corresponding to the first target mask image corresponds to a first resolution; the first resolution is the resolution corresponding to the first video image feature; and the image feature corresponding to the first target mask image and the first video image feature are fused to obtain the first feature fusion result.

[0147] In an exemplary embodiment, the fusion unit 630 is specifically configured to perform a convolution operation on the (N-1)th feature fusion result to obtain the convolution result of the feature fusion result, and determine the activation result corresponding to the convolution result of the feature fusion result through a preset activation function, as the (N-1)th fusion feature; the (N-1)th fusion feature corresponds to a second resolution; the second resolution is the resolution corresponding to the Nth video image feature; extract the image features corresponding to the Nth target mask image; the image features corresponding to the Nth target mask image correspond to the second resolution; and perform a fusion operation on the (N-1)th fusion feature, the image features corresponding to the Nth target mask image, and the Nth video image feature to obtain the Nth feature fusion result.

[0148] In an exemplary embodiment, the region segmentation feature is a key-value pair feature, which includes mask addressing information and mask generation information. The segmentation unit 650 is specifically configured to perform the following operations: adjust the mask generation information corresponding to the first mask image based on the similarity between the mask addressing information corresponding to the first mask image and the mask addressing information corresponding to the video frame to be predicted, to obtain adjusted mask generation information; decode the fusion information between the adjusted mask generation information and the mask generation information corresponding to the video frame to be predicted to obtain a second mask image; the second mask image is used to characterize the segmented region corresponding to the target object in the video frame to be predicted.

[0149] In an exemplary embodiment, the generation unit 640 is specifically configured to perform a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolution features with a first number of channels, which serve as mask addressing information corresponding to the video frame to be predicted; and to perform a convolution operation on the video image features at the smallest scale among the multi-scale image features to obtain convolution features with a second number of channels, which serve as mask generation information corresponding to the video frame to be predicted; wherein the first number of channels is less than the second number of channels.

[0150] In an exemplary embodiment, the generation unit 640 is specifically configured to perform a convolution operation on the fused image features to obtain a convolution operation result, and determine the convolution feature activation result corresponding to the convolution operation result through a preset activation function; perform a convolution operation on the convolution feature activation result to obtain a convolution feature with a third number of channels, which serves as the mask addressing information corresponding to the first mask image; and perform a convolution operation on the convolution feature activation result to obtain a convolution feature with a fourth number of channels, which serves as the mask generation information corresponding to the first mask image; wherein the third number of channels is less than the fourth number of channels.

[0151] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0152] Figure 7 This is a block diagram illustrating an electronic device 700 for performing the video object segmentation method described above, according to an exemplary embodiment. For example, the electronic device 700 may be a server. (Refer to...) Figure 7 The electronic device 700 includes a processing component 720, which further includes one or more processors, and memory resources represented by memory 722 for storing instructions, such as application programs, that can be executed by the processing component 720. The application programs stored in memory 722 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 720 is configured to execute instructions to perform the methods described above.

[0153] Electronic device 700 may further include: a power supply component 724 configured to perform power management of electronic device 700, a wired or wireless network interface 726 configured to connect electronic device 700 to a network, and an input / output (I / O) interface 728. Electronic device 700 may operate on an operating system stored in memory 722, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0154] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 722 including instructions, which can be executed by a processor of an electronic device 700 to perform the above-described method. The storage medium may be a computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0155] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by a processor of an electronic device 700 to perform the above-described method.

[0156] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0157] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0158] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video object segmentation method, characterized in that, include: Obtain the video frame to be predicted and extract the multi-scale image features corresponding to the video frame to be predicted; An inverse subpixel transform is performed on the first mask image to obtain target mask images at at least two different scales; the first mask image is used to characterize the segmented region corresponding to the target object in the target video frame, and the target video frame and the video frame to be predicted belong to the same video. The image features corresponding to each of the target mask images are fused with video image features at least two different scales in the multi-scale image features to obtain fused image features; based on the fused image features, the region segmentation features corresponding to the first mask image are generated, and based on the multi-scale image features, the region segmentation features corresponding to the video frame to be predicted are generated. Based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted, the segmentation region corresponding to the target object in the video frame to be predicted is predicted.

2. The video object segmentation method according to claim 1, characterized in that, The extraction of multi-scale image features corresponding to the video frame to be predicted includes: The video frame to be predicted is subjected to N feature extraction processes to obtain N video image features at different scales, which are used as the multi-scale image features. Wherein, the first video image feature is the result of feature extraction processing on the video frame to be predicted; the Nth video image feature is the result of feature extraction processing on the (N-1)th video image feature; N is greater than or equal to 2.

3. The video object segmentation method according to claim 2, characterized in that, Given that the N video image features at different scales are arranged in descending order of scale, the step of performing at least one inverse subpixel transformation on the first mask image to obtain at least two target mask images at different scales includes: The first mask image is subjected to N-1 inverse subpixel transformations to obtain N-1 transformed mask images of different scales, and the first mask image and the N-1 transformed mask images of different scales are used as N target mask images. Wherein, the first target mask image is the first mask image, and the Nth target mask image is the (N-1)th transformed mask image.

4. The video object segmentation method according to claim 3, characterized in that, The step of fusing the image features corresponding to each of the target mask images with video image features at least two different scales from the multi-scale image features to obtain fused image features includes: Based on the image features corresponding to each target mask image and the N video image features at different scales, N fusion operations are performed sequentially to obtain N feature fusion results, and the Nth feature fusion result is used as the fused image feature; The first feature fusion result is obtained by fusing the image features corresponding to the first target mask image with the first video image features; the Nth feature fusion result is obtained by fusing the (N-1)th feature fusion result, the image features corresponding to the Nth target mask image, and the Nth video image features.

5. The video object segmentation method according to claim 4, characterized in that, The step of fusing the image features corresponding to each of the target mask images with video image features at least two different scales from the multi-scale image features to obtain fused image features includes: A convolution operation is performed on the first target mask image to obtain an image convolution result. The activation result corresponding to the image convolution result is determined by a preset activation function and used as the image feature corresponding to the first target mask image. The image feature corresponding to the first target mask image corresponds to a first resolution. The first resolution is the resolution corresponding to the first video image feature. The image features corresponding to the first target mask image and the first video image features are fused together to obtain the first feature fusion result.

6. The video object segmentation method according to claim 4, characterized in that, The step of fusing the image features corresponding to each of the target mask images with video image features at least two different scales from the multi-scale image features to obtain fused image features includes: A convolution operation is performed on the (N-1)th feature fusion result to obtain the convolution result of the feature fusion result, and the activation result corresponding to the convolution result of the feature fusion result is determined by a preset activation function, which is used as the (N-1)th fusion feature; the (N-1)th fusion feature corresponds to the second resolution; the second resolution is the resolution corresponding to the Nth video image feature; Extract the image features corresponding to the Nth target mask image; the image features corresponding to the Nth target mask image correspond to the second resolution; The N-1th fusion feature, the image feature corresponding to the Nth target mask image, and the Nth video image feature are fused together to obtain the Nth feature fusion result.

7. The video object segmentation method according to claim 1, characterized in that, The region segmentation feature is a key-value pair feature, which includes mask addressing information and mask generation information. The step of predicting the segmented region corresponding to the target object in the video frame to be predicted based on the region segmentation feature corresponding to the first mask image and the region segmentation feature corresponding to the video frame to be predicted includes: Based on the similarity between the mask addressing information corresponding to the first mask image and the mask addressing information corresponding to the video frame to be predicted, the mask generation information corresponding to the first mask image is adjusted to obtain the adjusted mask generation information. The fusion information between the adjusted mask generation information and the mask generation information corresponding to the video frame to be predicted is decoded to obtain a second mask image; the second mask image is used to characterize the segmented region corresponding to the target object in the video frame to be predicted.

8. The video object segmentation method according to claim 7, characterized in that, The step of generating region segmentation features corresponding to the video frame to be predicted based on the multi-scale image features includes: A convolution operation is performed on the video image features at the smallest scale among the multi-scale image features to obtain convolution features with a first number of channels, which are used as the mask addressing information corresponding to the video frame to be predicted. The video image features at the smallest scale among the multi-scale image features are convolved to obtain convolutional features with a second number of channels, which are used as mask generation information for the video frame to be predicted. The number of channels in the first case is less than the number of channels in the second case.

9. The video object segmentation method according to claim 7, characterized in that, The step of generating region segmentation features corresponding to the first mask image based on the fused image features includes: The fused image features are subjected to a convolution operation to obtain the convolution operation result, and the activation result of the convolution feature corresponding to the convolution operation result is determined by a preset activation function. The convolution feature activation result is subjected to a convolution operation to obtain a convolution feature with a third number of channels, which is used as the mask addressing information corresponding to the first mask image. The convolution feature activation result is subjected to a convolution operation to obtain a convolution feature with a fourth channel, which is used as the mask generation information corresponding to the first mask image. The number of the third channel is less than the number of the fourth channel.

10. A video object segmentation device, characterized in that, include: The acquisition unit is configured to acquire the video frame to be predicted and extract the multi-scale image features corresponding to the video frame to be predicted. The transformation unit is configured to perform inverse subpixel transformation on the first mask image to obtain target mask images at at least two different scales; the first mask image is used to characterize the segmented region corresponding to the target object in the target video frame, and the target video frame and the video frame to be predicted belong to the same video. The fusion unit is configured to perform the fusion of image features corresponding to each of the target mask images with video image features at least two different scales in the multi-scale image features to obtain fused image features; The generation unit is configured to perform the following operations: generating region segmentation features corresponding to the first mask image based on the fused image features, and generating region segmentation features corresponding to the video frame to be predicted based on the multi-scale image features. The segmentation unit is configured to perform the task of predicting the segmented region corresponding to the target object in the video frame to be predicted, based on the region segmentation features corresponding to the first mask image and the region segmentation features corresponding to the video frame to be predicted.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video object segmentation method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the video object segmentation method as described in any one of claims 1 to 9.

Images