Methods, apparatus, devices, and storage media for local coupling segmentation of video portraits

Abstract

This disclosure presents embodiments of a method, apparatus, device, and storage medium for locally coupled segmentation of video portraits. One specific implementation of the method includes: determining the first frame of the video as a reference frame to extract motion information; in a twin encoder, using ResNet-50 as the backbone to extract spatial features of the target frame and the reference frame; leveraging the advantages of the FPN structure to fuse spatial semantic features and detail features from deep to shallow; the entire encoder sharing weights between the two frames to obtain consistent appearance features across different frames; unsupervised decoupling the consistent appearance features across different frames into part-specific features; utilizing cross-attention operations on each part to extract part-discriminative motion associations; then combining these associated features to generate motion features for the target frame; fusing the appearance features and motion features of the target frame; and predicting the segmentation result based on the fused features. This implementation effectively improves the performance of video portrait segmentation methods.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of computer technology, and more specifically to a method, apparatus, device, and storage medium for local coupling segmentation of video portraits. Background Technology

[0002] Unsupervised Video Object Segmentation (UVOS) aims to discover and segment foreground objects that are most relevant in appearance and motion from a continuous video stream. Video Portrait Segmentation (VPS) is a subtask of UVOS focusing on portrait-specific objects, aiming to discover and segment salient foreground portraits in video frames. As an efficient preprocessing method, VPS separates key portrait regions from redundant background regions in a video, laying the foundation for downstream operations such as background replacement and portrait transformation. It can be widely applied in virtual reality, video creation, and other fields, replacing green screen construction and shooting processes, and reducing manpower, material, and financial costs.

[0003] Existing unsupervised video target segmentation methods based on deep learning can be broadly categorized into three types: unsupervised video target segmentation methods based on recurrent neural network (RNN) structures, unsupervised video target segmentation methods based on optical flow estimation, and unsupervised video target segmentation methods based on inter-frame correlation. Early deep learning-based methods primarily relied on recurrent network structures to extract spatiotemporal features from videos, iteratively accumulating global temporal features across frames. However, this recurrent structure is prone to the accumulation of interference signals from complex scenes over time, leading to inaccurate segmentation results. Recent methods utilize optical flow maps generated by optical flow estimation models as cues for motion information, explicitly extracting spatial appearance features and temporal motion features through a spatial and temporal dual-branch structure, thereby capturing key targets that are significant in both the temporal and spatial domains. Inter-frame correlation-based methods extract temporal information by constructing temporal correlations between the target frame and reference frames, reducing temporal feature redundancy while locating and segmenting significant targets in both the spatiotemporal domains.

[0004] Unlike conventional video object segmentation tasks, the vast majority of human images on the internet come from diverse and complex environments, including private rooms, classrooms, workshops, and outdoor settings. The number of people, their postures, and their movements are also highly variable. This necessitates segmentation methods with superior motion resolution capabilities to accurately separate the complex movements of human figures from chaotic backgrounds. For conventional video object segmentation, such as cars and airplanes, the motion is holistic, with each part exhibiting a consistent motion state. However, due to the highly component-based structure of the human body, the motion of a human figure is significantly unbalanced, with inconsistent motion states across its various parts.

[0005] Existing video target segmentation methods treat spatiotemporally salient targets as a whole, uniformly extracting their motion information and generating segmentation results based on spatiotemporal fusion features. For conventional targets with strong overall characteristics, this approach can achieve good motion parsing; however, for human figures with complex motion states, the motion features generated by this approach are too coarse, making it difficult to characterize local details that differ from the overall motion state of the target. This results in these methods being unable to effectively process parts such as arms in human figures that differ from the overall motion state, thus affecting the accuracy, temporal coherence, and consistency of the segmentation results. Summary of the Invention

[0006] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0007] Some embodiments of this disclosure provide a method, apparatus, electronic device, and computer storage medium for local coupling segmentation of video portraits to solve one or more of the technical problems mentioned in the background section above.

[0008] In a first aspect, some embodiments of this disclosure provide a locally coupled segmentation method for video portraits. The method includes: determining the first frame of the video as a reference frame to extract motion information; in a twin encoder, using ResNet-50 as the backbone to extract spatial features of the target frame and the reference frame; leveraging the advantages of the FPN structure to fuse spatial semantic features and detail features from deep to shallow; the entire encoder sharing weights between the two frames to obtain consistent appearance features across different frames; unsupervised decoupling the consistent appearance features across different frames into part-specific features; utilizing cross-attention operations on each part to extract part-discriminative motion associations; then combining these associated features to generate motion features for the target frame; after obtaining the appearance and motion features of the target frame, fusing the appearance and motion features of the target frame; and predicting the segmentation result based on the fused features.

[0009] Secondly, some embodiments of this disclosure provide a local coupling segmentation apparatus for video portraits. The apparatus includes: a determination unit configured to determine the first frame of the video as a reference frame to extract motion information; in a twin encoder, using ResNet-50 as the backbone to extract spatial features of the target frame and the reference frame; leveraging the advantages of the FPN structure to fuse spatial semantic features and detail features from deep to shallow; the entire encoder sharing weights between the two frames to obtain consistent appearance features across different frames; a decoupling unit configured to unsupervisedly decouple the consistent appearance features across different frames into part-specific features; and utilize cross-attention operations on each part to extract part-discriminative motion associations, then combining these associated features to generate motion features of the target frame; and a fusion unit configured to fuse the appearance features and motion features of the target frame after obtaining them, and predict the segmentation result based on the fused features.

[0010] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0011] Fourthly, some embodiments of this disclosure provide a computer storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0013] Figure 1 This is a flowchart of some embodiments of the local coupling segmentation method for video portraits according to the present disclosure;

[0014] Figure 2 This is an unsupervised component decoupling flowchart of some embodiments of the local coupling segmentation method for video portraits according to the present disclosure;

[0015] Figure 3 This is a diagram of component feature cross-attention operations according to some embodiments of the local coupling segmentation method for video portraits disclosed herein;

[0016] Figure 4This is a spatiotemporal fusion module diagram of some embodiments of the local coupling segmentation method for video portraits according to the present disclosure;

[0017] Figure 5 This is a network architecture diagram of some embodiments of the local coupling segmentation method for video portraits according to the present disclosure;

[0018] Figure 6 This is a schematic diagram of the structure of some embodiments of the local coupling segmentation device for video portraits according to the present disclosure;

[0019] Figure 7 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0020] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0021] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0022] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0023] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0024] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0025] This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0026] Figure 1 A flow 100 of some embodiments of a locally coupled segmentation method for video portraits according to the present disclosure is shown. This locally coupled segmentation method for video portraits includes the following steps:

[0027] Step 101: The first frame of the video is determined as the reference frame to extract motion information. In the twin encoder, ResNet-50 is used as the backbone to extract the spatial features of the target frame and the reference frame. Taking advantage of the FPN structure, spatial semantic features and detail features are fused from deep to shallow. The entire encoder shares weights between the two frames, thereby obtaining consistent appearance features in different frames.

[0028] Step 102: Unsupervisedly decouple consistent appearance features across different frames into part-specific features, and use cross-attention operations on each part to extract part-discriminative motion associations. Then, combine these associated features to generate motion features for the target frame.

[0029] In practice, the Inter-Frame Partial Discrimination Attention (IPDA) module is used to extract motion association features from the reference frame to the target frame for each part of the portrait. This module includes three stages: unsupervised part decoupling, association extraction, and association combination, to address the imbalance of portrait motion.

[0030] First, the unsupervised component decoupling stage: In this stage, the portrait is unsupervisedly segmented into multiple components, and discriminative features of each component are extracted simultaneously. To enhance the guidance of saliency cues, the unsupervised component segmentation process is decomposed into two steps: the first step is supervised prediction of the saliency map of the portrait based on appearance features; the second step is self-supervised prediction of each component of the portrait. The unsupervised component decoupling flowchart of some embodiments of the locally coupled video portrait segmentation method disclosed herein is shown below. Figure 2 As shown.

[0031] Specifically, the appearance features extracted by the twin encoder are downsampled, and 3×3 and 1×1 convolution operations are used to compress the dimensionality of the appearance features to reduce computational cost. Then, a saliency prediction head is applied to predict the saliency map of the human face. Convolution operations are used to decouple the compressed features into part-discriminative features, and then a weight-shared prediction head is applied to the part-discriminative features to generate decoupled part predictions.

[0032] In some embodiments, firstly, the appearance features extracted from the twin encoder... , Downsampling is performed, and compression is achieved using 3×3 convolution and 1×1 convolution operations. and To reduce computational costs, a dimensionality prediction head is used. Secondly, a saliency prediction head is applied to predict the saliency map of the human face. and Then, using Grouping 1×1, 3×3, and 1×1 convolution operations to decouple the compressed features into part-specific discriminative features. and .in, Indicates the number of parts. Indicates the first Each component. Indicates the first frame. Indicates the reference frame. Then, in A weight-shared prediction head is applied to the discriminative features of each component to generate decoupled component predictions. and This weight-sharing mechanism ensures semantic consistency among component discrimination features. Predicted component mask. , It is by using a portrait saliency map , Decoupled component prediction , Multiplying them together, we get, where, This indicates the prediction of decoupled components. Taking the target frame as an example, It can be calculated using the following formula:

[0033] .

[0034] in, This indicates element-wise multiplication. The operations on the reference frame are the same as those on the target frame. All operations in the component decoupling phase share weights between the features of the target and reference frames, ensuring component correspondence and feature consistency between the two frames.

[0035] For human face segmentation, the weighted binary cross-entropy (weighted BCE) loss function and the L1 loss function are used. The weighted binary cross-entropy loss function is shown below:

[0036] .

[0037] in, , These represent the predicted mask and the labeled value, respectively. This indicates the ratio of foreground to background pixel count. These represent the height and width coordinates of the image. The L1 loss can be expressed as the following formula:

[0038] .

[0039] Both losses are applied simultaneously to the final prediction and other auxiliary outputs of the twin encoder to enhance the encoder's ability to extract human features.

[0040] For the component decoupling stage in the IPDA module, weighted binary cross-entropy loss and L1 loss are used in the saliency prediction step. In the self-supervised component segmentation prediction step, geometric concentration loss, semantic consistency loss, and area variance loss are used, with the first two being commonly used loss functions in self-supervised component segmentation tasks.

[0041] Geometric centralization loss aims to cluster the generated components into connected components, and its definition is shown in the following formula:

[0042] .

[0043] in, Indicates components The prediction mask. , express Height and width. This represents a small constant to prevent division by zero. express The square of the 2-norm. Indicates components The area of ​​the predicted mask is calculated using the following formula:

[0044] .

[0045] , express The centroid of is defined as shown in the following formula:

[0046] .

[0047] Second, the association extraction stage: Cross-attention modules are applied to the component discrimination features respectively to obtain the component discrimination motion features of the target frame. The component feature cross-attention operation diagram of some embodiments of the local coupling segmentation method for video portraits disclosed herein is shown below. Figure 3 As shown.

[0048] In some embodiments, when obtaining component discrimination features and Afterwards, respectively Application of component discrimination features for individual components A cross-attention module is used to obtain the component discrimination motion features of the target frame. Note that, unlike typical self-attention modules, the cross-attention module used is asymmetric, where the query features are derived from a component in the target frame. Distinguishing features The key and value features are derived from the corresponding components in the reference frame. Distinguishing features In this way, the component discrimination motion association is extracted from each component in the reference frame to the corresponding component in the target frame.

[0049] Semantic consistency loss aims to guide modules to generate component partitions where the semantic differences within each component are small, but the semantic differences between components are large. Its definition is shown in the following formula:

[0050] .

[0051] in, It refers to a human face mask. This represents the semantic features generated by a ResNet-18 pre-trained on ImageNet with fixed weights. Indicates components The representative pixel features are learnable. Represented by normalized pixel features or A matrix formed by stacking. express The product of a matrix and its transpose. express The square of the Frobenius norm.

[0052] The area variance loss is designed to prevent the generation of excessively large or small components, which would reduce the IPDA module's ability to resolve motion. Its definition is shown in the following formula:

[0053] .

[0054] in, Indicates variance.

[0055] Due to the significance of the prediction plot and Inaccurate results are caused by using only the predicted saliency map in self-supervised part segmentation, leading to unstable results. Therefore, a labeling mask was introduced during the training phase. , and the component mask generated using the annotation mask. , Target frame component mask The calculation is shown in the following formula, and the operation of the reference frame is the same as that of the target frame.

[0056] .

[0057] Since labeled values ​​are unavailable during inference, to enhance the consistency between training and inference predictions while ensuring accuracy, self-supervision is simultaneously used... , and , Total component segmentation loss of the target frame As shown in the following formula:

[0058] .

[0059] in, As shown in the following formula, the reference frame The calculation method is the same.

[0060] .

[0061] Third, the association and combination stage: assemble the component discrimination motion features of the target frame, and obtain the synthetic motion features by multiplying the motion features with the prediction mask of the corresponding component and adding them together.

[0062] In some embodiments, assembling the target frame Individual components determine motion characteristics By using the prediction mask of the corresponding component Multiplying the motion features by the features and then adding them together yields the synthetic motion features. .

[0063] .

[0064] Step 103: After obtaining the appearance features and motion features of the target frame, fuse the appearance features and motion features of the target frame, and predict the segmentation result based on the fused features.

[0065] In some embodiments, the spatiotemporal fusion module can be used to fuse the appearance features and motion features of the target frame, and the segmentation result can be predicted based on the fused features.

[0066] In practice, the aforementioned spatiotemporal fusion module is used to fuse the appearance features of the target frame. and motion characteristics First, 1×1 convolution operations are used to expand motion features. The dimension is determined by two steps. First, another 1×1 convolution operation is performed on the extended motion features, and the result is combined with the appearance features. Multiplication is performed to obtain more responsive spatiotemporal features for regions important in both appearance and motion. Third, the appearance features, motion features, and spatiotemporal features of the target frame are combined, and two 3×3 convolution operations are used to generate fused features. A spatiotemporal fusion module diagram of some embodiments of the locally coupled segmentation method for video portraits disclosed herein is shown below. Figure 4 As shown.

[0067] The network architecture diagrams of some embodiments of the locally coupled segmentation method for video portraits disclosed herein are as follows: Figure 5 As shown.

[0068] Further reference Figure 6 As an implementation of the methods shown in the above figures, this disclosure provides some embodiments of a local coupling segmentation device for video portraits, these device embodiments being similar to... Figure 1 Corresponding to the method embodiments shown, this local coupling segmentation device for video portraits can be specifically applied to various electronic devices.

[0069] like Figure 6 As shown, a local coupling segmentation apparatus 600 for video portraits in some embodiments includes: a determination unit 601, a decoupling unit 602, and a fusion unit 603. The determination unit 601 is configured to determine the first frame of the video as a reference frame to extract motion information. In a twin encoder, ResNet-50 is used as the backbone to extract spatial features of the target frame and the reference frame. The advantages of the FPN structure are utilized to fuse spatial semantic features and detail features from deep to shallow. The entire encoder shares weights between the two frames, thereby obtaining consistent appearance features across different frames. The decoupling unit 602 is configured to unsupervisedly decouple the consistent appearance features across different frames into part-specific features, and utilize cross-attention operations on each part to extract part-discriminative motion associations. These associated features are then combined to generate motion features for the target frame. The fusion unit 603 is configured to fuse the appearance and motion features of the target frame after obtaining them, and predict the segmentation result based on the fused features.

[0070] It is understandable that the units described in the local coupling segmentation device 600 for video portraits are related to the reference... Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the local coupling segmentation device 600 for video portraits and the units contained therein, and will not be repeated here.

[0071] The following is for reference. Figure 7 The diagram illustrates a structural schematic of an electronic device 700 suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0072] like Figure 7As shown, the electronic device 700 may include a processing unit 701 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device 700. The processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0073] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 700 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 7 Each box shown can represent a device or multiple devices as needed.

[0074] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702. When the computer program is executed by processing device 701, it performs the functions defined in the methods of some embodiments of this disclosure.

[0075] It should be noted that the computer storage medium described in some embodiments of this disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0076] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0077] The aforementioned computer storage medium may be included within the aforementioned electronic device; alternatively, it may exist independently and not be assembled into the electronic device. The aforementioned computer storage medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: determine the first frame of the video as the reference frame to extract motion information; in a twin encoder, use ResNet-50 as the backbone to extract spatial features of the target frame and the reference frame; leverage the advantages of the FPN structure to fuse spatial semantic features and detail features from deep to shallow; the entire encoder shares weights between the two frames, thereby obtaining consistent appearance features across different frames; unsupervisedly decouple the consistent appearance features across different frames into part-specific features, and utilize cross-attention operations on each part to extract part-discriminative motion associations; then combine these associated features to generate the motion features of the target frame; after obtaining the appearance and motion features of the target frame, fuse the appearance and motion features of the target frame, and predict the segmentation result based on the fused features.

[0078] Computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0079] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0080] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including a determining unit, a decoupling unit, and a fusion unit. The names of these units do not necessarily limit the specific unit itself; for example, a fusion unit may also be described as "a unit that, after obtaining the appearance and motion features of a target frame, fuses the appearance and motion features of the target frame, and predicts the segmentation result based on the fused features."

[0081] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0082] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A locally coupled segmentation method for video portraits, comprising: The first frame of the video is determined as the reference frame to extract motion information. In the twin encoder, ResNet-50 is used as the backbone to extract spatial features of the target frame and the reference frame. Taking advantage of the FPN structure, spatial semantic features and detail features are fused from deep to shallow. The entire encoder shares weights between the two frames, thereby obtaining consistent appearance features in different frames. Consistent appearance features across different frames are unsupervisedly decoupled into part-specific features, and cross-attention operations on each part are used to extract part-discriminative motion associations. These associated features are then combined to generate motion features for the target frame. After obtaining the appearance and motion features of the target frame, the appearance and motion features of the target frame are fused, and the segmentation result is predicted based on the fused features; The process of unsupervisedly decoupling consistent appearance features across different frames into part-specific features, extracting part-discriminative motion associations using cross-attention operations on each part, and then combining these associated features to generate motion features for the target frame includes: Appearance features extracted by twin encoder 、 Down-sampling is performed and 3x3 convolution and 1x1 convolution operations are used to compress the dimension of and to reduce the computational cost; Applying a saliency prediction head to predict a portrait saliency map and ; Utilizing Group 1x1, 3x3, and 1x1 convolution operations to decouple the compressed features into part discriminative features and wherein, denotes the number of parts, denotes the thpart, denotes the thframe, denotes the reference frame; In application of weight sharing prediction head on component discriminative features to generate decoupled component predictions and , predicted component masks , are obtained by multiplying the human portrait saliency map , and decoupled component predictions , , wherein represents decoupled component predictions, taking the target frame as an example, are calculated by the following formula: ， in, This indicates element-wise multiplication. The operations in the reference frame are the same as those in the target frame. All operations in the component decoupling stage share weights between the features of the target and reference frames. For human image segmentation, a weighted binary cross-entropy loss function and an L1 loss function are used. The weighted binary cross-entropy loss function is shown below: ， in, , These represent the predicted mask and the labeled value, respectively. This indicates the ratio of foreground to background pixel count. The L1 loss is expressed as the following formula, representing the height and width coordinates of the image: ， Both losses are applied simultaneously to the final prediction and other auxiliary outputs of the twin encoder to enhance the encoder's ability to extract human features; For the component decoupling stage in the inter-frame component discrimination attention module, weighted binary cross-entropy loss and L1 loss are used in the saliency prediction step, and geometric centralization loss, semantic consistency loss and area variance loss are used in the self-supervised component segmentation prediction step. Geometric centralization loss aims to cluster the generated components into connected components, and its definition is shown in the following formula: ， in, Indicates components Predictive mask, , express Height and width, This represents a small constant to prevent division by zero. express The square of the 2-norm, Indicates components The area of ​​the predicted mask is calculated using the following formula: ， , express The centroid of is defined as shown in the following formula: 。 2. The method according to claim 1, wherein, The method of unsupervisedly decoupling consistent appearance features across different frames into part-specific features, extracting part-discriminative motion associations using cross-attention operations on each part, and then combining these associated features to generate motion features for the target frame, further includes: A cross-attention module is applied to the component discrimination features to obtain the component discrimination motion features of the target frame.

3. The method according to claim 2, wherein, The method of unsupervisedly decoupling consistent appearance features across different frames into part-specific features, extracting part-discriminative motion associations using cross-attention operations on each part, and then combining these associated features to generate motion features for the target frame, further includes: The motion features of the components in the assembled target frame are determined by multiplying the motion features with the predicted masks of the corresponding components and then adding them together to obtain the synthetic motion features.

4. The method according to claim 1, wherein, The appearance and motion features of the fused target frame include: Use 1×1 convolution operations to expand the dimension of motion features; Another 1×1 convolution operation is applied to the extended motion features, and the result is multiplied with the appearance features to obtain more responsive spatiotemporal features for regions that are important in both appearance and motion. The appearance features, motion features, and spatiotemporal features of the target frame are combined, and two 3×3 convolution operations are used to generate fused features.

5. A local coupling segmentation device for video portraits, comprising: The determination unit is configured to determine the first frame of the video as the reference frame to extract motion information. In the twin encoder, ResNet-50 is used as the backbone to extract spatial features of the target frame and the reference frame. The advantages of the FPN structure are utilized to fuse spatial semantic features and detail features from deep to shallow. The entire encoder shares weights between the two frames, thereby obtaining consistent appearance features in different frames. The decoupling unit is configured to unsupervisedly decouple consistent appearance features across different frames into part-specific features, and utilize cross-attention operations on each part to extract part-discriminative motion associations, and then combine these associated features to generate motion features for the target frame. The fusion unit is configured to fuse the appearance features and motion features of the target frame after obtaining the appearance features and motion features of the target frame, and predict the segmentation result based on the fused features; The decoupling unit is further configured as follows: Appearance features extracted from twin encoders , Downsampling is performed, and compression is achieved using 3×3 convolution and 1×1 convolution operations. and To reduce computational costs; Applying a saliency prediction head to predict the saliency map of a person's face and ; use Grouping 1×1, 3×3, and 1×1 convolution operations to decouple the compressed features into part-specific discriminative features. and ,in, Indicates the number of parts. Indicates the first Each component Indicates the first frame, Indicates a reference frame; exist A weight-shared prediction head is applied to the discriminative features of each component to generate decoupled component predictions. and Predicted component mask , It is by using a portrait saliency map , Decoupled component prediction , Multiplying them together, we get, where, This represents the prediction of decoupled components, taking the target frame as an example. Calculated using the following formula: ， in, This indicates element-wise multiplication. The operations in the reference frame are the same as those in the target frame. All operations in the component decoupling stage share weights between the features of the target and reference frames. For human image segmentation, a weighted binary cross-entropy loss function and an L1 loss function are used. The weighted binary cross-entropy loss function is shown below: ， in, , These represent the predicted mask and the labeled value, respectively. This indicates the ratio of foreground to background pixel count. The L1 loss is expressed as the following formula, representing the height and width coordinates of the image: ， Both losses are applied simultaneously to the final prediction and other auxiliary outputs of the twin encoder to enhance the encoder's ability to extract human features; For the component decoupling stage in the inter-frame component discrimination attention module, weighted binary cross-entropy loss and L1 loss are used in the saliency prediction step, and geometric centralization loss, semantic consistency loss and area variance loss are used in the self-supervised component segmentation prediction step. Geometric centralization loss aims to cluster the generated components into connected components, and its definition is shown in the following formula: ， in, Indicates components Predictive mask, , express Height and width, This represents a small constant to prevent division by zero. express The square of the 2-norm, Indicates components The area of ​​the predicted mask is calculated using the following formula: ， , express The centroid of is defined as shown in the following formula: 。 6. An electronic device, comprising: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 4.

7. A computer storage medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 4.

Images