Method, device, terminal and medium for changing attributes of video specific attribute area

Abstract

The present invention discloses a method, device, terminal and medium for replacing the attributes of a specific attribute area of ​​a video, and relates to the field of video processing technology. The method of the present invention comprises: extracting target attribute features based on a reference image; for each video frame in the original video, extracting the original spatial features of the specific attribute area, deforming the original spatial features to a standard space, and performing attribute modulation with the target attribute features to obtain modulated attribute features; inversely deforming the modulated attribute features to the original video space to obtain a target video frame, and the specific attribute area of ​​the target video frame has the target attribute features of the reference image. The present invention effectively decouples the motion features of the replacement object in the video frame from the attribute features of the specific attribute area to be replaced by converting the video frame to a standard space for attribute replacement, thereby avoiding the influence on the motion features during the attribute replacement process, thereby effectively improving the temporal consistency and motion feature alignment of the video before and after the attribute replacement.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a method, device, terminal and medium for replacing attributes of a specific attribute area of ​​a video. Background Art

[0002] The method for replacing the attributes of a specific attribute area in a video involves splitting the original video into video frames and locally adjusting or replacing the attributes of the specific attribute area of ​​the object to be replaced in each video frame. Existing methods for replacing the attributes of specific attribute areas in videos are mainly divided into two types. One is to directly replace the attributes of the specific attribute area of ​​the object to be replaced in each video frame, and then splice the replacement results of each video frame into a new video. The other is to use a video diffusion model. To avoid excessive computational overhead, the video frames are segmented and input into the model separately. A temporal attention mechanism is introduced into each video segment to achieve the attribute replacement of the specific attribute area of ​​the object to be replaced.

[0003] However, existing methods for replacing specific attribute regions in videos cannot avoid modifying the pose and other motion characteristics of the object being replaced during the attribute replacement process. Consequently, these methods struggle to maintain the dynamic characteristics of the original video (such as the pose and motion of the object being replaced). This can lead to distortion (unnatural flickering and jitter) and decreased temporal consistency (audio and video desynchronization) in the replaced video.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is that, in response to the above-mentioned defects of the prior art, a method, device, terminal and medium for replacing the attributes of a specific attribute area of ​​a video are provided, aiming to solve the problem that the existing method for replacing the attributes of a specific attribute area of ​​a video cannot avoid modifying the motion characteristics such as the posture of the object to be replaced, resulting in distortion and decreased timing consistency in the video after the attribute replacement.

[0006] The technical solutions adopted by the present invention to solve the problem are as follows:

[0007] In a first aspect, an embodiment of the present invention provides a method for changing attributes of a specific attribute area of ​​a video, the method comprising:

[0008] Extract target attribute features based on the reference image;

[0009] For each video frame in the original video, extract the original spatial features of the specific attribute area, deform the original spatial features into a standard space, and then perform attribute modulation with the target attribute features to obtain modulated attribute features;

[0010] A target video frame is obtained by inversely deforming the modulated attribute features to the original video space, and a specific attribute region of the target video frame has the target attribute features of the reference image.

[0011] In one embodiment, deforming the original spatial feature into a standard space and then performing attribute modulation with the target attribute feature includes:

[0012] Capturing standard key points of a specific attribute area in a video frame in a standard space, and calculating image key points of the specific attribute area in the original video space;

[0013] Deformation estimation from original video space to standard space based on standard key points and image key points;

[0014] transforming the original spatial features based on the deformation estimation to obtain standard spatial features;

[0015] Local attribute modulation is performed on the specific attribute area of ​​the video frame based on the standard spatial features and the target attribute features.

[0016] In one embodiment, predicting deformation estimation from the original video space to the standard space based on the standard key points and the image key points includes:

[0017] Performing a key point capture operation on the video frame to obtain standard key points corresponding to the specific attribute area in the standard space, and motion parameter information of the specific attribute area;

[0018] Calculating the image key points corresponding to the specific attribute area in the original video space according to the standard key points and the motion parameter information;

[0019] Based on the standard key points and the image key points, a deformation estimation from the original video space to the standard space is predicted.

[0020] In one embodiment, inversely transforming the modulation attribute feature to the original video space to obtain a target video frame includes:

[0021] Predicting an inverse deformation estimation from the standard space to the original video space based on the standard key points and the image key points;

[0022] performing inverse deformation on the modulation attribute feature based on the inverse deformation estimation to obtain an updated spatial feature of the original video space;

[0023] The updated spatial features are decoded to obtain a target video frame.

[0024] In one embodiment, performing local attribute modulation on a specific attribute region of the video frame based on the standard spatial feature and the target attribute feature includes:

[0025] The target attribute feature is a multi-scale attribute modulation vector;

[0026] The standard spatial feature is fused with the multi-scale attribute modulation vector scale by scale through a plurality of cascaded local attribute modulation modules to obtain the modulation attribute feature.

[0027] In one embodiment, the standard spatial feature is fused with the multi-scale attribute modulation vector scale by scale through a plurality of cascaded local attribute modulation modules to obtain the modulation attribute feature, including:

[0028] Inputting the standard spatial features and the attribute modulation vector of the corresponding scale into the first layer local attribute modulation module for fusion;

[0029] The fusion features output by the local attribute modulation module of the previous layer and the attribute modulation vector of the corresponding scale are input into the local attribute modulation module of the next layer for fusion;

[0030] The fusion feature output by the last layer of local attribute modulation module is used as the modulation attribute feature.

[0031] In one embodiment, the local attribute modulation module includes:

[0032] A mask prediction module outputs a predicted mask based on the standard spatial features; if the mask prediction module is located in the first-layer local attribute modulation module, the predicted mask is output based on the standard spatial features; if the mask prediction module is located in a local attribute modulation module of another layer, the predicted mask is output based on the fused features output by the local attribute modulation module of the previous layer as the updated standard spatial features;

[0033] A first convolution module extracts a first feature based on the standard spatial feature; if the first convolution module is located in a local attribute modulation module of a first layer, the first feature is extracted based on the standard spatial feature; if the first convolution module is located in a local attribute modulation module of another layer, the first feature is extracted based on the fused feature output by the local attribute modulation module of the previous layer as the updated standard spatial feature;

[0034] A second convolution module extracts a second feature based on the standard spatial feature and the attribute modulation vector of the corresponding scale; wherein, if the second convolution module is located in the local attribute modulation module of the first layer, the second feature is extracted based on the standard spatial feature and the attribute modulation vector of the corresponding scale; if the second convolution module is located in the local attribute modulation module of another layer, the fused feature output by the local attribute modulation module of the previous layer is used as the updated standard spatial feature, and the second feature is extracted based on the attribute modulation vector of the corresponding scale;

[0035] The first feature and the second feature are fused according to the predicted mask.

[0036] In one embodiment, the convolution weights of the second convolution module are adjusted based on an attribute modulation vector of a corresponding scale.

[0037] In one embodiment, fusing the first feature and the second feature according to the predicted mask includes:

[0038] Determining weight values ​​corresponding to the first feature and the second feature respectively based on the predicted mask;

[0039] Weighted fusion is performed according to the first feature, the second feature and their corresponding weight values.

[0040] In one embodiment, after obtaining the modulation attribute feature, the method further includes:

[0041] The modulation attribute features are repaired through a plurality of three-dimensional convolution blocks with jump connections.

[0042] In a second aspect, an embodiment of the present invention further provides a device for replacing attributes of a specific attribute region of a video, the device comprising:

[0043] An image processing module is used to extract target attribute features based on a reference image;

[0044] An attribute modulation module is used to extract the original spatial features of a specific attribute area for each video frame in the original video, deform the original spatial features into a standard space, and then perform attribute modulation with the target attribute features to obtain modulated attribute features;

[0045] The video frame generation module is used to reversely transform the modulated attribute features to the original video space to obtain a target video frame, wherein the specific attribute area of ​​the target video frame has the target attribute features of the reference image.

[0046] In a third aspect, an embodiment of the present invention further provides a terminal comprising a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing a method for replacing attributes of a specific attribute area of ​​a video as described above; and the processor is used to execute the programs.

[0047] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of the attribute replacement method for a specific attribute area of ​​a video as described above.

[0048] Beneficial effects of the present invention: The embodiment of the present invention effectively decouples the motion characteristics of the replacement object in the video frame from the attribute characteristics of the specific attribute area to be replaced by converting the video frame into a standard space for attribute replacement, thereby avoiding the influence of the attribute replacement process on the motion characteristics, thereby effectively improving the timing consistency and motion feature alignment of the video before and after the attribute replacement, and obtaining a target video with high fidelity and strong timing consistency after the attribute replacement. The embodiment of the present invention only needs to input a reference image and an original video, and can replace the attributes of the specific attribute area in the reference image with the specific attribute area to be replaced in the original video with high fidelity to obtain the target video. The specific attribute area in the target video has the attribute characteristics of the specific attribute area in the reference image, and is suitable for the technical fields of video processing such as virtual anchors, film and television editing, privacy protection, VR, AR, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 4 is a flow chart of a method for replacing attributes of a video specific attribute area provided by an embodiment of the present invention.

[0051] Figure 2 This is a principle block diagram of a method for replacing attributes of a video-specific attribute area provided by an embodiment of the present invention.

[0052] Figure 3 It is a structural diagram of a local attribute modulation module provided by an embodiment of the present invention.

[0053] Figure 4 It is a module diagram of a device for replacing attributes of a specific attribute area of ​​a video provided by an embodiment of the present invention.

[0054] Figure 5 This is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention discloses a method, device, terminal, and medium for changing the attributes of a specific attribute region in a video. To make the objectives, technical solutions, and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention.

[0056] It will be understood by those skilled in the art that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. It should be understood that the term "and / or" used includes all or any unit and all combinations of one or more associated listed items.

[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0058] In response to the above-mentioned defects of the prior art, the present invention provides a method for replacing the attributes of a specific attribute area in a video, the method comprising: extracting target attribute features based on a reference image; for each video frame in the original video, extracting the original spatial features of the specific attribute area, deforming the original spatial features to a standard space, and then performing attribute modulation with the target attribute features to obtain modulated attribute features; inversely deforming the modulated attribute features to the original video space to obtain a target video frame, wherein the specific attribute area of ​​the target video frame has the target attribute features of the reference image. In addition to attribute features, the object to be replaced may also have motion features. The present invention effectively decouples the motion features of the object to be replaced in the video frame from the attribute features of the specific attribute area to be replaced by converting the video frame to a standard space for attribute replacement, thereby avoiding the influence of the attribute replacement process on the motion features, thereby effectively improving the temporal consistency and motion feature alignment of the video before and after the attribute replacement, and obtaining a high-fidelity target video with strong temporal consistency after the attribute replacement. The embodiment of the present invention only requires the input of a reference image and an original video, and can replace the attributes of the specific attribute area in the reference image with the specific attribute area to be replaced in the original video with high fidelity to obtain the target video. The specific attribute area in the target video has the attribute characteristics of the specific attribute area in the reference image, and is suitable for technical fields of video processing such as virtual anchors, film and television editing, privacy protection, VR, AR, etc.

[0059] For example, a specific attribute region refers to a local region with specific functions, semantics, or visual features in an image, such as a human body-related region, a vehicle-related region, or a specific region in a medical image. Assume that the specific attribute region in a video includes a face region, and the features of the face region include facial attribute features, facial motion features (expression features, posture features), etc. Obtain a reference image A, and extract the facial features of person a based on the reference image A. Obtain an original video B containing person b, and convert a video frame in the original video B into The original spatial features are converted to the standard space and modulated attribute features are generated together with the facial features of person a, so that the facial attribute features of person b are replaced with the facial attribute features of person a. Finally, the modulated attribute features are converted back to the original video space and the target video frame is generated. . Target video frame Video frames from the original video In comparison, only the facial attribute features of person B are modified, and the motion characteristics of person B in the original video, such as posture, facial expression, and mouth movement, are retained.

[0060] like Figure 1 As shown, the method specifically includes the following steps:

[0061] Step S100: extracting target attribute features based on a reference image.

[0062] Specifically, the reference image can be any image input by the user. The reference image contains feature representations of specific attribute regions that need to be applied to the object to be replaced in the original video. Attribute features of the specific attribute regions of the reference image are extracted through a specific encoding network to obtain target attribute features, such as facial attribute features. In actual application scenarios, the reference image can be directly input into a specific encoding model for attribute feature extraction; or the reference image can be cropped to remove background information unrelated to the specific attribute region, and then the attribute features of the cropped local image can be extracted through the encoding network.

[0063] For example, if the reference image is a face image and the target attribute features are facial attribute features, a face recognition model can be used to complete the feature extraction process to obtain facial attribute features. The face image is input into a pre-trained face recognition model, and the features of the last two layers of the model are extracted as facial attribute features.

[0064] The pre-trained face recognition model can use the following calculation method to implement the feature extraction process:

[0065] , formula (1);

[0066] in, is the face image obtained after cropping and extraction; It is a pre-trained face recognition model; is the extracted facial attribute features.

[0067] In one implementation, an ArcFace model or a CosFace model may be used as a face recognition model.

[0068] Step S200: For each video frame in the original video, extract the original spatial features of the specific attribute area, deform the original spatial features into a standard space, and then perform attribute modulation with the target attribute features to obtain modulated attribute features.

[0069] Specifically, the original video can be any video in which the attributes of a specific attribute region need to be replaced. The original video space refers to the space defined by the original pose of the object to be replaced in the video frame. The pose is generally defined by the key points of the object to be replaced. The standard space refers to the space defined by a preset standard pose of the object to be replaced. For example, in the standard space of a face, the standard pose of a face is front-facing, eyes looking straight ahead, and mouth closed. Taking a video frame from the original video as an example, a feature encoder extracts the image features of the specific attribute region in the original video space, thereby obtaining the original spatial features. To prevent the motion features of the object to be replaced in the video frame from being affected by the attribute replacement process, this embodiment converts the original spatial features of the specific attribute region to the standard space to decouple the motion features of the object to be replaced in the video frame from the attribute features of the specific attribute region to be replaced. The original spatial features are then attribute-modulated using the target attribute features in the standard space to obtain modulated attribute features. Finally, the modulated attribute features are converted back to the original video space to generate the target video frame after the attribute replacement of the specific attribute region.

[0070] In one implementation, deforming the original spatial feature into a standard space and then performing attribute modulation with the target attribute feature includes:

[0071] Capturing standard key points of a specific attribute area in a video frame in a standard space, and calculating image key points of the specific attribute area in the original video space;

[0072] Deformation estimation from original video space to standard space based on standard key points and image key points;

[0073] transforming the original spatial features based on the deformation estimation to obtain standard spatial features;

[0074] Local attribute modulation is performed on the specific attribute area of ​​the video frame based on the standard spatial features and the target attribute features.

[0075] Converting the original spatial features to the standard space to change the attributes of a specific attribute area involves the mutual conversion between the original video space and the standard space. Therefore, this embodiment will perform deformation estimation prediction based on key point capture on each video frame in the original video to predict the deformation estimation from the original video space to the standard space. Figure 2As shown, taking a video frame from the original video as an example, keypoint capture technology is first used to obtain keypoints of the specific attribute region of the object to be replaced in the video frame in a standard pose (e.g., a standard frontal orientation), thereby obtaining standard keypoints. Standard keypoints reflect the geometric shape and position of the specific attribute region in the standard space. The corresponding keypoints of the specific attribute region in the original video space are then calculated to obtain image keypoints. Standard keypoints and image keypoints can reflect the positional changes of the same keypoint in different spaces. Therefore, the correspondence between standard keypoints and image keypoints can be used to predict the deformation estimate required to transform the image keypoint from the original video space to the standard space. After obtaining the deformation estimate, the original spatial features are deformed from the original video space to the standard space using the deformation estimate, thereby obtaining the standard spatial features. In the standard space, the target attribute features are injected into the specific attribute region of the standard spatial features to achieve local attribute modulation. This embodiment performs local attribute modulation on the standard space features and the target attribute features, ensuring high-quality attribute conversion while reducing artifacts caused by attribute replacement and effectively preserving the motion characteristics and audio-visual synchronization of the object to be replaced in the original video.

[0076] In one implementation, predicting deformation estimation from the original video space to the standard space based on the standard key points and the image key points includes:

[0077] Performing a key point capture operation on the video frame to obtain standard key points corresponding to the specific attribute area in the standard space, and motion parameter information of the specific attribute area;

[0078] Calculating the image key points corresponding to the specific attribute area in the original video space according to the standard key points and the motion parameter information;

[0079] Based on the standard key points and the image key points, a deformation estimation from the original video space to the standard space is predicted.

[0080] Specifically, a keypoint capture operation is performed on the video frame to obtain the standard keypoints corresponding to the specific attribute region of the object to be replaced in the standard space, and the motion parameter information of the specific attribute region in the original video space is obtained. The motion parameter information reflects the dynamic behavior of the specific attribute region in the original video space, such as changes in posture, shape, and position. By analyzing the standard keypoints and motion parameter information, the keypoints corresponding to the specific attribute region in the original video space can be inferred, thereby obtaining the image keypoints. Based on the correspondence between the standard keypoints and the image keypoints, the deformation estimation from the original video space to the standard space is predicted.

[0081] For example, Figure 2As shown, for each video frame in the original video B , capture the key points of the specific attribute area in the standard front direction, that is, obtain the standard key points in the standard space , and its motion parameter information in the original video space: posture rotation matrix , attitude rotation matrix and translation matrices Based on the standard key points and motion parameter information, the image key points of the video frame in the original video space can be calculated as follows:

[0082] , formula (2);

[0083] According to the standard key points captured in the standard space And the calculated image key points in the original video space The deformation estimation module estimates the deformation required to transform the key points of the image in the original video space into the standard space. The deformation estimation calculation process in the deformation estimation module is as follows:

[0084] , formula (3);

[0085] in, represents the pose estimator, Denotes deformation estimation.

[0086] In another implementation, when the specific attribute area is a facial area, the key point capture operation and the deformation estimation operation can be completed using an existing portrait-driven method, such as FaceV2V (facial recognition model) or LivePortrait (controllable portrait video generation framework).

[0087] In one implementation, performing local attribute modulation on the standard spatial feature according to the target attribute feature to generate the modulated attribute feature includes:

[0088] The target attribute feature is a multi-scale attribute modulation vector;

[0089] The standard spatial feature is fused with the multi-scale attribute modulation vector scale by scale through a plurality of cascaded local attribute modulation modules to obtain the modulation attribute feature.

[0090] The target attribute feature in this embodiment is a multi-scale attribute modulation vector, which is modulated with the original spatial feature for multiple times to obtain the modulated attribute feature. Specifically, the attribute modulation vector of each scale corresponds to a different receptive field, which is used to capture the features of the specific attribute area in the reference image at different scale levels, avoiding the risk of detail loss caused by the attribute modulation of a single scale. In order to accurately control the attribute modulation of the attributes of the specific attribute area and realize local attribute modulation, this embodiment pre-constructs a plurality of local attribute modulation modules, and the plurality of local attribute modulation modules are in a cascade relationship, that is, the output of the local attribute modulation module of the upper layer is the input of the local attribute modulation module of the lower layer. Through the cascade of multiple local attribute modulation modules, the multi-scale attribute modulation vector can be fused with the standard spatial feature in stages and scale by scale, so as to more finely control the local attribute modulation effect and avoid errors that may be caused by one-time fusion. Through the cascade of multiple local attribute modulation modules, the video frame features after the specific attributes are replaced in the standard space, that is, the modulated attribute features, can be obtained.

[0091] In one implementation, the target attribute feature is a multi-scale attribute modulation vector; and extracting the target attribute feature based on a reference image includes:

[0092] extracting initial attribute features based on the reference image;

[0093] The initial attribute features are input into a multi-layer perceptron to obtain a multi-scale attribute modulation vector.

[0094] like Figure 2 As shown in the figure, reference image A is input into a specific encoding network. The feature representation output by the encoding network is the initial attribute features of the specific attribute region in reference image A. A multi-layer perceptron then outputs multiple attribute modulation vectors of different scales based on these initial attribute features. Using these multiple attribute modulation vectors of different scales, it is possible to simultaneously balance global consistency and local details in the specific attribute region of the reference image, thereby improving the visual quality of the target video frame after the specific attribute is subsequently replaced.

[0095] In one implementation, the standard spatial feature is fused with the multi-scale attribute modulation vector scale by scale through a plurality of cascaded local attribute modulation modules to obtain the modulation attribute feature, including:

[0096] Inputting the standard spatial features and the attribute modulation vector of the corresponding scale into the first layer local attribute modulation module for fusion;

[0097] The fusion features output by the local attribute modulation module of the previous layer and the attribute modulation vector of the corresponding scale are input into the local attribute modulation module of the next layer for fusion;

[0098] The fusion feature output by the last layer of local attribute modulation module is used as the modulation attribute feature.

[0099] Specifically, each scale of the attribute modulation vector corresponds to a local attribute modulation module. Each local attribute modulation module receives two input data. The two input data of the first-layer local attribute modulation module are the standard spatial features and an attribute modulation vector of the corresponding scale; the two input data of the local attribute modulation modules of other layers are the fused features output by the local attribute modulation module of the previous layer and an attribute modulation vector of the corresponding scale. In the first-layer local attribute adjustment module, the attribute modulation vector is fused with the standard spatial features; in each of the remaining layers of local attribute modulation modules, the attribute modulation vector is fused with the fused features output by the previous layer. By performing local attribute modulation in layers and scale by scale, the multi-scale attribute modulation vector can be fully integrated into the standard spatial features. The fused features output by the last-layer local attribute adjustment module are the final feature representation obtained after modulation by all local attribute modules, which incorporate the features of the target attribute features at different scales. Therefore, the fused features output by the last layer are used as the modulated attribute features.

[0100] In one implementation, the local attribute modulation module includes:

[0101] A mask prediction module outputs a predicted mask based on the standard spatial features; if the mask prediction module is located in the first-layer local attribute modulation module, the predicted mask is output based on the standard spatial features; if the mask prediction module is located in a local attribute modulation module of another layer, the predicted mask is output based on the fused features output by the local attribute modulation module of the previous layer as the updated standard spatial features;

[0102] A first convolution module extracts a first feature based on the standard spatial feature; if the first convolution module is located in a local attribute modulation module of a first layer, the first feature is extracted based on the standard spatial feature; if the first convolution module is located in a local attribute modulation module of another layer, the first feature is extracted based on the fused feature output by the local attribute modulation module of the previous layer as the updated standard spatial feature;

[0103] A second convolution module extracts a second feature based on the standard spatial feature and the attribute modulation vector of the corresponding scale; wherein, if the second convolution module is located in the local attribute modulation module of the first layer, the second feature is extracted based on the standard spatial feature and the attribute modulation vector of the corresponding scale; if the second convolution module is located in the local attribute modulation module of another layer, the fused feature output by the local attribute modulation module of the previous layer is used as the updated standard spatial feature, and the second feature is extracted based on the attribute modulation vector of the corresponding scale;

[0104] The first feature and the second feature are fused according to the predicted mask.

[0105] like Figure 3 As shown, the local attribute modulation module includes a mask prediction module, a first convolution module, and a second convolution module. The mask prediction module may include a convolution layer and an activation function. The mask prediction module can output a predicted mask, which is used to select specific attribute area information of the reference image, so that the target attribute features of the reference image are selectively injected into the specific attribute area of ​​the object to be replaced in the video frame without affecting other non-specific attribute areas. By guiding the fusion of the features output by the two convolution modules through the predicted mask, adaptive local attribute replacement can be achieved, and the naturalness and accuracy of the attribute replacement effect can be improved.

[0106] Specifically, in the first-layer local attribute modulation module, the standard spatial features are input into the prediction mask module, the first convolution module, and the second convolution module. The prediction mask module generates a predicted mask based on the standard spatial features. The first convolution module extracts features based on the standard spatial features and outputs the first feature. The dimension of the standard spatial features is consistent with the dimension of the attribute modulation vector of the corresponding scale. The second convolution module extracts features based on the standard spatial features and the attribute modulation vector of the corresponding scale and outputs the second feature. By guiding the fusion of the first and second features through the predicted mask, feature modulation can be selectively performed during the feature fusion process, more naturally replacing the specific attribute areas of the object to be replaced with the target attribute features, avoiding the impact on non-specific attribute areas (such as background, hairstyle, etc.). The dynamic motion characteristics of the object to be replaced in the original video, such as posture, facial expressions, and oral movements, can be accurately preserved, thereby achieving a high-fidelity attribute replacement effect.

[0107] In the local attribute modulation modules of other layers, the fused features output by the local attribute modulation module of the previous layer are input as the updated standard spatial features into the prediction mask module, the first convolution module, and the second convolution module of the local attribute modulation module of the next layer. The prediction mask module obtains a predicted mask based on the updated standard spatial features; the first convolution module extracts features based on the updated standard spatial features and outputs the first feature; the second convolution module extracts features based on the updated standard spatial features and the attribute modulation vector of the corresponding scale and outputs the second feature. The predicted mask guides the fusion of the first and second features. The fused features output by the local attribute modulation module of the last layer are obtained, that is, the modulation attribute features.

[0108] In one implementation, the convolution weights of the second convolution module are adjusted based on an attribute modulation vector of a corresponding scale.

[0109] Specifically, this embodiment introduces an attribute modulation vector of corresponding scale in the second convolution module to dynamically adjust the convolution weight, so that the convolution operation of the second convolution module pays more attention to the specific attribute area, thereby realizing an adaptive attribute replacement operation.

[0110] For example, the first convolution module in each local attribute modulation module is a standard convolution module, and the second convolution module is a modulated convolution module. In the standard convolution module, a convolution layer is directly used for feature extraction. The feature extraction process of the standard convolution module can be expressed as the following formula:

[0111] , formula (5);

[0112] in, It is the standard spatial feature or the fusion feature output by the local attribute modulation module of the previous layer; is the weight of the convolution kernel; It is the first feature output by the standard convolution module.

[0113] In the modulated convolution module, the attribute modulation vector of the corresponding scale is used to adjust the weight of the convolution layer, and the weight Normalization is performed to prevent unstable output. The feature extraction process of the modulated convolution module can be expressed as the following formula:

[0114] Formula (6);

[0115] in, It is a constant used to ensure numerical stability, and its value is usually very small; is the attribute modulation vector of the corresponding scale; It is the second feature output by the modulated convolution module.

[0116] In one implementation, fusing the first feature and the second feature according to the predicted mask includes:

[0117] Determining weight values ​​corresponding to the first feature and the second feature respectively based on the predicted mask;

[0118] Weighted fusion is performed according to the first feature, the second feature and their corresponding weight values.

[0119] Specifically, the predicted mask can be used to assign different weights to the first and second features. The first and second features are then combined according to the weights assigned by the predicted mask to generate a comprehensive feature representation, known as the fused feature. This weighted fusion approach preserves the complementary information between the two features while dynamically adjusting the importance of each feature, thereby achieving differentiated modulation of different regions.

[0120] For example, the mask prediction module obtains a predicted mask for a specific attribute area, and then uses this predicted mask to dynamically combine the first feature output by the first convolution module and the second feature output by the second convolution module. The combination can be done using a weighted fusion method, such as weighted average fusion. The calculation process of weighted average fusion is expressed as the following formula:

[0121] , formula (7);

[0122] in, is the predicted mask, which is represented as a mask vector of the length of the feature vector, and each element in the mask vector has a value of 1 or 0; is the first feature; is the second characteristic; The predicted mask is dot-multiplied with the second feature vector, and the new vector obtained by subtracting the predicted mask from 1 is dot-multiplied with the first feature vector. The sum of the two is then weighted averaged and fused. This allows the target attribute features to be selectively injected into the specific attribute areas of the object to be replaced in the video frame without affecting other attribute areas.

[0123] Step S300: Inversely transform the modulated attribute features to the original video space to obtain a target video frame, wherein the specific attribute region of the target video frame has the target attribute features of the reference image.

[0124] Specifically, the modulated attribute features are converted from the standard space back to the original video space by inverse deformation, so that the video frame features after the specific attributes in the original video space are replaced can be obtained, and the target video frame can be generated. In the target video frame, the specific attribute area of ​​the object to be replaced has completed the attribute replacement based on the target attribute features of the reference image. The target video frame of each video frame in the original video is obtained, and all the target video frames are combined in sequence to obtain the target video. This embodiment converts the video frame to the standard space to replace the attributes of the specific attribute area, so that the object to be replaced in the original video can accurately retain the motion characteristics of the object to be replaced, such as posture, facial expression, mouth movement and other motion characteristics while replacing the specific attributes.

[0125] In one implementation, inversely transforming the modulation attribute feature to the original video space to obtain a target video frame includes:

[0126] Predicting an inverse deformation estimation from the standard space to the original video space based on the standard key points and the image key points;

[0127] performing inverse deformation on the modulation attribute feature based on the inverse deformation estimation to obtain an updated spatial feature of the original video space;

[0128] The updated spatial features are decoded to obtain a target video frame.

[0129] Specifically, standard keypoints and image keypoints can reflect the positional changes of the same keypoint in different spaces. Therefore, the correspondence between standard keypoints and image keypoints can also predict the inverse deformation estimation required to convert the standard keypoints from the standard space back to the original video space. Modulation attribute features are video frame features that have undergone a specific attribute change in the standard space. Using inverse deformation estimation, the modulation attribute features are converted from the standard space back to the original video space, resulting in video frame features that have undergone a specific attribute change in the original video space, i.e., the updated space features. Finally, the updated space features are decoded to generate the target video frame.

[0130] For example, according to the standard key points captured in the standard space And the calculated image key points in the original video space , the preset deformation estimation module estimates the inverse deformation required to convert the standard key points in the standard space back to the original video space. The deformation estimation module calculates the inverse deformation estimation process as follows:

[0131] , formula (4);

[0132] in, represents the pose estimator, Denotes the inverse deformation estimation. As shown in formula (3) and formula (4), by exchanging the order of standard key points and image key points, the deformation estimation and inverse deformation estimation between the original video space and the standard space can be predicted simultaneously.

[0133] In one implementation, after obtaining the modulation attribute feature, the method further includes:

[0134] The modulation attribute features are repaired through a plurality of three-dimensional convolution blocks with jump connections.

[0135] Specifically, modulation attribute features are the output feature representations after layer-by-layer modulation by multiple local attribute modulation modules, which may result in some information loss. This embodiment uses several 3D convolutional blocks with skip connections to construct a feature restoration module. After obtaining the modulation attribute features, the feature restoration module fine-tunes them to correct any potential information loss, thereby achieving better attribute replacement results.

[0136] To facilitate understanding of the technical solution of the present invention, Figure 2 As shown, an implementation process of the method of the present invention is provided:

[0137] Obtain target attribute features (a) of a specific attribute area based on the reference image A;

[0138] For the video frames in the original video B Perform key point capture and deformation estimation to obtain deformation estimation (d) and inverse deformation estimation (e);

[0139] Through deformation estimation (d), the video frame The original spatial features of the specific attribute area in the image are deformed to the standard space, and local attribute modulation is performed with the target attribute feature (a) to obtain the modulated attribute feature (c);

[0140] After repairing the modulation attribute features (c), the inverse deformation estimation (e) is used to inversely deform the original video space to generate the target video frame. . Target video frame The target attribute features are injected into the specific attribute region of (a), but the motion features of the original video B are retained.

[0141] The advantages of the present invention are:

[0142] 1. The present invention proposes a method for replacing the attributes of a specific attribute region in a video. All video frames in the original video are uniformly deformed to a standard space for specific attribute replacement. In the standard space, the attribute characteristics of the specific attribute region are decoupled from the motion characteristics, thereby preventing the original motion characteristics of the video from being modified or changed during the specific attribute replacement process. After the specific attribute replacement is completed, the video is reversely deformed back to the original video space, so that the target video after the specific attribute replacement can still maintain the motion characteristics of the original video in terms of timing and without jitter.

[0143] 2. This paper proposes a local attribute modulation module. By precisely modulating the specific attribute areas of the object being replaced, it avoids erroneous modifications to non-specific attribute areas such as irrelevant background and hairstyle, and improves the naturalness and quality of the video after the specific attribute replacement.

[0144] The technical effect of the present invention proves that:

[0145] Taking facial attribute replacement in a portrait image as an example, the facial features of a man in a reference image are transferred to the original video to replace a woman's face. This generates a target video with the man's facial features, while areas with non-specific attributes, such as the woman's hairstyle and background, remain unchanged. The present method and other attribute replacement methods were used to replace facial attributes in facial images. Table 1 compares the results of these different attribute replacement methods.

[0146] Table 1. Comparison results of different attribute replacement methods

[0147]

[0148] Among these, SimSwap, FSGAN, DiffSwap, E4S, Face Adapter, and REFace are other attribute replacement methods. Table 1 shows that our method (CanonSwap) significantly outperforms these other attribute replacement methods across multiple quantitative metrics. This demonstrates that our method can achieve more consistent timing, pose, and high-fidelity video facial attribute replacement.

[0149] Based on the above embodiment, the present invention also provides a device for replacing the attributes of a specific attribute area of ​​a video. Figure 4 As shown, the device includes:

[0150] Image processing module 01, used to extract target attribute features based on the reference image;

[0151] The attribute modulation module 02 is used to extract the original spatial features of the specific attribute area for each video frame in the original video, deform the original spatial features into a standard space, and then perform attribute modulation with the target attribute features to obtain modulated attribute features;

[0152] The video frame generation module 03 is configured to reverse-transform the modulated attribute features to the original video space to obtain a target video frame, wherein the specific attribute region of the target video frame has the target attribute features of the reference image.

[0153] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 5 As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for changing the attributes of a specific attribute area of ​​a video is implemented. The display screen of the terminal can be a liquid crystal display or an electronic ink display.

[0154] Those skilled in the art will understand that Figure 5 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0155] In one implementation, the terminal has one or more programs stored in its memory and is configured to be executed by one or more processors. The one or more programs include instructions for performing a method for replacing attributes of a specific attribute area of ​​a video.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware using a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0157] In summary, the present invention discloses a method, device, terminal and medium for replacing the attributes of a specific attribute area of ​​a video, and relates to the technical field of video processing. The method comprises: extracting target attribute features based on a reference image; for each video frame in the original video, extracting the original spatial features of the specific attribute area, deforming the original spatial features to a standard space, and performing attribute modulation with the target attribute features to obtain modulated attribute features; inversely deforming the modulated attribute features to the original video space to obtain a target video frame, wherein the specific attribute area of ​​the target video frame has the target attribute features of the reference image. The present invention effectively decouples the motion features of the replacement object in the video frame from the attribute features of the specific attribute area to be replaced by converting the video frame to a standard space for attribute replacement, thereby avoiding the influence of the attribute replacement process on the motion features, thereby effectively improving the temporal consistency and motion feature alignment of the video before and after the attribute replacement. After the attribute replacement, a target video with high fidelity and strong temporal consistency can be obtained. It is applicable to the technical fields of video processing such as virtual anchors, film and television editing, privacy protection, VR, and AR.

[0158] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for changing the attributes of a specific attribute area of ​​a video, characterized in that: The method comprises: Extracting target attribute features based on a reference image; the target attribute features are multi-scale attribute modulation vectors; For each video frame in the original video, the original spatial features of the specific attribute area are extracted, and the original spatial features are deformed to the standard space and then attribute modulated with the target attribute features to obtain the modulated attribute features, including: capturing the standard key points of the specific attribute area in the video frame in the standard space, and calculating the image key points of the specific attribute area in the original video space; predicting the deformation estimation from the original video space to the standard space based on the standard key points and the image key points; converting the original spatial features based on the deformation estimation to obtain the standard spatial features; performing local attribute modulation on the specific attribute area of ​​the video frame based on the standard spatial features and the target attribute features: through a plurality of cascaded local attribute modulation modules, the standard spatial features are fused with the multi-scale attribute modulation vectors scale by scale to obtain the modulated attribute features; the local attribute modulation module includes: a mask prediction module, which outputs a predicted mask according to the standard spatial features; a first convolution module, which extracts a first feature according to the standard spatial features; a second convolution module, which extracts a second feature according to the standard spatial features and the attribute modulation vector of the corresponding scale; and fusing the first feature and the second feature according to the predicted mask; A target video frame is obtained by inversely deforming the modulated attribute features to the original video space, and a specific attribute region of the target video frame has the target attribute features of the reference image.

2. The method for changing the attributes of a video specific attribute area according to claim 1, characterized in that: Deformation estimation from the original video space to the standard space is predicted based on standard key points and image key points, including: Performing a key point capture operation on the video frame to obtain standard key points corresponding to the specific attribute area in the standard space, and motion parameter information of the specific attribute area; Calculating the image key points corresponding to the specific attribute area in the original video space according to the standard key points and the motion parameter information; Based on the standard key points and the image key points, a deformation estimation from the original video space to the standard space is predicted.

3. The method for changing the attributes of a video specific attribute area according to claim 1, wherein: The target video frame is obtained by inversely deforming the modulation attribute feature to the original video space, including: Predicting an inverse deformation estimation from the standard space to the original video space based on the standard key points and the image key points; performing inverse deformation on the modulation attribute feature based on the inverse deformation estimation to obtain an updated spatial feature of the original video space; The updated spatial features are decoded to obtain a target video frame.

4. The method for changing the attributes of a video specific attribute area according to claim 1, wherein: The standard spatial feature is fused with the multi-scale attribute modulation vector scale by scale through a plurality of cascaded local attribute modulation modules to obtain the modulation attribute feature, including: Inputting the standard spatial features and the attribute modulation vector of the corresponding scale into the first layer local attribute modulation module for fusion; The fusion features output by the local attribute modulation module of the previous layer and the attribute modulation vector of the corresponding scale are input into the local attribute modulation module of the next layer for fusion; The fusion feature output by the last layer of local attribute modulation module is used as the modulation attribute feature.

5. The method for changing the attributes of a video specific attribute area according to claim 4, characterized in that: The local attribute modulation module includes: A mask prediction module outputs a predicted mask based on the standard spatial features; if the mask prediction module is located in the first-layer local attribute modulation module, the predicted mask is output based on the standard spatial features; if the mask prediction module is located in a local attribute modulation module of another layer, the predicted mask is output based on the fused features output by the local attribute modulation module of the previous layer as the updated standard spatial features; A first convolution module extracts a first feature based on the standard spatial feature; if the first convolution module is located in a local attribute modulation module of a first layer, the first feature is extracted based on the standard spatial feature; if the first convolution module is located in a local attribute modulation module of another layer, the first feature is extracted based on the fused feature output by the local attribute modulation module of the previous layer as the updated standard spatial feature; A second convolution module extracts a second feature based on the standard spatial feature and the attribute modulation vector of the corresponding scale; wherein, if the second convolution module is located in the local attribute modulation module of the first layer, the second feature is extracted based on the standard spatial feature and the attribute modulation vector of the corresponding scale; if the second convolution module is located in the local attribute modulation module of another layer, the fused feature output by the local attribute modulation module of the previous layer is used as the updated standard spatial feature, and the second feature is extracted based on the attribute modulation vector of the corresponding scale; The first feature and the second feature are fused according to the predicted mask.

6. The method for changing the attributes of a video specific attribute area according to claim 5, characterized in that: The convolution weights of the second convolution module are adjusted based on the attribute modulation vector of the corresponding scale.

7. The method for changing the attributes of a video specific attribute area according to claim 6, wherein: Fusing the first feature and the second feature according to the predicted mask, comprising: Determining weight values ​​corresponding to the first feature and the second feature respectively based on the predicted mask; Weighted fusion is performed according to the first feature, the second feature and their corresponding weight values.

8. The method for changing the attributes of a video specific attribute area according to claim 1, wherein: After obtaining the modulation attribute feature, the method further includes: The modulation attribute features are repaired through a plurality of three-dimensional convolution blocks with jump connections.

9. A device for replacing attributes of a specific attribute area of ​​a video, characterized in that: The device comprises: An image processing module, configured to extract target attribute features based on a reference image; the target attribute features are multi-scale attribute modulation vectors; The attribute modulation module is used to extract the original spatial features of the specific attribute area for each video frame in the original video, deform the original spatial features to the standard space, and then perform attribute modulation with the target attribute features to obtain modulated attribute features, including: capturing the standard key points of the specific attribute area in the video frame in the standard space, and calculating the image key points of the specific attribute area in the original video space; predicting the deformation estimation from the original video space to the standard space based on the standard key points and the image key points; converting the original spatial features based on the deformation estimation to obtain the standard spatial features; performing local attribute modulation on the specific attribute area of ​​the video frame based on the standard spatial features and the target attribute features: through a plurality of cascaded local attribute modulation modules, the standard spatial features are fused scale by scale with the multi-scale attribute modulation vector to obtain the modulated attribute features; the local attribute modulation module includes: a mask prediction module, which outputs a predicted mask based on the standard spatial features; a first convolution module, which extracts a first feature based on the standard spatial features; a second convolution module, which extracts a second feature based on the standard spatial features and the attribute modulation vector of the corresponding scale; and fusing the first feature and the second feature based on the predicted mask. The video frame generation module is used to reversely transform the modulated attribute features to the original video space to obtain a target video frame, wherein the specific attribute area of ​​the target video frame has the target attribute features of the reference image.

10. A terminal, characterized in that: The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the attribute replacement method of the video specific attribute area as described in any one of claims 1 to 8; and the processor is used to execute the programs.

11. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the attribute replacement method for a video specific attribute area as claimed in any one of claims 1 to 8.

Images