Image behavior migration method and device based on operation mirror redirection and storage medium

Abstract

The invention provides an image behavior migration method and device based on operation mirror redirection and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an original video clip to be subjected to image behavior migration; extracting an action track of the original role and a pose sequence of the original camera based on the original video clip; wherein the action track of the original role is represented by using a general role model and a parameter sequence of the general role model; obtaining a target role model, and assigning a value to the target role model based on the parameter sequence of the general role model, the target role model and the general role model having the same skeleton topology; obtaining a target camera, and performing pose optimization on the target camera based on the pose sequence of the original camera to obtain a pose sequence of the target camera; and redirecting the pose sequence of the target camera to the target role model to generate a migrated video clip corresponding to the original video clip, so that the risk of semantic mismatching of a visual effect presented by a role under an image migration behavior can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to an image behavior migration method and device based on lens movement redirection and a storage medium. BACKGROUND

[0002] With the rapid development of digital media technology, the demand for visual content in the film and game industries is growing. In these fields, image behaviors such as camera movements and character actions play a key role in storytelling and visual composition, and are key factors that affect user immersion and emotional experience.

[0003] In order to improve production efficiency and reduce production costs, it has become a trend to migrate image behaviors between different projects, scenes or characters. Current deep learning-based methods can only extract camera movements and character actions and mechanically apply them to characters of different sizes when extracting and migrating image behaviors, resulting in a mismatch between the semantics of the visual effects of the new character and the migrated image behaviors after video generation based on image behavior migration. SUMMARY

[0004] To solve the problems in the prior art, the present application provides an image behavior migration method and device based on lens movement redirection and a storage medium.

[0005] The present application provides an image behavior migration method based on lens movement redirection, comprising: obtaining an original video segment to be subjected to image behavior migration; extracting a motion trajectory of an original character and a pose sequence of an original camera based on the original video segment; wherein the motion trajectory of the original character is represented using a general character model and a parameter sequence of the general character model; obtaining a target character model, assigning values to the target character model based on the parameter sequence of the general character model, wherein the skeleton topology of the target character model and the general character model is the same; obtaining a target camera, optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model, and obtaining a pose sequence of the target camera; generating a migration video segment corresponding to the original video segment based on the pose sequence of the target camera and the target character model after the assignment.

[0006] According to the image behavior migration method based on lens movement redirection provided by the present application, the pose sequence of the original camera is extracted based on the original video segment, comprising: extracting the parameter sequence of the general character model corresponding to the original character and the projection parameters of the joint nodes of the original character based on the original video segment, and determining the parameter sequence and the projection parameters of the first frame of video. solving a first frame original camera pose according to the parameter sequence and the projection parameter of the first frame video frame; determining each frame original camera pose based on the first frame original camera pose to determine the pose sequence of the original camera.

[0007] According to the image behavior migration method based on lens movement redirection provided by the application, the determination of each frame original camera pose based on the first frame original camera pose comprises: determining the first frame original camera pose as a current frame original camera pose, and determining a next frame original camera predicted pose based on the current frame original camera pose; obtaining a predicted projection parameter of the parameter sequence of the general character model corresponding to the original character in the next frame video frame in the camera plane of the next frame original camera predicted pose, determining a first difference degree of the predicted projection parameter and a projection parameter of a joint node of the original character extracted from the next frame video frame through a first loss function; rendering a character projection mask of the parameter sequence of the general character model corresponding to the original character in the next frame video frame in the camera picture of the next frame original camera predicted pose, and determining a second difference degree of the predicted character projection mask and a character projection mask of the original character extracted from the next frame video frame through a second loss function; determining a next frame original camera pose based on the next frame original camera predicted pose through the first difference degree and the second difference degree; determining the next frame original camera pose as a current frame original camera pose, and repeating the process of determining a next frame original camera predicted pose based on the current frame original camera pose to obtain each frame original camera pose.

[0008] According to the image behavior migration method based on lens movement redirection provided by the application, before optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model, the method further comprises: extracting semantic consistent key points of the general character model and the target character model based on bone sampling; wherein the generation of the semantic consistent key points is irrelevant to the mesh; establishing a dense mapping between the general character model and the target character model through the semantic consistent key points; the optimization of the pose sequence of the original camera based on the mapping between the target character model and the general character model comprises: optimizing the pose sequence of the original camera based on the dense mapping between the target character model and the general character model.

[0009] According to the image behavior migration method based on lens movement redirection provided by the application, the pose sequence of the original camera is optimized based on the dense mapping between the target role model and the general role model, and the method comprises the following steps: Determine the semantic consistent key point visibility mask of the video frame corresponding to each pose in the pose sequence of the original camera. Based on the semantic consistent key point visibility mask, determine the semantic consistent key point visibility loss value of the video frame corresponding to each pose in the pose sequence of the original camera through a visibility loss function, and determine the visible position loss value of the video frame corresponding to each pose in the pose sequence of the original camera through a visible position loss function. Based on the semantic consistent key point visibility loss value and the visible position loss value, optimize the pose of the target camera.

[0010] According to the image behavior migration method based on lens movement redirection provided by the application, the pose sequence of the original camera is optimized based on the dense mapping between the target role model and the general role model, and the method comprises the following steps: Freeze the rotation parameters of the pose sequence of the original camera, and optimize the remaining parameters of the pose sequence of the original camera based on the dense mapping between the target role model and the general role model.

[0011] The application further provides an image behavior migration device based on lens movement redirection, comprising the following modules: The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the image behavior migration method based on lens movement redirection as described above when executing the computer program.

[0012] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the image behavior migration method based on lens movement redirection as described above.

[0013] The application further provides a computer program product comprising a computer program, wherein the computer program is executable by a processor to implement the image behavior migration method based on lens movement redirection as described above.

[0014] The present invention provides a method, apparatus, and storage medium for image behavior transfer based on camera movement redirection. By separately extracting the motion trajectory of the original character and the pose sequence of the original camera, and further optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model, the present invention can achieve visually semantically consistent camera behavior transfer between characters of different body types. This effectively reduces the risk of semantic mismatch in the visual effects of the new character under the transferred image behavior after image behavior transfer based on camera movement redirection. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the image behavior transfer method based on camera movement redirection provided by the present invention.

[0017] Figure 2 This is a schematic diagram of the image behavior transfer device based on camera movement redirection provided by the present invention.

[0018] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The technical solutions of the present invention will now be described with reference to the accompanying drawings in the embodiments of the present invention.

[0021] Figure 1 This is a flowchart illustrating the image behavior transfer method based on camera movement redirection provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 101: Obtain the original video clips for which image behavior transfer is to be performed.

[0022] There are many ways to obtain the original video clips to be transferred in image behavior, such as downloading from public video libraries or shooting custom clips yourself. This invention does not limit the methods.

[0023] Among them, image behavior refers to the key behaviors that adjust the narrative and visual composition of images. For example, images can be obtained from game production and film special effects shooting, and image behavior can be camera movement or camera movement and character actions, etc.

[0024] Image behavior transfer refers to extracting image behaviors from one video clip and applying those behaviors to another video clip. Image behavior transfer typically involves transferring the image behaviors of one character to a new character.

[0025] Here, the original video clip refers to the video clip that provides image behavior, and the corresponding migrated video clip refers to the video clip that applies image behavior to the original video clip. The original video clip is a licensed video clip.

[0026] Step 102: Extract the motion trajectory of the original character and the pose sequence of the original camera based on the original video clip; wherein, the motion trajectory of the original character is represented using a general character model and the parameter sequence of the general character model.

[0027] There are many ways to extract the motion trajectory of the original character based on the original video clip, such as human body mesh restoration method, pose estimation algorithm, etc., and this invention does not limit this.

[0028] There are many ways to extract the pose sequence of the original camera based on the original video clips. For example, it can be done through Structure from Motion (SfM), Simultaneous Localization and Mapping (SLAM), etc. This invention does not limit the methods.

[0029] Here, the original character refers to the character in the original video clip who is related to the video action; the original character can be a single person in the original video clip. The motion trajectory of the original character refers to the movement path of the original character's joints.

[0030] The raw camera refers to the virtual camera that captures and renders the movements of 3D characters in the original video footage; it can also be called a virtual camera. The pose sequence of the raw camera, also known as the raw camera motion, refers to a time-ordered collection of the spatial states of the raw camera, used to describe the movement trajectory and viewpoint changes of the raw camera in a 3D scene.

[0031] A generic character model refers to a standardized, parameterized model representing a 3D character. For example, a generic character model can be a generic linearly skinned multi-people linear model (SMPL). The parameter sequence of a generic character model refers to a set of time-series ordered parameters that can reconstruct the character's form. Understandably, a time-series-based parameter sequence enables the generic character model to express the 3D motion trajectory of the original character in the original video clip.

[0032] Step 103: Obtain the target character model and assign values ​​to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; Obtain the target camera and optimize the pose sequence of the original camera based on the mapping between the target character model and the general character model to obtain the pose sequence of the target camera.

[0033] The target character model refers to a 3D model with the same skeletal topology as the general character model.

[0034] After obtaining the target character model, the joints of the target character model's skeleton topology can be assigned values ​​based on the parameter sequence of the joints in the skeleton topology of the general character model, so that the parameter sequence of the target character model is the same as that of the general character model, thereby transferring the character actions of the original character in the original video clip to the new character.

[0035] A target camera refers to a virtual camera that captures and renders the movements of 3D characters within a video clip; it can also be called a virtual camera. The pose sequence of a target camera, also known as target camera motion, refers to a time-ordered set of spatial states of the target camera, used to describe its motion trajectory and viewpoint changes within a 3D scene.

[0036] Step 104: Generate a migration video segment corresponding to the original video segment based on the pose sequence of the target camera and the assigned target character model.

[0037] The camera motion of the aforementioned target camera is obtained by optimizing the mapping between the target character model and the general character model. Therefore, adapting the pose sequence of the target camera to the new target character model can ensure the visual semantic consistency between the transferred video clip and the original video clip from the camera's perspective. This ensures that even if there are differences in body size between the target character and the original character, the image of the target character under the target camera's movement can maintain a similar composition and visual effect as the image of the original character under the original camera's movement.

[0038] The image behavior transfer method based on camera repositioning provided in this invention extracts the motion trajectory of the original character and the pose sequence of the original camera separately, and further optimizes the pose sequence of the original camera based on the mapping between the target character model and the general character model to obtain the pose sequence of the target camera. This method can achieve visually semantically consistent camera behavior transfer between characters of different body types, effectively reducing the risk of semantic mismatch in the visual effects of the new character under the transferred image behavior after image behavior transfer based on camera repositioning.

[0039] Based on the above embodiments, extracting the pose sequence of the original camera from the original video clip includes: Based on the original video clip, extract the parameter sequence of the general character model corresponding to the original character and the projection parameters of the joints of the original character, and determine the parameter sequence and projection parameters of the first video frame; Based on the parameter sequence and projection parameters of the first video frame, the original camera pose of the first frame is determined to form the original camera pose sequence. Based on the original camera pose of the first frame, the original camera pose of each frame is determined frame by frame to determine the pose sequence of the original camera.

[0040] Among them, the projection parameter refers to the two-dimensional projection position of the original character's joints on the video frame.

[0041] Specifically, the parameter sequence of the generic character model corresponding to the original character can determine the 3D motion of the generic character model, and thus determine the 3D coordinates of the joints of the original character. The projection parameters of the joints of the original character can be the 2D coordinates of the joints of the original character. In this way, a PnP (Perspective-n-Point) problem can be constructed based on the parameter sequence and projection parameters, and the first frame of the original camera pose sequence can be automatically calculated using mathematical methods.

[0042] Determining the original camera pose of each frame based on the original camera pose of the first frame means: using the original camera pose of the first frame as the initial value for the second frame, and optimizing the original camera pose of the second frame; using the original camera pose determined in the second frame as the initial value for the third frame, and optimizing the original camera pose of the third frame, and so on. This leverages the characteristic that the original camera moves continuously in the original video clip, and the original camera pose changes relatively little between adjacent video frames. This ensures that the calculation of the original camera pose for each frame is based on adjustments to the original camera pose of the previous frame, accelerating optimization convergence and reducing error accumulation.

[0043] Understandably, compared to the first frame of original camera pose obtained through manual calibration, determining the parameter sequence of the general character model corresponding to the original character in the entire video segment, and then determining the three-dimensional coordinates of the original character's joints, results in a more accurate first frame of original camera pose.

[0044] Based on this, the error in the optimization process of determining the original camera pose of each frame frame by frame based on the original camera pose of the first frame is also smaller, which can improve the accuracy of the obtained original camera pose sequence as a whole.

[0045] Based on any of the above embodiments, the step of determining the original camera pose of each frame frame by frame based on the original camera pose of the first frame includes: The original camera pose of the first frame is determined as the original camera pose of the current frame, and the predicted original camera pose of the next frame is determined based on the original camera pose of the current frame. The parameter sequence of the general character model corresponding to the original character in the next video frame is obtained as the predicted projection parameters of the camera plane of the original camera's predicted pose in the next video frame. A first difference between the predicted projection parameters and the projection parameters of the joints of the original character extracted in the next video frame is determined by a first loss function. The parameter sequence of the general character model corresponding to the original character in the next video frame is rendered as a character projection mask of the camera image of the original camera's predicted pose in the next video frame. A second difference between the predicted character projection mask and the character projection mask of the original character extracted from the next video frame is determined by a second loss function. Based on the predicted pose of the original camera in the next frame, the original camera pose in the next frame is determined by optimization using the first difference degree and the second difference degree; The original camera pose of the next frame is determined to be the original camera pose of the current frame. The process of determining the predicted original camera pose of the next frame based on the original camera pose of the current frame is repeated to obtain the original camera pose of each frame.

[0046] Specifically, the original camera pose of the current frame can be used as the basis for initializing the original camera pose of the next frame, and the predicted pose of the original camera pose of the next frame can be determined by prediction based on the initialized original camera pose of the next frame.

[0047] As mentioned earlier, the parameter sequence of the general character model can determine the three-dimensional motion of the general character model. In this way, the predicted projection parameters of the original character in the camera plane of the predicted pose of the original camera in the next frame can be determined by the three-dimensional motion of the original character. The predicted projection parameters can be the predicted two-dimensional coordinates of the joints of the three-dimensional motion of the original character in the camera plane of the predicted pose of the original camera in the next frame.

[0048] As mentioned earlier, the projection parameters of the joints of the original character in the next video frame can be the two-dimensional coordinates of the joints of the original character.

[0049] In this way, the first difference between the predicted projection parameters and the projection parameters of the joints of the original character in the next frame can be determined by the first loss function, thereby optimizing the predicted pose of the original camera in the next frame by the difference in the joint position information.

[0050] For example, a trained differentiable Dynamic Neural Radiance Field (DNeRF) renderer can be used to render the parameter sequence of the general character model corresponding to the original character in the next video frame, thereby obtaining the character projection mask of the camera image of the original camera's predicted pose in the next frame.

[0051] There are many ways to extract the character projection mask of the original character from the next video frame, such as through segmentation models, motion detection segmentation, etc., and this invention does not limit this method.

[0052] In this way, the second loss function can determine the second difference between the character projection mask of the original character in the camera image of the original camera's predicted pose in the next video frame and the character projection mask of the original character in the next video frame, thereby optimizing the original camera's predicted pose in the next frame by the difference in the composition of the image.

[0053] Understandably, by using the first degree of difference in the joint position information and the second degree of difference in the image composition, the three-dimensional motion trajectory and character movements of the original character can be captured and tracked more robustly from multiple dimensions of joints and overall appearance, thereby improving the accuracy of the next frame original camera pose determination based on the predicted pose of the next frame original camera.

[0054] Specifically, after obtaining the original camera pose of the next frame, the original camera pose of the next frame can be used as the original camera pose of the current frame to initialize the predicted pose of the original camera of the next frame. The process of optimizing the predicted pose of the original camera of the next frame by the first difference degree of the joint position information part and the second difference degree of the image composition part to obtain the original camera pose of the next frame is repeated to obtain the original camera pose of each frame.

[0055] Furthermore, in one embodiment, based on the predicted pose of the next frame's original camera, the original camera pose of the next frame is determined by optimization using the first difference degree and the second difference degree, including: Determine the weighted sum of the first difference and the second difference; The original camera pose for the next frame is determined based on the predicted pose of the original camera in the next frame and the weighted sum optimization.

[0056] It is understandable that in the process of optimizing the predicted pose of the next frame's original camera, the influence of the matching accuracy of the joint positions and the visual consistency of the overall appearance of the character on the optimization process can be adjusted by adjusting the weighting factors of the first and second differences, so as to adapt to the needs of camera pose estimation in different scenarios.

[0057] After obtaining the 3D character motion and the pose sequence of the original camera based on the general character model using the aforementioned method, in order to achieve character-centric camera repositioning, based on any of the above embodiments, before optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model, the method further includes: Semantically consistent key points are extracted from the general character model and the target character model based on skeletal sampling; wherein, the generation of the semantically consistent key points is independent of the mesh. A dense mapping between the general role model and the target role model is established using the semantically consistent key points; The optimization of the original camera pose sequence based on the mapping between the target character model and the general character model includes: The pose sequence of the original camera is optimized based on the dense mapping between the target character model and the general character model.

[0058] Among them, the skeleton of the general character model can be uniformly sampled to obtain sampling key points, and multiple rays can be emitted outward from each sampling key point at uniformly spaced angles on a plane perpendicular to the skeleton. Semantically consistent sensors (SCS) can be obtained through the intersection of each ray and the mesh of the general character model.

[0059] The method for obtaining the semantically consistent keypoints of the target character model is the same as the method for obtaining the semantically consistent keypoints of the general character model, and will not be repeated here. Specifically, the interval for uniformly sampling the skeleton of the general character model is the same as the interval for uniformly sampling the target character model, and the uniform interval of the angles at which multiple rays emanate outward from the sampled keypoints of the general character model are the same as the uniform interval of the angles at which multiple rays emanate outward from the sampled keypoints of the target character model.

[0060] It is understandable that establishing a dense mapping between the general character model and the target character model through semantically consistent key points can improve the accuracy of 3D perception of character movements through the semantically consistent key points of the dense mapping, so that the camera movements before and after the migration can still maintain semantic consistency from the camera's perspective, even when there are large differences in character size and other aspects.

[0061] Based on any of the above embodiments, optimizing the pose sequence of the original camera based on the dense mapping between the target character model and the general character model includes: Determine the semantically consistent keypoint visibility mask for each pose in the original camera pose sequence. Based on the semantically consistent keypoint visibility mask, the semantically consistent keypoint visibility loss value of the video frame corresponding to each pose in the original camera pose sequence is determined by the visibility loss function, and the visible position loss value of the video frame corresponding to each pose in the original camera pose sequence is determined by the visible position loss function. The pose of the target camera is optimized based on the semantically consistent keypoint visibility loss value and the visibility position loss value.

[0062] The semantically consistent keypoint visibility mask is a binary mask used to indicate whether semantically consistent keypoints in a video frame are visible within the camera's field of view.

[0063] For example, taking the general character model as the general skinned multi-people linear model (SMPL), the semantically consistent keypoint visibility mask for each pose in the video frame can be calculated and determined using the following formula: Usable Representing the target character model, determining the projection parameters of the target character model in the corresponding pose of the original camera pose sequence can also be called the reprojection position. ,in In Used to indicate the first i Whether the camera is facing forward or backward in each frame of the video. Specifically, how is this determined? Indicates the first i Whether the camera faces forward or backward in a video frame can be determined according to actual needs; this invention does not impose any limitations on this.

[0064] Based on this, it is possible to base on and The visibility loss value of the semantically consistent keypoints in the video frame corresponding to each pose in the original camera pose sequence is determined by the visibility loss function.

[0065] For example, the visibility loss function could be: in, h For the first i The height of a video frame. w For the first i The width of a video frame.

[0066] In one embodiment, the visible boundary and its horizontal center can be determined by a semantically consistent keypoint visibility mask, and a visible position loss function can be constructed based on the visible boundary and its horizontal center to determine the visible position loss value of the video frame corresponding to each pose in the original camera pose sequence.

[0067] For example, the visible location loss function can be: in, It is the first i The minimum reprojection of visible semantically consistent keypoints in a video frame. x coordinate, It is the first i The largest reprojection of visible semantically consistent keypoints in a video frame. x coordinate, It is the first i The minimum reprojection of visible semantically consistent keypoints in a video frame. y coordinate, It is the first i The largest reprojection of visible semantically consistent keypoints in a video frame. y coordinate.

[0068] Understandably, based on establishing a dense mapping between the general character model and the target character model, the visibility loss function can control the visible semantically consistent key points to not exceed the target camera's field of view in order to maintain their visibility. The visible position loss function can further constrain the projection position of the semantically consistent key points. In this way, the target camera behavior obtained by co-optimizing the original camera behavior through the visibility loss function and the visible position loss function helps to maintain visual semantic consistency under the viewpoint when there is a large difference in body size between the original character and the new character.

[0069] If the predicted pose of the original camera in the next frame is determined directly based on the original camera pose in the current frame, and the predicted pose of the original camera in the next frame is optimized to determine the original camera pose in the next frame, it may cause the camera pose to rotate excessively, resulting in the view of the target camera on the target character model being different from the view of the original camera on the original character model.

[0070] To address this issue, based on any of the above embodiments, the pose sequence of the original camera is optimized based on the dense mapping between the target character model and the general character model, including: The rotation parameters of the original camera pose sequence are frozen, and the remaining parameters of the original camera pose sequence are optimized based on the dense mapping between the target character model and the general character model.

[0071] Freezing the rotation parameters of the original camera's pose sequence can be achieved by directly assigning the rotation parameters of the original camera's pose sequence to the pose sequence of the target camera, so that the rotation parameters of the target camera's pose sequence and the original camera's pose sequence are assigned the same values.

[0072] The remaining parameters may include the camera's translation parameters in three dimensions, such as forward, backward, left, right, up, and up.

[0073] It is understandable that the optimization of excessive camera pose rotation is difficult to correct in subsequent optimization steps. By freezing the rotation parameters of the original camera pose sequence, the optimization of the original camera pose sequence can avoid excessive camera pose rotation, thereby increasing the accuracy of the optimized target camera pose sequence.

[0074] To illustrate the functionality of the image behavior transfer method based on camera repositioning provided in this implementation, a specific example is given below.

[0075] Obtain the original video segment for which image behavior transfer is to be performed; extract the motion trajectory of the original character based on the original video segment; the motion trajectory of the original character is represented using a general character model and the parameter sequence of the general character model; Based on the original video clip, extract the parameter sequence of the general character model corresponding to the original character in the first video frame and the projection parameters of the joints of the original character; solve for the first frame original camera pose of the original camera pose sequence based on the parameter sequence and the projection parameters; determine the first frame original camera pose as the current frame original camera pose, and determine the next frame original camera predicted pose based on the current frame original camera pose; obtain the predicted projection parameters of the parameter sequence of the general character model corresponding to the original character in the next video frame on the camera plane of the next frame original camera predicted pose, and determine the predicted projection parameters and extract the projection of the joints of the original character in the next video frame using a first loss function. The parameters are firstly differentiable; the parameter sequence of the general character model corresponding to the original character in the next frame of the video frame is rendered as the character projection mask of the camera image of the original camera predicted pose in the next frame; the second difference between the predicted character projection mask and the character projection mask of the original character extracted from the next frame of the video frame is determined by a second loss function; based on the original camera predicted pose of the next frame, the original camera pose of the next frame is optimized and determined by the first difference and the second difference; the original camera pose of the next frame is determined as the original camera pose of the current frame, and the process of determining the original camera predicted pose of the next frame based on the original camera pose of the current frame is repeated to obtain the original camera pose of each frame, so as to determine the pose sequence of the original camera; A target character model is obtained, and values ​​are assigned to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; a target camera is obtained, and semantically consistent keypoints are extracted from the general character model and the target character model based on skeleton sampling; wherein the generation of the semantically consistent keypoints is independent of the mesh; a dense mapping is established between the general character model and the target character model through the semantically consistent keypoints; a visibility mask of the semantically consistent keypoints of the video frames corresponding to each pose in the pose sequence of the original camera is determined; based on the visibility mask of the semantically consistent keypoints, a visibility loss value of the semantically consistent keypoints of the video frames corresponding to each pose in the pose sequence of the original camera is determined by a visibility loss function, and a visible position loss value of the video frames corresponding to each pose in the pose sequence of the original camera is determined by a visible position loss function; the pose of the target camera is optimized based on the visibility loss value of the semantically consistent keypoints and the visible position loss value to obtain the pose sequence of the target camera; wherein the rotation parameters of the pose sequence of the original camera are frozen, and the remaining parameters of the pose sequence of the original camera are optimized; Based on the pose sequence of the target camera and the assigned target character model, a migration video segment corresponding to the original video segment is generated.

[0076] The image behavior transfer method based on camera repositioning provided in this invention extracts the original character's motion trajectory and the original camera's pose sequence separately, and then performs camera repositioning and transfer of camera behavior centered on the character, which can provide visually consistent results for characters of various body types.

[0077] Furthermore, when transferring camera behavior, semantic consistency keypoints are introduced to establish a dense mapping between the general character model and the target character model with different body sizes. Based on this, the pose of the target camera is optimized through the visibility loss function and the visibility position loss function to transfer camera behavior. This helps to maintain visual semantic consistency under the viewpoint when there is a large difference in body size between the original character and the new character.

[0078] Furthermore, in this embodiment of the invention, constructing a new dataset facilitates the creation and rendering of characters using a non-3D engine approach. There are many ways to create and render characters using a non-3D engine based on the new dataset, such as developing new toolchains or extending existing tools; this invention does not limit this approach. The new dataset includes 68 cinematic shots and 10 high-quality characters with different body types. These high-quality characters with different body types are bound to SMPL skeleton topologies. There are many ways to bind SMPL skeleton topologies to characters; this invention does not limit this approach.

[0079] The following describes the image behavior transfer device based on camera redirection provided by the present invention. The image behavior transfer device based on camera redirection described below can be referred to in correspondence with the image behavior transfer method based on camera redirection described above.

[0080] Figure 2 This is a schematic diagram of the image behavior transfer device based on camera movement redirection provided by the present invention, as shown below. Figure 2 As shown, the device includes: The original video segment acquisition module 201 is used to acquire the original video segment to be transferred for image behavior. The original image behavior acquisition module 202 is used to extract the motion trajectory of the original character and the pose sequence of the original camera based on the original video clip; wherein, the motion trajectory of the original character is represented by a general character model and the parameter sequence of the general character model; The migration image behavior acquisition module 203 is used to acquire a target character model, assign values ​​to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; acquire a target camera, optimize the pose sequence of the original camera based on the mapping between the target character model and the general character model, and obtain the pose sequence of the target camera; The migration video segment acquisition module 204 is used to generate a migration video segment corresponding to the original video segment based on the pose sequence of the target camera and the assigned target character model.

[0081] Based on any of the above embodiments, the original image behavior acquisition module 202 is specifically used for: Based on the original video segment, extract the parameter sequence of the general character model corresponding to the original character in the first video frame and the projection parameters of the joints of the original character; The first frame of the original camera pose sequence is determined by solving the parameter sequence and the projection parameters. Based on the original camera pose of the first frame, the original camera pose of each frame is determined frame by frame to determine the pose sequence of the original camera.

[0082] Based on any of the above embodiments, the original image behavior acquisition module 202 is specifically used for: The original camera pose of the first frame is determined as the original camera pose of the current frame, and the predicted original camera pose of the next frame is determined based on the original camera pose of the current frame. The parameter sequence of the general character model corresponding to the original character in the next video frame is obtained as the predicted projection parameters of the camera plane of the original camera's predicted pose in the next video frame. A first difference between the predicted projection parameters and the projection parameters of the joints of the original character extracted in the next video frame is determined by a first loss function. The parameter sequence of the general character model corresponding to the original character in the next video frame is rendered as a character projection mask of the camera image of the original camera's predicted pose in the next video frame. A second difference between the predicted character projection mask and the character projection mask of the original character extracted from the next video frame is determined by a second loss function. Based on the predicted pose of the original camera in the next frame, the original camera pose in the next frame is determined by optimization using the first difference degree and the second difference degree; The original camera pose of the next frame is determined to be the original camera pose of the current frame. The process of determining the predicted original camera pose of the next frame based on the original camera pose of the current frame is repeated to obtain the original camera pose of each frame.

[0083] Based on any of the above embodiments, the apparatus further includes a dense mapping establishment module, used for: Semantically consistent key points are extracted from the general character model and the target character model based on skeletal sampling; wherein, the generation of the semantically consistent key points is independent of the mesh. A dense mapping between the general role model and the target role model is established using the semantically consistent key points; The migration image behavior acquisition module 203 is specifically used for: The pose sequence of the original camera is optimized based on the dense mapping between the target character model and the general character model.

[0084] Based on any of the above embodiments, the migration image behavior acquisition module 203 is specifically used for: Determine the semantically consistent keypoint visibility mask for each pose in the original camera pose sequence. Based on the semantically consistent keypoint visibility mask, the semantically consistent keypoint visibility loss value of the video frame corresponding to each pose in the original camera pose sequence is determined by the visibility loss function, and the visible position loss value of the video frame corresponding to each pose in the original camera pose sequence is determined by the visible position loss function. The pose of the target camera is optimized based on the semantically consistent keypoint visibility loss value and the visibility position loss value.

[0085] Based on any of the above embodiments, the migration image behavior acquisition module 203 is specifically used to freeze the rotation parameters of the pose sequence of the original camera and optimize the remaining parameters of the pose sequence of the original camera based on the dense mapping between the target character model and the general character model.

[0086] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other through the communications bus 340. The processor 310 can call logical instructions in the memory 330 to execute an image behavior transfer method based on camera movement redirection. This method includes: acquiring an original video segment to be transferred; extracting the motion trajectory of an original character and the pose sequence of an original camera based on the original video segment; wherein the motion trajectory of the original character is represented using a general character model and a parameter sequence of the general character model; acquiring a target character model and assigning values ​​to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; acquiring a target camera and optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model to obtain a pose sequence of the target camera; and generating a transferred video segment corresponding to the original video segment based on the pose sequence of the target camera and the assigned target character model.

[0087] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image behavior transfer method based on camera repositioning provided by the above methods. The method includes: acquiring an original video segment to be transferred; extracting the motion trajectory of an original character and the pose sequence of an original camera based on the original video segment; wherein the motion trajectory of the original character is represented using a general character model and a parameter sequence of the general character model; acquiring a target character model and assigning values ​​to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; acquiring a target camera and optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model to obtain the pose sequence of the target camera; and generating a transferred video segment corresponding to the original video segment based on the pose sequence of the target camera and the assigned target character model.

[0089] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image behavior transfer method based on camera repositioning provided by the above methods. The method includes: acquiring an original video segment for which image behavior transfer is to be performed; extracting the motion trajectory of an original character and the pose sequence of an original camera based on the original video segment; wherein the motion trajectory of the original character is represented using a general character model and a parameter sequence of the general character model; acquiring a target character model and assigning values ​​to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; acquiring a target camera and optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model to obtain a pose sequence of the target camera; and generating a transferred video segment corresponding to the original video segment based on the pose sequence of the target camera and the assigned target character model.

[0090] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for image behavior transfer based on camera movement redirection, characterized in that, include: Obtain the original video clips for which image behavior transfer will be performed; The motion trajectory of the original character and the pose sequence of the original camera are extracted based on the original video clips; wherein, the motion trajectory of the original character is represented using a general character model and the parameter sequence of the general character model; Obtain a target character model, and assign values ​​to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; obtain a target camera, and optimize the pose sequence of the original camera based on the mapping between the target character model and the general character model to obtain the pose sequence of the target camera; Based on the pose sequence of the target camera and the assigned target character model, a migration video segment corresponding to the original video segment is generated.

2. The image behavior transfer method based on camera movement redirection according to claim 1, characterized in that, Extracting the pose sequence of the original camera based on the original video clips includes: Based on the original video clip, extract the parameter sequence of the general character model corresponding to the original character and the projection parameters of the joints of the original character, and determine the parameter sequence and projection parameters of the first video frame; Based on the parameter sequence and projection parameters of the first video frame, the original camera pose of the first frame is determined to form the original camera pose sequence. Based on the original camera pose of the first frame, the original camera pose of each frame is determined frame by frame to determine the pose sequence of the original camera.

3. The image behavior transfer method based on camera movement redirection according to claim 2, characterized in that, The step of determining the original camera pose of each frame based on the original camera pose of the first frame includes: The original camera pose of the first frame is determined as the original camera pose of the current frame, and the predicted original camera pose of the next frame is determined based on the original camera pose of the current frame. The parameter sequence of the general character model corresponding to the original character in the next video frame is obtained as the predicted projection parameters of the camera plane of the original camera's predicted pose in the next video frame. A first difference between the predicted projection parameters and the projection parameters of the joints of the original character extracted in the next video frame is determined by a first loss function. The parameter sequence of the general character model corresponding to the original character in the next video frame is rendered as a character projection mask of the camera image of the original camera's predicted pose in the next video frame. A second difference between the predicted character projection mask and the character projection mask of the original character extracted from the next video frame is determined by a second loss function. Based on the predicted pose of the original camera in the next frame, the original camera pose in the next frame is determined by optimization using the first difference degree and the second difference degree; The original camera pose of the next frame is determined to be the original camera pose of the current frame. The process of determining the predicted original camera pose of the next frame based on the original camera pose of the current frame is repeated to obtain the original camera pose of each frame.

4. The image behavior transfer method based on camera movement redirection according to claim 1, characterized in that, Before optimizing the pose sequence of the original camera based on the mapping between the target character model and the general character model, the method further includes: Semantically consistent key points are extracted from the general character model and the target character model based on skeletal sampling; wherein, the generation of the semantically consistent key points is independent of the mesh. A dense mapping between the general role model and the target role model is established using the semantically consistent key points; The optimization of the original camera pose sequence based on the mapping between the target character model and the general character model includes: The pose sequence of the original camera is optimized based on the dense mapping between the target character model and the general character model.

5. The image behavior transfer method based on camera movement redirection according to claim 4, characterized in that, Optimizing the pose sequence of the original camera based on the dense mapping between the target character model and the general character model includes: Determine the semantically consistent keypoint visibility mask for each pose in the original camera pose sequence. Based on the semantically consistent keypoint visibility mask, the semantically consistent keypoint visibility loss value of the video frame corresponding to each pose in the original camera pose sequence is determined by the visibility loss function, and the visible position loss value of the video frame corresponding to each pose in the original camera pose sequence is determined by the visible position loss function. The pose of the target camera is optimized based on the semantically consistent keypoint visibility loss value and the visibility position loss value.

6. The image behavior transfer method based on camera movement redirection according to claim 1 or 4, characterized in that, Optimizing the pose sequence of the original camera based on the dense mapping between the target character model and the general character model includes: The rotation parameters of the original camera pose sequence are frozen, and the remaining parameters of the original camera pose sequence are optimized based on the dense mapping between the target character model and the general character model.

7. An image behavior transfer device based on camera movement redirection, characterized in that, include: The original video clip acquisition module is used to acquire the original video clips to be used for image behavior transfer. The original image behavior acquisition module is used to extract the motion trajectory of the original character and the pose sequence of the original camera based on the original video clip; wherein, the motion trajectory of the original character is represented using a general character model and the parameter sequence of the general character model; The migration image behavior acquisition module is used to acquire a target character model, assign values ​​to the target character model based on the parameter sequence of the general character model, wherein the target character model and the general character model have the same skeleton topology; acquire a target camera, and optimize the pose sequence of the original camera based on the mapping between the target character model and the general character model to obtain the pose sequence of the target camera; The migration video clip acquisition module is used to generate a migration video clip corresponding to the original video clip based on the pose sequence of the target camera and the assigned target character model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the image behavior transfer method based on camera repositioning as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the image behavior transfer method based on camera repositioning as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the image behavior transfer method based on camera repositioning as described in any one of claims 1 to 6.

Images