Method and apparatus for generating animation of 3D virtual object, terminal device and medium

Abstract

The application is suitable for the technical field of image processing, and provides a 3D virtual object animation generation method and device, a terminal equipment and a medium; the animation generation method comprises: sequentially performing three-dimensional reconstruction on a reference object in each video frame of a target video to obtain three-dimensional reconstruction information of the reference object in each video frame; generating a first animation sequence corresponding to the reference object according to the three-dimensional reconstruction information of the reference object in all video frames and a preset object model; the preset object model comprises a preset skeleton; performing skeleton reorientation on a target skeleton of a target virtual object based on the preset skeleton to obtain a reoriented skeleton; the structure of the target skeleton is the same as or similar to that of the preset skeleton; performing animation reorientation on the reoriented skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object, thereby improving the animation generation efficiency of the target virtual object.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a method, apparatus, terminal equipment and medium for generating animations of 3D virtual objects. Background Technology

[0002] Currently, three-dimensional (3D) technology is widely used in games, movies, and virtual reality (VR) fields. In the application of 3D technology, it is usually necessary to design 3D virtual objects and a series of actions for these objects based on the application scenario to generate animations. For example, when 3D technology is applied to the gaming field, it is typically necessary to design 3D game characters and a series of actions based on the 3D game scene, and then generate animations for the 3D game characters based on these actions.

[0003] In generating animations for 3D virtual objects, the common approach is to first create key animation frames corresponding to the key actions of the 3D virtual object, and then generate animation frames corresponding to other actions of the 3D virtual object based on the key animation frames. However, the creation of key animation frames requires professional expertise, and the process is complex and time-consuming, reducing the efficiency of 3D virtual object animation generation. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, terminal device, and medium for generating animations of 3D virtual objects, in order to solve the technical problem of low efficiency in the existing generation of animations of 3D virtual objects.

[0005] In a first aspect, embodiments of this application provide a method for generating animations of 3D virtual objects, including:

[0006] The reference object in each frame of the target video is reconstructed in three dimensions in sequence to obtain the three-dimensional reconstruction information of the reference object in each frame of the video.

[0007] A first animation sequence corresponding to the reference object is generated based on the 3D reconstruction information of the reference object in all the video frames and the preset object model; the preset object model includes a preset skeleton.

[0008] Based on the preset skeleton, the target skeleton of the target virtual object is redirected to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton.

[0009] Based on the first animation sequence, the redirected skeleton is re-animated to obtain the second animation sequence corresponding to the target virtual object.

[0010] In an optional implementation of the first aspect, generating the first animation sequence corresponding to the reference object based on the 3D reconstruction information of the reference object in all the video frames and a preset object model includes:

[0011] Get the preset object model;

[0012] Based on the 3D reconstruction information of the reference object in each video frame, the structural parameters of the preset object model in each first animation frame of the first animation sequence are set respectively;

[0013] Based on a preset first animation frame rate, all the first animation frames are sequentially arranged to form the first animation sequence.

[0014] In one optional implementation of the first aspect, the 3D reconstruction information includes mesh point information, skeletal keypoint information, root node position information, and shooting parameter information; the step of setting the structural parameters of the preset object model in each frame of the first animation sequence based on the 3D reconstruction information of the reference object in each video frame includes:

[0015] For each video frame, the spatial position of the preset object model in the corresponding first animation frame is determined based on the shooting parameters of the reference object in the video frame and the position information of the root node.

[0016] Based on the spatial position of the preset object model in the corresponding first animation frame, the position of the root node of the preset object model in the corresponding first animation frame is adjusted;

[0017] Based on the mesh point information and skeletal key point information of the reference object in the video frame, the pose of the preset object model in the corresponding first animation frame is set.

[0018] In an optional implementation of the first aspect, before the skeleton redirection of the target skeleton of the target virtual object based on the preset skeleton, the method further includes:

[0019] The bones to be merged are determined from the preset skeleton, and the bones to be merged are merged to obtain the merged skeleton;

[0020] Obtain the target object model of the target virtual object; the target object model includes an initial skeleton;

[0021] Based on the merged skeleton, the corresponding bones in the initial skeleton are set to obtain the target skeleton of the target virtual object.

[0022] In an optional implementation of the first aspect, the preset object model further includes a preset mesh; the target virtual object model further includes a target mesh; and the step of retargeting the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton includes:

[0023] Match the skeletal key points of the target skeleton with the skeletal key points of the preset skeleton.

[0024] In one optional implementation of the first aspect, the step of sequentially performing 3D reconstruction on the reference object in each frame of the target video to obtain 3D reconstruction information of the reference object in each video frame includes:

[0025] All video frames of the target video are imported frame by frame into a preset 3D reconstruction model for processing, and 3D reconstruction information of the reference object in each video frame is obtained.

[0026] Secondly, embodiments of this application provide an animation generation apparatus for 3D virtual objects, comprising:

[0027] The 3D reconstruction unit is used to sequentially perform 3D reconstruction on the reference object in each video frame of the target video to obtain the 3D reconstruction information of the reference object in each video frame.

[0028] An animation sequence generation unit is used to generate a first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all the video frames and a preset object model; the preset object model includes a preset skeleton.

[0029] The skeleton redirection unit is used to redirect the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton; the structure of the target skeleton is the same as or similar to the structure of the preset skeleton.

[0030] An animation redirection unit is used to redirect the animation of the redirection skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object.

[0031] In one alternative implementation of the second aspect, the animation sequence generation unit includes:

[0032] The first acquisition unit is used to acquire a preset object model;

[0033] The parameter setting unit is used to set the structural parameters of the preset object model in each frame of the first animation sequence based on the three-dimensional reconstruction information of the reference object in each frame of the video frame.

[0034] The sorting unit is used to arrange all the first animation frames in order to form the first animation sequence according to a preset first animation frame rate.

[0035] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the optional methods described in the first aspect above.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any of the optional methods of the first aspect above.

[0037] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to perform the method described in the first aspect or any optional manner of the first aspect.

[0038] The 3D virtual object animation generation method, apparatus, terminal device, computer-readable storage medium, and computer program product provided in this application have the following beneficial effects:

[0039] The 3D virtual character animation generation method provided in this embodiment involves sequentially performing 3D reconstruction on the reference object in each frame of the target video to obtain the 3D reconstruction information of the reference object in each frame. Then, based on the 3D reconstruction information of the reference object in all video frames and a preset object model, a first animation sequence corresponding to the reference object is generated. Since the structure of the preset skeleton of the preset object model is the same as or similar to the structure of the target skeleton of the target virtual object, converting the target video into the corresponding first animation sequence can provide a reference for the subsequent animation simulation of the target virtual object. Afterward, the target skeleton of the target virtual object is first redirected based on the preset skeleton to obtain a redirected skeleton. Then, the redirected skeleton is animated based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object. This allows the motion of the reference object in the target video to be mapped to the virtual object. In other words, this scheme can automatically simulate the motion of the reference object in the target video and obtain the animation sequence of the target virtual object. Compared with the traditional method, it eliminates the need for professionals to create key animation frames for the target virtual object, thereby improving the animation generation efficiency of the target virtual object. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating the implementation of an animation generation method for a 3D virtual object provided in this application embodiment;

[0042] Figure 2 A schematic diagram of a target virtual object provided in an embodiment of this application;

[0043] Figure 3 A schematic diagram illustrating a generation process of a first animation sequence provided in an embodiment of this application;

[0044] Figure 4 A flowchart illustrating the implementation of a method for generating a first animation sequence according to an embodiment of this application;

[0045] Figure 5 A flowchart illustrating the implementation of a skeleton redirection process provided in this application embodiment;

[0046] Figure 6 A schematic diagram illustrating a generation process for a second animation sequence provided in an embodiment of this application;

[0047] Figure 7 This is a schematic diagram of the structure of an animation generation device for 3D virtual objects provided in an embodiment of this application;

[0048] Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0049] It should be noted that the terminology used in the embodiments of this application is only for explaining specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0050] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0051] The 3D virtual object animation generation method provided in this application can be applied to three-dimensional (3D) digital creation or virtual reality (VR) scenes. 3D digital creation can include, but is not limited to, 3D film and television creation or 3D game creation.

[0052] For example, in a 3D game creation scenario, the animation generation method for 3D virtual objects provided in this application can be used to generate animation sequences for 3D virtual objects in the game. For instance, when it is necessary to construct a target 3D virtual character in a 3D game scene, a video containing a character can be recorded first. The posture of the character in different video frames can be different. By performing 3D reconstruction on the character in each video frame, the 3D reconstruction information of the character in each video frame is obtained. Then, based on the 3D reconstruction information of the character in all video frames and a preset object model, a first animation sequence corresponding to the character is generated. Since the skeleton structure of the preset object model is the same as or similar to the skeleton structure of the target 3D virtual character, converting the video into the corresponding first animation sequence can provide a reference for the subsequent construction of the animation model of the target 3D virtual character. Afterward, the target skeleton of the target 3D virtual character is first redirected based on the preset skeleton of the preset object model to obtain a redirected skeleton. Then, the redirected skeleton is animated based on the first animation sequence to obtain a second animation sequence corresponding to the target 3D virtual character, thereby enabling the mapping of the character's movements in the video to a target 3D virtual character in the 3D game scene.

[0053] The executor of the 3D virtual object animation generation method provided in this application embodiment can be a terminal device. Terminal devices can include electronic devices such as mobile phones, tablets, laptops, and desktop computers. Please refer to... Figure 1 , Figure 1 The flowchart illustrates an implementation method for generating animations of 3D virtual objects, as provided in this application embodiment. This method may include steps S101 to S104, detailed below:

[0054] In S101, the reference object in each frame of the target video is reconstructed in three dimensions in sequence to obtain the three-dimensional reconstruction information of the reference object in each frame of the video.

[0055] In a target scenario (e.g., a 3D game creation scenario), when it is necessary to create 3D virtual objects and generate animation sequences of these objects, the terminal device can acquire the target video. The target video can include several video frames, each containing a reference object. The poses of the reference objects in different video frames can be the same or different.

[0056] A reference object refers to the object that the target virtual object references when simulating actions. Therefore, the skeleton structure of the reference object needs to be the same as or similar to the skeleton structure of the target virtual object. The target virtual object can be any 3D virtual object among those to be created. For example, when the target virtual object is a humanoid object (e.g., ...), ... Figure 2 When referring to 21 or 22), the reference object can be a person.

[0057] The target video can be a real-time recorded video containing a reference object, a pre-recorded or produced video containing a reference object, a video downloaded from the internet containing a reference object, or a video containing a reference object obtained through other means. There are no special restrictions on the method of obtaining the target video here.

[0058] After acquiring the target video, the terminal device can perform frame-by-frame processing to obtain the individual video frames within the target video. For example, such as... Figure 3 As shown, if Figure 3 31 in the image represents the target video. After the terminal device performs frame-by-frame processing on the target video 31, it can obtain multiple video frames 32.

[0059] After obtaining each video frame from the target video, the terminal device can sequentially perform 3D reconstruction of the reference object in each video frame. The purpose of performing 3D reconstruction of the reference object in each video frame is to obtain the 3D reconstruction information of the reference object in each video frame.

[0060] The process of performing 3D reconstruction of the reference object in each video frame in sequence refers to performing 3D reconstruction of the reference object in each video frame according to the order in which each video frame is arranged in the target video.

[0061] In one embodiment of this application, the terminal device can use a 3D reconstruction model to process each video frame sequentially to obtain the 3D reconstruction information of the reference object in each video frame. Using a 3D reconstruction model allows for the rapid and efficient acquisition of the 3D reconstruction information of the reference object in each video frame.

[0062] The 3D reconstruction model can be obtained by training a deep learning model using a pre-defined sample dataset. Each data point in the pre-defined sample dataset includes a sample image and its 3D reconstruction information. The sample image contains a reference object. When training the deep learning model using this pre-defined sample dataset, the sample image from each data point is used as the input to the deep learning model, and the 3D reconstruction information of the sample image from each data point is used as the output of the deep learning model. After training, the deep learning model can learn the correspondence between an image containing a reference object and the 3D reconstruction information of the reference object. The terminal device can then identify the trained deep learning model as the 3D reconstruction model.

[0063] As an example rather than a limitation, when the reference object is a person, the 3D reconstruction information of the reference object may include: mesh point information, skeletal key point information, root node information, and shooting parameter information.

[0064] Among them, grid point information is used to describe the outline of the reference object.

[0065] Skeletal keypoint information is used to describe the pose of a reference object. For example, skeletal keypoint information can be the axis-angle information of skeletal keypoints. These skeletal keypoints can include the start and end points of various human bones. The axis-angle information of skeletal keypoints is used to describe the three-dimensional rotation information of the skeletal keypoints relative to their parent nodes.

[0066] The root node information is used to describe the pelvic position of the reference object. The root node information can be described by the coordinates of the reference object's pelvic key points in a preset coordinate system. The preset coordinate system can be set according to actual needs.

[0067] Shooting parameter information describes the camera parameters when the reference object is photographed. Shooting parameter information may include, but is not limited to, the camera's zoom level.

[0068] In this embodiment, after the terminal device obtains the 3D reconstruction information of the reference object in each video frame, it can store the skeletal keypoint information and root node information of the reference object in each video frame in an array, and store the arrays corresponding to all video frames in a first preset file according to the order of each video frame in the target video. As an example and not a limitation, the first preset file can be a PKL format file. PKL format is a file format used by Python software to save files; Python is a computer programming language.

[0069] In S102, a first animation sequence corresponding to the reference object is generated based on the three-dimensional reconstruction information of the reference object in all the video frames and the preset object model; the preset object model includes a preset skeleton.

[0070] A preset object model refers to a standard 3D reference object template. A preset object model can be described by a preset skeleton and a preset mesh. The preset skeleton is the same as or similar to the skeleton of the reference object. For example, if the reference object is a person, the preset object model is a standard 3D human body template.

[0071] It should be noted that the preset object model has a pre-set initial pose. For an example, please refer to [link / reference needed]. Figure 3 , Figure 3 The number 33 in the figure is a standard 3D human body template, and its current pose is the initial pose mentioned above.

[0072] The terminal device can adjust the pose and / or position of a preset object model based on the 3D reconstruction information of the reference object in each video frame, thus obtaining the first animation frame corresponding to each video frame. All the first animation frames corresponding to the video frames together form the first animation sequence. For example... Figure 3 As shown, the terminal device adjusts the pose and / or position of the preset object model 33 according to the 3D reconstruction information of the reference object in each video frame 32 to obtain the first animation frame 34 corresponding to each frame 32. The first animation frames 34 corresponding to all video frames 32 constitute the first animation sequence.

[0073] In one embodiment of this application, S102 may include, for example: Figure 4 S41 to S43 are described in detail below:

[0074] In S41, obtain the preset object model.

[0075] In one possible implementation, a preset object model can be pre-stored in an object model library. Based on this, the terminal device can retrieve the pre-stored preset object model from the object model library.

[0076] In S42, based on the 3D reconstruction information of the reference object in each video frame, the structural parameters of the preset object model in each frame of the first animation sequence are set.

[0077] Specifically, setting the structural parameters of the preset object model in the first animation frame of each frame can be achieved by setting the position and posture of the preset object model in each animation frame.

[0078] In one possible implementation, the terminal device can first generate multiple first animation frames containing a preset object model, the number of which is consistent with the number of video frames included in the target video; then set the initial position of the preset object model in each first animation frame; and then adjust the position and posture of the preset object model in the corresponding first animation frame based on the 3D reconstruction information of each video frame.

[0079] The position information of the preset object in the first animation frame can be determined by the shooting parameters of the reference object and the position information of the root node in the corresponding video frame; the pose of the preset object model can be determined by the mesh point information and skeletal key point information of the reference object in the corresponding video frame.

[0080] In one possible implementation, S42 may include the following steps:

[0081] Step a: For each video frame, determine the spatial position of the preset object model in the corresponding first animation frame based on the shooting parameters and the position information of the root node of the reference object.

[0082] In this implementation, after the terminal device obtains the shooting parameters and root node position information of the reference object in each video frame, it can calculate the spatial position of the preset object model in the first animation frame corresponding to each video frame based on the scaling ratio and root node position information in the shooting parameters and in combination with the weak perspective projection relationship.

[0083] Step b: Adjust the root node position of the preset object model in the first animation frame based on the spatial position of the preset object model in the corresponding first animation frame.

[0084] After obtaining the spatial position of the preset object model in each first animation frame, the terminal device adjusts the position of the root node of the preset object model in each first animation frame so that the position of the root node of the preset object model is consistent with the spatial position of the preset object model in the corresponding first animation frame.

[0085] Step c: Based on the mesh point information and the skeletal key point information, set the pose of the preset object model in the corresponding first animation frame.

[0086] After the terminal device adjusts the spatial position of the preset object model in the first animation frame of each frame, it can adjust the pose of the preset object model in the first animation frame of each frame based on the grid point information and skeletal key point information of the reference object in the corresponding video frame, so that the pose of the preset object model in the first animation frame of each frame is the same as the pose of the reference object model in the corresponding video frame.

[0087] Specifically, the terminal device can first adjust the axial angles of each skeletal key point of the preset object model in each first animation frame based on the skeletal key point information of the reference object in the corresponding video frame, so that the axial angles of each skeletal key point of the preset object model in each first animation frame are the same as the axial angles of each skeletal key point of the reference object model in the corresponding video frame; then, the terminal device can adjust the grid points of the preset object model in each first animation frame based on the grid point information of the reference object in the corresponding video frame, so that the grid points of the preset object model in each first animation frame are the same as the grid points of the reference object model in the corresponding video frame.

[0088] In S43, according to the preset first animation frame rate, all the first animation frames are arranged in sequence to form the first animation sequence.

[0089] In this embodiment, the preset first animation frame rate can be set according to actual needs. For example, the preset first animation frame rate can be the same as the frame rate of the target video, or the preset first animation frame rate can be n times the frame rate of the target video, where n can be an integer greater than 1.

[0090] The terminal device can sort all the first animation frames according to the order of the video frames corresponding to each first animation frame in the target video. The sorted first animation frames constitute the first animation sequence. Specifically, the first animation frame corresponding to the first video frame is taken as the first frame of the first animation sequence, the first animation frame corresponding to the second video frame is taken as the second frame of the first animation sequence, and so on, so that the first animation frames corresponding to the above video frames can be arranged into the first animation sequence in the above order.

[0091] After receiving the first animation sequence, the terminal device can store it in a second preset file. For example, the second preset file can be an FBX format file. FBX format is a file format used by Filmbox software for saving files. Filmbox is software that facilitates the exchange of model, material, motion, and camera information between 3D design software programs.

[0092] The above scheme, by acquiring a preset object model, sets the structural parameters of the preset object in the corresponding first animation frame for each video frame based on the 3D reconstruction information of the reference object in each video frame. This allows for the rapid transfer of the motion and pose of the reference object in the video frame to the preset object in the first animation frame. Subsequently, each of the first animation frames with the set structural parameters is arranged sequentially. All the sorted first animation frames constitute the first animation sequence, thereby mapping the target video to the corresponding animation sequence and rapidly mapping the pose and position of the reference object in each video frame to the preset object model in the corresponding first animation frame. This ensures that the preset object model in each first animation frame has the same pose and position as the reference object in the corresponding video frame.

[0093] In S103, the target skeleton of the target virtual object is redirected based on the preset skeleton to obtain a redirected skeleton; the structure of the redirected skeleton is the same as or similar to the structure of the preset skeleton.

[0094] The target virtual object can be any 3D virtual object that needs to be created in the target scene, such as a 3D virtual character. The target scene can include a 3D game creation scene or a VR scene, etc.

[0095] Each target virtual object corresponds to a default target object model. The target object model can be described by an initial skeleton and an initial mesh volume.

[0096] The terminal device can first determine the target skeleton of the target virtual object based on the preset skeleton and the initial skeleton of each target virtual object, and then perform skeleton redirection on the target skeleton of the target virtual object based on the preset skeleton to obtain the redirected skeleton.

[0097] The structure of the target skeleton is the same as or similar to that of the preset skeleton.

[0098] In this embodiment of the application, the skeleton redirection process for the target skeleton refers to the process of matching the same skeletal key points of the target skeleton with those of the preset skeleton.

[0099] In one embodiment of this application, the terminal device can be via, as... Figure 5 The target skeleton of each target virtual object is obtained through steps S51 to S53, as detailed below:

[0100] In S51, the bones to be merged are determined from the preset skeleton, and the bones to be merged are merged to obtain the merged skeleton.

[0101] In this embodiment, the bones to be merged can be determined based on the actual situation of the preset skeleton. The bones to be merged can be different in different skeletons. This embodiment does not impose any particular limitations on the merging rules and methods for the bones to be merged in the preset skeleton; the skeleton formed by the merged bones can describe the action posture of the reference object.

[0102] In S52, the target object model of the target virtual object is obtained; the target object model includes an initial skeleton.

[0103] In one possible implementation, the target object model of the target virtual object can be pre-stored in a virtual object model library corresponding to the target scene. Based on this, the terminal device can obtain the target object model of the target virtual object from the virtual object model library.

[0104] In S53, the corresponding bones in the initial skeleton are set based on the merged skeleton to obtain the target skeleton of the target virtual object.

[0105] In this embodiment of the application, after the terminal device merges the bones to be merged in the preset skeleton, it can merge the corresponding bones in the initial skeleton based on the merging rules and merging methods of the merged bones, and determine the initial skeleton after bone merging as the target skeleton of the target virtual object.

[0106] In one embodiment of this application, since the naming rules of the skeletal key points of the target skeleton may be different from those of the skeletal key points of the preset skeleton, it is necessary to match the skeletal key points of the target skeleton with the skeletal key points of the preset skeleton in order to correspond the same skeletal key points of the target skeleton with the preset skeleton, thereby realizing the redirection of the target skeleton.

[0107] In S104, the redirected skeleton is re-animated based on the first animation sequence to obtain the second animation sequence corresponding to the target virtual object.

[0108] Animation redirection refers to copying motion data from one skeleton to another. Therefore, animation redirection based on the first animation sequence means copying the motion data of the preset object model in the first animation sequence from the preset skeleton to the redirected skeleton of the target object model, thereby obtaining the second animation sequence of the target object model.

[0109] In this embodiment, the terminal device can first generate multiple frames containing the target object model after skeleton retargeting, wherein the number of second animation frames is the same as the number of first animation frames; then set the initial position of the target object model in each second animation frame, for example, the initial position of the target object model in each second animation frame can be set in the center of each second animation frame; then adjust the action and posture of the target object model in the second animation frame based on the preset position and posture of the object model in each first animation frame.

[0110] The position of the target object model in the second animation frame can be determined by the position of the root node of the preset object model in the corresponding first animation frame; the pose of the target object model can be determined by the pose of the preset object model in the corresponding first animation frame.

[0111] In one possible implementation, S104 may include the following steps:

[0112] Step d: For each second animation frame, adjust the spatial position of the target object model in the corresponding second animation frame according to the spatial position of the preset object model in each first animation frame.

[0113] In this implementation, the terminal device can obtain the spatial position of the preset object model in the first animation frame of each frame, and adjust the spatial position of the target object model in the corresponding second animation frame of each frame so that the spatial position of the preset object model in the first animation frame is the same as the spatial position of the target object model in the second animation frame. In other words, the position of the root node of the preset object in the first animation frame is the same as the position of the root node of the target object model in the second animation frame.

[0114] Step e: For each second animation frame, the terminal device adjusts the pose of the target object model so that the pose of the target object model is the same as that of the reference object model.

[0115] After the terminal device adjusts the spatial position of the target object model in each second animation frame, it can adjust the pose of the target object model in each second animation frame based on the mesh point information and skeletal key point information of the preset object model in the corresponding first animation frame, so that the pose of the target object model in each second animation frame is the same as the pose of the preset object model in the corresponding first animation frame.

[0116] Specifically, the terminal device can first adjust the axial angles of each skeletal key point of the target object model in each second animation frame based on the skeletal key point information of the reference object in the first animation frame corresponding to each second animation frame, so that the axial angles of each skeletal key point of the target object model in each second animation frame are the same as the axial angles of each skeletal key point of the preset object model in the corresponding first animation frame; then, the terminal device can adjust the mesh points of the target object model in each second animation frame based on the mesh point information of the preset object in the first animation frame corresponding to each second animation frame, so that the mesh points of the target object model in each second animation frame are the same as the mesh points of the preset object model in the corresponding first animation frame.

[0117] Step f: According to the preset second first animation frame rate, assemble all the adjusted second first animation frames into the second animation sequence in sequence.

[0118] In this embodiment, the preset second animation frame rate can be set according to actual needs. For example, the preset second animation frame rate can be the same as the frame rate of the first animation, or the preset second animation frame rate can be n times the frame rate of the first animation, where n can be an integer greater than 1.

[0119] The terminal device can sort all the second animation frames according to the order in which each second animation frame corresponds to the first animation frame. All the sorted second animation frames constitute the second animation sequence. Specifically, the second animation frame corresponding to the first animation frame of the first frame is taken as the first frame of the second animation sequence, the second animation frame corresponding to the first animation frame of the second frame is taken as the second frame of the second animation sequence, and so on. This allows the second animation frames corresponding to each of the first animation frames to be arranged into the second animation sequence in the aforementioned order.

[0120] After receiving the second animation sequence, the terminal device can store it in a third preset file. For example, the third preset file can be an FBX format file. FBX format is a file format used by Filmbox software for saving files. Filmbox is software that facilitates the exchange of model, material, motion, and camera information between 3D design software programs.

[0121] For example, such as Figure 6 As shown, if Figure 6 In the example, 611 represents multiple video frames of the target video 61. The first animation frame obtained from each video frame can be 621, and all the first animation frames form the first animation sequence 61. The second animation frame obtained from each first animation frame can be 631, and all the second animation frames 631 form the second animation sequence 63.

[0122] The above scheme involves sequentially reconstructing the reference object in each frame of the target video to obtain the 3D reconstruction information of the reference object in each frame. Then, based on the 3D reconstruction information of the reference object in all video frames and a preset object model, a first animation sequence corresponding to the reference object is generated. Since the structure of the preset skeleton of the preset object model is the same as or similar to the structure of the target skeleton of the target virtual object, converting the target video into the corresponding first animation sequence provides a reference for subsequent animation simulation of the target virtual object. Next, the target skeleton of the target virtual object is first redirected based on the preset skeleton to obtain a redirected skeleton. Then, the redirected skeleton is animated based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object. This allows the motion of the reference object in the target video to be mapped to the virtual object. In other words, this scheme can automatically simulate the motion of the reference object in the target video and obtain the animation sequence of the target virtual object. Compared to traditional methods, it eliminates the need for professionals to create key animation frames for the target virtual object, thus improving the efficiency of animation generation for the target virtual object.

[0123] Based on the 3D virtual object animation generation method provided in the above embodiments, this application further provides a 3D virtual object animation generation apparatus for implementing the above method embodiments. Please refer to [link to relevant documentation]. Figure 7 , Figure 7 A schematic diagram of the structure of an animation generation device for 3D virtual objects provided in an embodiment of this application is shown. Figure 7 As shown, the animation generation device for 3D virtual objects may include: a 3D reconstruction unit 71, an animation sequence generation unit 72, a skeleton redirection unit 73, and an animation redirection unit 74. Wherein:

[0124] The 3D reconstruction unit 71 is used to sequentially perform 3D reconstruction on the reference object in each video frame of the target video to obtain the 3D reconstruction information of the reference object in each video frame.

[0125] The animation sequence generation unit 72 is used to generate the first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all video frames and the preset object model; the preset object model includes a preset skeleton.

[0126] The skeleton redirection unit 73 is used to redirect the target skeleton of the target virtual object based on the preset skeleton to obtain the redirected skeleton; the structure of the redirected skeleton is the same as or similar to the structure of the preset skeleton.

[0127] The animation redirection unit 74 is used to redirect the animation of the redirection skeleton based on the first animation sequence to obtain the second animation sequence corresponding to the target virtual object.

[0128] Optionally, the animation sequence generation unit 72 includes a first acquisition unit, a parameter setting unit, and a sorting unit. Wherein:

[0129] The first acquisition unit is used to acquire a preset object model.

[0130] The parameter setting unit is used to set the structural parameters of the preset object model in each frame of the first animation sequence based on the three-dimensional reconstruction information of the reference object in each video frame.

[0131] The sorting unit is used to arrange all the first animation frames in order to form the first animation sequence according to a preset first animation frame rate.

[0132] Optionally, the parameter setting unit includes a position determination unit, a position adjustment unit, and an attitude adjustment unit. Wherein:

[0133] The position determination unit is used to determine the spatial position of the preset object model in the corresponding first animation frame for each video frame, based on the shooting parameters of the reference object in the video frame and the position information of the root node.

[0134] The position adjustment unit is used to adjust the position of the root node of the preset object model in the corresponding first animation frame based on the spatial position of the preset object model in the corresponding first animation frame.

[0135] The pose adjustment unit is used to set the pose of the preset object model in the corresponding first animation frame based on the grid point information and skeletal key point information of the reference object in the video frame.

[0136] Optionally, the 3D virtual object animation generation device also includes a skeleton merging unit, a second acquisition unit, and a skeleton setting unit. Wherein:

[0137] The skeleton merging unit is used to determine the bones to be merged from the preset skeleton, merge the bones to be merged, and obtain the merged skeleton.

[0138] The second acquisition unit is used to acquire the target object model of the target virtual object; the target object model includes an initial skeleton.

[0139] The skeleton setting unit is used to set the corresponding bones in the initial skeleton based on the merged skeleton to obtain the target skeleton of the target virtual object.

[0140] Optionally, the skeleton redirection unit 73 is specifically used for:

[0141] Match the skeletal key points of the target skeleton with the skeletal key points of the preset skeleton.

[0142] Optionally, the 3D reconstruction unit 71 is specifically used for:

[0143] All video frames of the target video are imported frame by frame into a preset 3D reconstruction model for processing, and 3D reconstruction information of the reference object in each video frame is obtained.

[0144] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. Their specific functions and technical effects can be referred to the method embodiments section, and will not be repeated here.

[0145] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 8 As shown, the terminal device 8 provided in this embodiment may include: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, such as a program corresponding to a 3D virtual object animation generation method. When the processor 80 executes the computer program 82, it implements the steps described above in the embodiment of the 3D virtual object animation generation method, for example... Figure 1 S101 to S104 are shown. Alternatively, when the processor 80 executes the computer program 82, it implements the functions of each module / unit in the above-described embodiment of the 3D virtual object animation generation device, for example... Figure 7 The functions of units 71 to 74 shown.

[0146] For example, computer program 82 can be divided into one or more modules / units, one or more of which are stored in memory 81 and executed by processor 80 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 82 in terminal device 8. For example, computer program 82 can be divided into a 3D reconstruction unit, an animation sequence generation unit, a skeleton retargeting unit, and an animation retargeting unit; the specific functions of each unit are described in [reference needed]. Figure 7 The relevant descriptions in the corresponding embodiments are not repeated here.

[0147] Those skilled in the art will understand that Figure 8 This is merely an example of terminal device 8 and does not constitute a limitation on terminal device 8. It may include more or fewer components than shown, or combine certain components, or use different components.

[0148] The processor 80 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0149] The memory 81 can be an internal storage unit of the terminal device 8, such as a hard disk or RAM of the terminal device 8. The memory 81 can also be an external storage device of the terminal device 8, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card equipped on the terminal device 8. Furthermore, the memory 81 can include both internal and external storage units of the terminal device 8. The memory 81 is used to store computer programs and other programs and data required by the terminal device. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units is merely an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the power supply control device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0152] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0153] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0154] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for generating animations of 3D virtual objects, characterized in that, include: The reference object in each frame of the target video is reconstructed in three dimensions in sequence to obtain the three-dimensional reconstruction information of the reference object in each frame of the video. A first animation sequence corresponding to the reference object is generated based on the 3D reconstruction information of the reference object in all the video frames and the preset object model; the preset object model includes a preset skeleton. Based on the preset skeleton, the target skeleton of the target virtual object is redirected to obtain the redirected skeleton; The structure of the target skeleton is the same as or similar to the structure of the preset skeleton; Based on the first animation sequence, the redirected skeleton is animated and redirected to obtain the second animation sequence corresponding to the target virtual object; The step of generating the first animation sequence corresponding to the reference object based on the 3D reconstruction information of the reference object in all the video frames and the preset object model includes: Get the preset object model; Based on the 3D reconstruction information of the reference object in each video frame, the structural parameters of the preset object model in each first animation frame of the first animation sequence are set respectively; Based on a preset first animation frame rate, all the first animation frames are sequentially arranged to form the first animation sequence. The 3D reconstruction information includes mesh point information, skeletal key point information, root node position information, and shooting parameter information; based on the 3D reconstruction information of the reference object in each video frame, the structural parameters of the preset object model in each frame of the first animation sequence are set, including: For each video frame, the spatial position of the preset object model in the corresponding first animation frame is determined based on the shooting parameters of the reference object in the video frame and the position information of the root node. Based on the spatial position of the preset object model in the corresponding first animation frame, the position of the root node of the preset object model in the corresponding first animation frame is adjusted; Based on the mesh point information and skeletal key point information of the reference object in the video frame, the pose of the preset object model in the corresponding first animation frame is set so that the pose of the preset object model in the first animation frame in each frame is the same as the pose of the reference object in the corresponding video frame.

2. The method for generating animations of 3D virtual objects according to claim 1, characterized in that, Before the skeleton redirection of the target virtual object based on the preset skeleton, the method further includes: The bones to be merged are determined from the preset skeleton, and the bones to be merged are merged to obtain the merged skeleton; Obtain the target object model of the target virtual object; the target object model includes an initial skeleton; Based on the merged skeleton, the corresponding bones in the initial skeleton are set to obtain the target skeleton of the target virtual object.

3. The method for generating animations of 3D virtual objects according to claim 2, characterized in that, The preset object model further includes a preset mesh; the target object model further includes a target mesh; the process of retargeting the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton includes: Match the skeletal key points of the target skeleton with the skeletal key points of the preset skeleton.

4. The method for generating animations of 3D virtual objects according to any one of claims 1 to 3, characterized in that, The step of sequentially performing 3D reconstruction on the reference object in each frame of the target video to obtain 3D reconstruction information of the reference object in each frame of the video includes: All video frames of the target video are imported frame by frame into a preset 3D reconstruction model for processing, and 3D reconstruction information of the reference object in each video frame is obtained.

5. An animation generation device for 3D virtual objects, characterized in that, include: The 3D reconstruction unit is used to sequentially perform 3D reconstruction on the reference object in each video frame of the target video to obtain the 3D reconstruction information of the reference object in each video frame. An animation sequence generation unit is used to generate a first animation sequence corresponding to the reference object based on the three-dimensional reconstruction information of the reference object in all the video frames and a preset object model; the preset object model includes a preset skeleton. The skeleton redirection unit is used to redirect the target skeleton of the target virtual object based on the preset skeleton to obtain a redirected skeleton. The structure of the target skeleton is the same as or similar to the structure of the preset skeleton; An animation redirection unit is used to redirect the animation of the redirection skeleton based on the first animation sequence to obtain a second animation sequence corresponding to the target virtual object; The animation sequence generation unit is specifically used for: Get the preset object model; Based on the 3D reconstruction information of the reference object in each video frame, the structural parameters of the preset object model in each first animation frame of the first animation sequence are set respectively; Based on a preset first animation frame rate, all the first animation frames are sequentially arranged to form the first animation sequence. The 3D reconstruction information includes mesh point information, skeletal key point information, root node position information, and shooting parameter information; the animation redirection unit is specifically used for: For each video frame, the spatial position of the preset object model in the corresponding first animation frame is determined based on the shooting parameters of the reference object in the video frame and the position information of the root node. Based on the spatial position of the preset object model in the corresponding first animation frame, the position of the root node of the preset object model in the corresponding first animation frame is adjusted; Based on the mesh point information and skeletal key point information of the reference object in the video frame, the pose of the preset object model in the corresponding first animation frame is set so that the pose of the preset object model in the first animation frame in each frame is the same as the pose of the reference object in the corresponding video frame.

6. The 3D virtual object animation generation device according to claim 5, characterized in that, The animation sequence generation unit includes: The first acquisition unit is used to acquire a preset object model; The parameter setting unit is used to set the structural parameters of the preset object model in each frame of the first animation sequence based on the three-dimensional reconstruction information of the reference object in each frame of the video frame. The sorting unit is used to arrange all the first animation frames in order to form the first animation sequence according to a preset first animation frame rate.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 4.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Images