Stylized object driving method and device, medium and electronic device

Abstract

The present disclosure provides a driving method for a stylized object, a driving device for a stylized object, a computer-readable medium and an electronic device, and relates to the field of image processing technology. The method comprises: intercepting a neutral target object with a neutral expression in a video containing a target object, and calculating a characteristic deformation parameter based on the neutral target object; reconstructing a stylized neutral target object based on a neutral preset object, a characteristic deformation parameter and a first deformation parameter; generating a stylized target expression base based on the neutral preset object, the characteristic deformation parameter and the second deformation parameter; and driving the stylized neutral target object based on the video and the stylized target expression base. The present disclosure can avoid the step of requiring a designer to specially design an expression base when driving different stylized objects, thereby reducing the design cost; at the same time, it can realize the driving of stylized objects with "thousands of faces for thousands of people", improve the driving effect, and enhance the scalability and portability.

Description

Background Art

[0002] 3D face modeling is a very important technology and is widely used in movies, games, virtual reality, online social networking, face recognition and tracking, etc. Expression-driven is an application direction of 3D face reconstruction. A 3D face model is established by designing a parametric face model, fusing a deformable expression base, and face reconstruction. After completing the reconstruction of the 3D face model, it is usually necessary to solve the label parameters and posture parameters of each frame image to drive the face model in real time so that the shape, expression, and posture of the 3D face model are consistent with the input image.

[0003] However, in the related driving technology, only the expression driving of real human faces or the driving of stylized faces such as fixed cartoon faces can be performed. If different stylized faces need to be driven, the designer needs to design a fusion deformation expression base specifically for each stylized face, which increases the design cost; at the same time, if a new stylized face needs to be added, the fusion deformation expression base needs to be redesigned, which has poor scalability and portability. Summary of the invention

[0004] The purpose of the present disclosure is to provide a stylized object method, a stylized object device, a computer-readable medium and an electronic device, thereby reducing the design cost at least to a certain extent and improving the scalability and portability.

[0005] According to a first aspect of the present disclosure, a method for driving a stylized object is provided, comprising: intercepting a neutral target object in a video containing a target object, in which the target object is a neutral expression, and calculating characteristic deformation parameters based on the neutral target object; wherein the characteristic deformation parameters include deformation parameters for converting a neutral preset object into the neutral target object; reconstructing a stylized neutral target object based on the neutral preset object, the characteristic deformation parameters and a first deformation parameter; wherein the first deformation parameter includes deformation parameters for converting the neutral preset object into a stylized neutral preset object; generating a stylized target expression base based on the neutral preset object, the characteristic deformation parameters and the second deformation parameters; wherein the second deformation parameters include deformation parameters for converting the neutral preset object into the stylized preset expression base; and driving the stylized neutral target object based on the video and the stylized target expression base.

[0006] According to a second aspect of the present disclosure, a driving device for a stylized object is provided, comprising: a parameter determination module, for intercepting a neutral target object with a neutral expression in a video containing a target object, and calculating characteristic deformation parameters based on the neutral target object; wherein the characteristic deformation parameters include deformation parameters for converting a neutral preset object into a neutral target object; a stylized reconstruction module, for reconstructing a stylized neutral target object based on the neutral preset object, the characteristic deformation parameters and a first deformation parameter; wherein the first deformation parameter includes deformation parameters for converting the neutral preset object into a stylized neutral preset object; an expression base generation module, for generating a stylized target expression base based on the neutral preset object, the characteristic deformation parameters and a second deformation parameter; wherein the second deformation parameter includes deformation parameters for converting the neutral preset object into a stylized preset expression base; and an object driving module, for driving the stylized neutral target object based on the video and the stylized target expression base.

[0007] According to a third aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: a processor; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0009] A driving method for a stylized object provided by an embodiment of the present disclosure calculates characteristic deformation parameters of a neutral target object intercepted in a video, then reconstructs a stylized neutral object target based on a neutral preset object, characteristic deformation parameters and a first deformation parameter, generates a stylized target expression base based on the neutral preset object, characteristic deformation parameters and a second deformation parameter, and then drives the stylized neutral target object based on the video and the stylized target expression base. The embodiment of the present disclosure can automatically reconstruct a stylized neutral target object corresponding to the target object in the video based on the neutral target object in the video, and automatically generate a stylized target expression base corresponding to the target object in the video, thereby avoiding the need for a designer to specifically design an expression base when driving different stylized objects, thereby reducing the design cost; at the same time, by driving based on the stylized neutral target object corresponding to the target object in the video and the stylized target expression base, the driving of stylized objects with "thousands of faces" can be realized, thereby improving the driving effect and enhancing the scalability and portability of the stylized object driving.

[0010] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0012] Figure 1 A schematic diagram showing an exemplary system architecture to which embodiments of the present disclosure may be applied;

[0013] Figure 2 A flowchart schematically illustrates a method for driving a stylized object in an exemplary embodiment of the present disclosure;

[0014] Figure 3 A flowchart schematically illustrates a method for acquiring a neutral target object in an exemplary embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram of a graphical user interface for reminding a user to maintain a neutral expression in an exemplary embodiment of the present disclosure is schematically shown;

[0016] Figure 5 A schematic diagram of the structure of a characteristic parameter model in an exemplary embodiment of the present disclosure is schematically shown;

[0017] Figure 6 A schematic diagram schematically illustrates a principle of calculating a first deformation parameter in an exemplary embodiment of the present disclosure;

[0018] Figure 7 A flowchart schematically illustrates a method for reconstructing a stylized neutral target object in an exemplary embodiment of the present disclosure;

[0019] Figure 8 A schematic diagram schematically illustrates a principle of determining a first stylization parameter in an exemplary embodiment of the present disclosure;

[0020] Fig. 9 A schematic diagram schematically illustrating two expression bases in an exemplary embodiment of the present disclosure;

[0021] Fig.10 A schematic diagram schematically illustrates a principle of calculating N groups of second deformation parameters in an exemplary embodiment of the present disclosure;

[0022] Fig.11 A flowchart schematically illustrates a method for generating a stylized target expression base in an exemplary embodiment of the present disclosure;

[0023] Fig.12 A schematic diagram schematically illustrates a principle of determining a second stylization parameter in an exemplary embodiment of the present disclosure;

[0024] Fig.13 A flowchart of a method for driving a stylized neutral target object based on a video and a stylized target expression base in an exemplary embodiment of the present disclosure is schematically shown;

[0025] Fig.14 A flowchart schematically illustrates another method for driving a stylized object in an exemplary embodiment of the present disclosure;

[0026] Fig.15 A flowchart of a method for reconstructing a stylized neutral target face in an exemplary embodiment of the present disclosure is schematically shown;

[0027] Fig.16 A flowchart schematically illustrates another method for generating a stylized target expression base in an exemplary embodiment of the present disclosure;

[0028] Fig.17 A flowchart of a method for driving a stylized neutral target face in an exemplary embodiment of the present disclosure is schematically shown;

[0029] Fig.18 A schematic diagram schematically shows the composition of a driving device for stylized objects in an exemplary embodiment of the present disclosure;

[0030] Fig.19 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0033] Figure 1 A schematic diagram of a system architecture of an exemplary application environment to which a method and apparatus for driving a stylized object according to an embodiment of the present disclosure can be applied is shown.

[0034] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with image and video processing functions, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0035] The driving method of the stylized object provided in the embodiment of the present disclosure is generally executed in the terminal devices 101, 102, and 103, and accordingly, the driving device of the stylized object is generally arranged in the terminal devices 101, 102, and 103. However, it is easy for those skilled in the art to understand that the driving method of the stylized object provided in the embodiment of the present disclosure can also be executed by the server 105 or by the terminal devices 101, 102, 103 and the server 105, and accordingly, the driving device of the stylized object can also be arranged in the server 105 or the terminal devices 101, 102, 103 and the server 105, which is not particularly limited in the present exemplary embodiment.

[0036] For example, in an exemplary embodiment, the terminal devices 101, 102, and 103 may capture a video containing a target object and send it to the server 105 through the network 104. The server 105 may capture a neutral target object in the video, reconstruct a stylized neutral target object, and generate a stylized target expression base. The stylized neutral target object may then be driven based on the video and the stylized target expression base, and the driving result may be sent to the terminal devices 101, 102, and 103 for display. Alternatively, after reconstructing the stylized neutral target object and generating the stylized target expression base in the server 105, the stylized neutral target object and the stylized target expression base may be directly sent to the terminal devices 101, 102, and 103, so that the terminal devices 101, 102, and 103 may drive the stylized neutral target object based on the video and the stylized target expression base, and display the driving result.

[0037] In the related art, usually a designer designs a fixed number of character models, and then drives the fixed data character models directly by collecting performers' data.

[0038] For example, in an exemplary embodiment, the action script design of the cartoon character is completed according to the needs of the plot, and then a virtual 3D character is created through three-dimensional software, and imported into the graphics engine to generate a character model; then the action of the performer with Markers pasted on the face, limb joints and torso is captured by a motion capture camera, and the image sequence is saved, and then analyzed and processed to identify the Markers therein, and calculate their spatial positions at each moment, thereby obtaining their motion trajectory. Afterwards, the obtained Marker motion trajectory is imported into the 3D character model through action assignment and action generation software, and the 3D character model is driven to move, and an animation sequence consistent with the motion trajectory is obtained. Although this method can be driven directly by capturing the performer's actions, it can still only be driven based on the created fixed character model, and the driving effect of "thousands of faces" cannot be achieved.

[0039] Based on one or more of the above problems, this example embodiment provides a method for driving a stylized object. Figure 2 As shown, the driving method of the stylized object may include the following steps S210 to S240:

[0040] In step S210, a neutral target object with a neutral expression is captured from a video containing a target object, and a feature deformation parameter is calculated based on the neutral target object.

[0041] Among them, the characteristic deformation parameters include deformation parameters for converting a neutral preset object into a neutral target object, that is, they can characterize the characteristic attributes of the neutral target object relative to the neutral preset object. It should be noted that, usually, in the process of 3D face reconstruction, the deformation matrix usually includes a 3*3 dimensional affine parameter matrix or a projection parameter matrix, so the characteristic deformation parameters may include 9*V dimensional parameters; wherein V represents the number of vertices of the 3D model. In addition, in specific application scenarios, different deformation matrices can be set according to different requirements, and no special limitation is made to this.

[0042] Among them, the target object may include the face of a living object such as a person or an animal contained in the video; a neutral expression refers to an expression without a clear emotional tendency; and a neutral target object includes a target object in a neutral expression state.

[0043] By calculating the characteristic deformation parameters, the features of the target object contained in the video can be extracted, and based on the characteristic deformation parameters, the features of the target object contained in the video can be represented in the stylized neutral target object and the stylized target expression base, laying the foundation for the "thousand faces for thousands of people" driving effect.

[0044] In an exemplary embodiment, when a target object is captured as a neutral target object with a neutral expression in a video containing a target object, a more accurate neutral target object can be obtained by target object detection. Figure 3 As shown, it may include step S310 and step S320:

[0045] In step S310, target object detection is performed on each video frame in the video to determine characteristic attributes of the target object in each video frame.

[0046] The characteristic attribute of the target object in each video frame may include the expression attribute of the target object in the video frame. In addition, the characteristic attribute may also include other attributes of the target object in the video frame to assist the expression attribute in obtaining a complete neutral target object that is convenient for calculating characteristic deformation parameters.

[0047] For example, the feature attributes may include the posture attributes of the target object in the video frame, which are used to ensure whether the target object in the video frame is facing the camera module that shoots the video, so that the acquired neutral target object is a complete target object; for another example, the feature attributes may include the size attributes of the target object in the video frame, which are used to determine whether the target object in the video frame meets the size required for calculating the feature deformation parameters; for another example, the feature attributes may also include the pixel attributes of the target object, which are used to determine whether the clarity of the video frame meets the clarity required for calculating the feature deformation parameters.

[0048] In step S320, it is determined that the target object is a neutral video frame with a neutral expression based on the feature attribute, and the target object in the neutral video frame is intercepted to obtain a neutral target object.

[0049] In an exemplary embodiment, after obtaining the characteristic attributes of each video frame, it is possible to determine which video frame has a target object with a neutral expression according to the characteristic attributes, and then determine the video frame as a neutral video frame, and directly intercept the target object based on the neutral video frame to obtain a neutral object. By performing target object detection on the target object, the position of the target object in the video frame can be more accurately determined, and then the expression attribute of the target object in the video frame can be determined, so as to accurately determine which video frame in the video has a neutral expression.

[0050] It should be noted that, in order to obtain a neutral target object that is convenient for calculating feature deformation parameters, a corresponding threshold value may be set for the feature attribute, so as to judge whether each video frame meets the attribute for subsequent calculation of feature deformation parameters according to the threshold value, and then screen the video frames with neutral expressions of the target object to determine the neutral video frames for neutral target object capture. For example, when the target object is a human face, a size threshold value may be set to judge whether the size of the face is greater than the minimum size, or less than the maximum size, etc.

[0051] In an exemplary embodiment, when intercepting a target object, the target object can also be detected based on an object detection algorithm, and the coordinate information corresponding to the target object obtained by the object detection can be retained to accurately intercept the neutral target object in the video frame. Wherein, when the target object is different, the object detection algorithm can be set differently. For example, when the target object is a face, the coordinate information of the face frame can be determined based on face detection, and then the face frame can be intercepted; or the coordinate information of the face key points can be determined based on face key point detection, and then the face can be intercepted based on the key point coordinate information.

[0052] In an exemplary embodiment, in addition to the above-mentioned method, reference is made to Figure 4 As shown, a reminder can also be generated on the graphical user interface for acquiring the video to remind the user within a period of time during video recording that a video frame with a neutral expression of the target object needs to be captured to ensure that a neutral target object can be acquired in the video.

[0053] In addition, in a video, there may be a situation where multiple video frames can be used as neutral video frames. In this case, any video frame can be directly selected as a neutral video frame, and characteristic deformation parameters can be calculated based on the video frame; multiple video frames can also be used as neutral video frames, deformation parameters can be calculated based on each neutral video frame, and then the deformation parameters calculated from multiple video frames can be averaged, and the final average value can be used as the final characteristic deformation parameter; other methods can also be used for processing according to actual needs.

[0054] In an exemplary embodiment, when calculating characteristic deformation parameters based on the neutral target object, the calculation can be performed by parameter correspondence, or directly by machine learning or deep learning such as convolutional neural network. For example, characteristic parameters of a preset neutral preset object and characteristic parameters of a neutral target object can be extracted, and then the two characteristic parameters can be compared to obtain deformation parameters for converting the neutral preset object into the neutral target object.

[0055] In an exemplary embodiment, the neutral target object may be input into the feature parameter model to obtain the deformation parameters for converting the neutral preset object into the neutral target object. Figure 5 As shown, the feature parameter model can be a deep neural network composed of a network trunk and a deformation parameter regression layer. The intercepted neutral target object is input into the deep neural network, that is, the feature deformation parameters of the predetermined dimension are output.

[0056] Among them, the network trunk and deformation parameter regression layer of the deep neural network can be set according to specific needs. For example, the trunk of the deep neural network can include the network trunks commonly used in convolutional neural networks such as MobileNet, ResNet, and Xception; if 9*V-dimensional feature deformation parameters are required, the output dimension of the deep neural network regression layer can be set to 9*V. In addition, when training the deep neural network, supervised regression training can be performed using losses such as three-dimensional object vertices and object key points.

[0057] In addition, in order to facilitate processing of the neutral target object, the neutral target object may be preprocessed before calculating the characteristic deformation parameters based on the neutral target object. Specifically, the preprocessing process may include image reconstruction processing, image enhancement processing, normalization processing and other processing processes.

[0058] For example, when calculating feature deformation parameters based on the feature parameter model, the image of the neutral target object can be reconstructed to the size required by the feature parameter model, and then input into the feature parameter model to calculate the feature deformation parameters; for another example, the image of the neutral target object can be enhanced to improve the clarity of the image of the neutral target object; for another example, the image of the neutral target object can be normalized to normalize the pixel values ​​corresponding to each pixel point on the captured image of the neutral target object to facilitate calculation.

[0059] In step S220, a stylized neutral target object is reconstructed based on the neutral preset object, the characteristic deformation parameter and the first deformation parameter.

[0060] The first deformation parameter includes the deformation parameter of converting the neutral preset object into the stylized neutral preset object, that is, it can characterize the characteristic attributes of the stylized neutral preset object relative to the neutral preset object. Figure 6 As shown, the first deformation parameter can be obtained by calculating based on the neutral preset object and the stylized neutral preset object based on an iterative method such as Newton's method and the least square method. It should be noted that since the neutral preset object and the stylized neutral preset object are fixed models designed in advance by the designer, the first deformation parameter can be calculated in advance and stored in the execution subject, and can be directly obtained and used when reconstructing the stylized neutral target object to speed up the calculation.

[0061] It should be noted that, generally, when a neutral preset object is deformed, the deformation matrix generally includes a 3*3 dimensional affine parameter matrix or a projection parameter matrix, so the first deformation parameter may also include a 9*V dimensional parameter; wherein V represents the number of vertices of the three-dimensional model. In addition, in specific application scenarios, different deformation matrices may be set according to different requirements, and no special limitation is made to this.

[0062] In an exemplary embodiment, when reconstructing a stylized neutral target object based on the neutral preset object, the characteristic deformation parameter and the first deformation parameter, referring to Figure 7 As shown, the following steps S710 and S720 may be included:

[0063] In step S710, the characteristic deformation parameter and the first deformation parameter are combined to obtain a first stylized parameter.

[0064] Among them, since the characteristic deformation parameter can represent the characteristic attribute of the neutral target object relative to the neutral preset object, the first deformation parameter can represent the characteristic attribute of the stylized neutral preset object relative to the neutral preset object. Figure 8 As shown, the characteristic deformation parameter and the first deformation parameter can be combined to obtain the first stylized parameter that can characterize the characteristic attribute of the stylized neutral target object relative to the neutral preset object, that is, the first stylized parameter includes the deformation parameter that converts the neutral preset object into the stylized neutral target object.

[0065] It should be noted that, in order to facilitate the combination of the characteristic deformation parameter and the first deformation parameter, it is usually necessary to set the characteristic deformation parameter and the first deformation parameter to a corresponding format.

[0066] For example, when the calculation method used by the characteristic deformation parameter and the first deformation parameter to deform the neutral preset object is multiplication and division, the characteristic deformation parameter and the first deformation parameter can be set to a format that can be multiplied. Specifically, assuming that a vertex of the neutral preset object is a, the deformation transformation of the vertex by the characteristic deformation parameter is a*matrix 1, and the deformation transformation of the vertex by the first deformation parameter is a / matrix 2. At this time, if the characteristic deformation parameter and the first deformation parameter are both in matrix format, it is necessary to make the matrix format used by the two capable of multiplication calculation. For another example, when the calculation method used by the characteristic deformation parameter and the first deformation parameter to deform the neutral preset object is addition and subtraction, the characteristic deformation parameter and the first deformation parameter can be set to a format that can be added. Assuming that a vertex of the neutral preset object is b, the deformation transformation of the vertex by the characteristic deformation parameter is b+matrix 3, and the deformation transformation of the vertex by the first deformation parameter is b-matrix 4. At this time, if the characteristic deformation parameter and the first deformation parameter are both in matrix format, the matrix formats used by the two need to be such that addition calculation can be performed.

[0067] In step S720, the neutral preset object is deformed based on the first stylized parameter to obtain a stylized neutral target object.

[0068] Specifically, after obtaining the first stylized parameter, the neutral preset object can be directly deformed based on the first stylized parameter to obtain a stylized neutral target object. For example, for vertex a in the above example, the first stylized parameter deforms the vertex to a*matrix1 / matrix2; for another example, for vertex b in the above example, the first stylized parameter deforms the vertex to b+matrix3-matrix4.

[0069] In step S230, a stylized target expression base is generated based on the neutral preset object, the characteristic deformation parameter and the second deformation parameter.

[0070] Among them, the second deformation parameter includes the deformation parameter of converting the neutral preset object into the stylized preset expression base, that is, it can characterize the characteristic attributes of the stylized preset expression base relative to the neutral preset object. In the specific calculation, the second deformation parameter can be obtained by calculating based on the neutral preset object and the stylized preset expression base and the iterative method such as Newton's method. At the same time, since the neutral preset object and the stylized preset expression base are also fixed models involved in advance by the designer, the second deformation parameter can also be calculated in advance and stored in the execution subject, and when generating the stylized target expression base, it can be directly obtained and used to speed up the calculation.

[0071] It should be noted that the stylized preset expression base includes the expression base required to drive the neutral preset object. Therefore, when designers design the stylized preset expression base, they need to design and produce it according to the neutral preset object.

[0072] In addition, when driving, the driving may require multiple expression bases as a basis, so the second deformation parameter may also include multiple groups. For example, Fig. 9 Two expression bases are shown as an example, from left to right, they are left eye closed and right eye closed. At this time, the second deformation parameters may include two groups, namely, the first group of second deformation parameters corresponding to the left eye closed and the second group of second deformation parameters corresponding to the right eye closed; when calculating multiple groups of second deformation parameters, it is necessary to calculate each stylized preset expression base based on the preset neutral face to obtain the corresponding second deformation parameters of each stylized preset expression base. For example, referring to Fig.10 As shown, N groups of second deformation parameters can be obtained by performing calculations on the neutral preset object (3*V) and N stylized preset expression bases (3*V) respectively.

[0073] It should be noted that, under normal circumstances, when a neutral preset object is deformed, the deformation matrix usually includes a 3*3 dimensional affine parameter matrix or a projection parameter matrix, so the second deformation parameter may also include a 9*V dimensional parameter; wherein V represents the number of vertices of the three-dimensional model; in addition, in specific application scenarios, different deformation matrices can be set according to different requirements, and no special limitation is made to this; when multiple expression bases are required as the basis for driving, designers need to decouple the expression bases from each other when designing stylized preset expression bases to avoid mutual influence.

[0074] In an exemplary embodiment, when generating a stylized target expression base based on a neutral preset object, a characteristic deformation parameter, and a second deformation parameter, referring to Fig.11 As shown, the following steps S1110 and S1120 may be included:

[0075] In step S1110, the characteristic deformation parameter and the second deformation parameter are combined to obtain a second stylized parameter.

[0076] Among them, since the characteristic deformation parameter can represent the characteristic attribute of the neutral target object relative to the neutral preset object, the second deformation parameter can represent the characteristic attribute of the stylized preset expression base relative to the neutral preset object. Fig.12 As shown, the characteristic deformation parameter and the second deformation parameter can be combined to obtain the second stylized parameter that can characterize the characteristic attributes of the stylized target expression base relative to the neutral preset object, that is, the second stylized parameter includes the deformation parameter that converts the neutral preset object into the stylized target expression base.

[0077] It should be noted that, in order to facilitate the combination of the characteristic deformation parameter and the second deformation parameter, it is usually necessary to set the characteristic deformation parameter and the second deformation parameter to a corresponding format. The specific details have been described in detail in the implementation method of combining the characteristic deformation parameter and the first deformation parameter. The undisclosed details can be referred to the implementation method content of this part, so it will not be repeated here.

[0078] In step S1120, the neutral preset object is deformed based on the second stylized parameter to obtain a stylized target expression base.

[0079] Specifically, after obtaining the second stylized parameter, the neutral preset object may be directly deformed based on the second stylized parameter to obtain a stylized target expression base.

[0080] In step S240 , the stylized neutral target object is driven based on the video and the stylized target expression base.

[0081] In an exemplary embodiment, when driving a stylized neutral target object based on a video and a stylized target expression base, referring to Fig.13 As shown, the following steps S1310 to S1330 may be included:

[0082] In step S1310, key point detection is performed on the current video frame in the video to determine the expression parameters and posture parameters of the target object in the current video frame.

[0083] In an exemplary embodiment, when driving based on a video, it is necessary to make the stylized neutral target object perform the same expression and posture as the target object in the video. Therefore, it is necessary to perform key point detection on the current video frame in the video to obtain the expression parameters and posture parameters of the target object in the current video frame, and then control the stylized neutral target object to perform the corresponding expression and posture based on the expression parameters and posture parameters.

[0084] Among them, the posture parameters may include parameter information such as rotation, translation and scale of the target object in the current video frame; the expression parameters may include the weighting coefficients corresponding to the stylized target expression base required to execute the expression of the target object in the current video frame.

[0085] For example, key point detection can be performed on the current video frame through deep learning and other algorithms to obtain parameters such as the position corresponding to the key points contained in the target object in the current video frame, and then the position and other parameters can be solved to obtain the label parameters and posture parameters of the target object in the current video frame.

[0086] In step S1320, the expression of the stylized neutral target object is transformed according to the expression parameters and the stylized target expression base to obtain an expression driving result corresponding to the current video frame.

[0087] In an exemplary embodiment, after obtaining the expression parameters, the stylized target expression bases and the stylized neutral target objects can be weightedly combined using the expression parameters (i.e., the weighting coefficients corresponding to each stylized target expression base for executing the expression of the target object in the current video frame) to obtain the expression driving result corresponding to the current video frame.

[0088] For example, in an exemplary embodiment, the stylized target expression base and the stylized neutral target object may be weighted combined based on the linear weighted superposition method shown in the following formula (1) to obtain the expression driving result corresponding to the current video frame:

[0089]

[0090] Among them, S represents the expression driving result corresponding to the current video frame, represents the stylized neutral target object reconstructed based on the video, α irepresents the expression parameter corresponding to the i-th stylized target expression base, Bi is the i-th stylized target expression base; N represents the total number of stylized target expression bases generated based on the neutral preset object, the characteristic deformation parameter and the second deformation parameter.

[0091] In step S1330, the expression driving result is transformed in posture based on the posture parameters to obtain the driving result corresponding to the current video frame.

[0092] In an exemplary embodiment, after obtaining the expression driving result, the expression driving result can be subjected to posture transformation based on the posture parameters to obtain the driving result corresponding to the current video frame. Specifically, in this step, the expression driving result can be subjected to posture driving based on the posture parameters to obtain the driving result in which both the expression and the posture correspond to the current video frame in the video. For example, the expression driving result can be subjected to posture transformation such as translation, rotation, and scale transformation based on the posture parameters to obtain the driving result combining expression driving and posture driving.

[0093] Furthermore, in an exemplary embodiment, the posture driving of the expression-driven result can also be indirectly achieved by transforming the posture of the current camera corresponding to the expression-driven result. Specifically, in this step, the posture parameters can be negated, and then the posture transformation of the current camera corresponding to the expression-driven result can be performed based on the negated posture parameters. For example, assuming that the posture represented by the posture parameter is a translation of the target object by 2 units along the positive direction of the x-axis, it is negated, that is, translated by 2 units in the negative direction of the x-axis. At this time, the current camera can be translated by 2 units in the negative direction of the x-axis.

[0094] Since the expression-driven result may usually contain multiple vertices, if the expression-driven result is directly transformed in posture, it is necessary to calculate the posture transformation for each vertex based on the posture parameters corresponding to each video frame, which is a large amount of calculation. Therefore, the reverse posture transformation can be directly performed on a point of the current camera to indirectly achieve the posture transformation of the expression-driven result, reducing the amount of calculation.

[0095] In addition, in order to avoid repeated calculations, after the steps of reconstructing a stylized neutral target object and generating a stylized target expression base are performed for a video containing a target object, the video can be marked, and the reconstructed stylized neutral target object and the generated stylized target expression base can be stored locally. The next time the video needs to be driven by a stylized object, it can be determined based on the mark that the video has already undergone reconstruction of the stylized neutral target object and generation of the stylized target expression base once, so the stored stylized neutral target object and stylized target expression base can be directly obtained for direct calculation.

[0096] It should be noted that, in an exemplary embodiment, in order to further avoid repeated calculations, the reconstructed stylized neutral target object and the generated stylized target expression base can also be bound and stored, and when it is sent to other terminal devices, the stylized object can be driven directly.

[0097] In the above embodiment, on the one hand, since the stylized neutral target object and the stylized target expression base are generated based on the characteristic deformation parameters that characterize the characteristic attributes of the target object, the stylized neutral target object and the stylized target expression base that are relative to the target object in the video can be obtained, laying the foundation for the "thousands of faces" driving result; on the other hand, since each video frame in the video contains the characteristic attributes that characterize the expression and posture of the target object in each video frame, the driving result can be made to follow the expression and posture of the target object in each video frame in the video to perform the relative expression and posture, achieving the purpose of real-time driving; on the other hand, the stylized neutral target object and the stylized target expression base can be automatically calculated or stored in the local first deformation parameter and the second deformation parameter based on the input video and the neutral preset object designed in advance by the designer. Therefore, when driving, the "thousands of faces" expression driving can be automatically performed according to the input video, and the portability is high.

[0098] At the same time, since designers only need to design neutral preset objects, stylized neutral preset objects and stylized preset expression bases in advance, or design neutral preset objects in advance and set the first deformation parameters and the second deformation parameters, the "thousands of faces" expression driving corresponding to all videos can be completed, reducing the production cost of achieving the "thousands of faces" effect; in addition, when more expressions need to be realized, it is only necessary to design more stylized preset expression bases or set more groups of second deformation parameters on the original basis to achieve it, which has strong scalability.

[0099] The following takes the target object as the target face, refer to Figures 14 to 17 The technical solution of the embodiment of the present disclosure is described in detail:

[0100] Reference Fig.14 As shown, the driving method of the stylized object may include the following steps S1401 to S1411:

[0101] Step S1401, obtaining a video containing a target face;

[0102] Specifically, the video can be obtained directly from the camera module or from storage.

[0103] Step S1403, reading the video flag, and determining whether the video has been reconstructed with a stylized neutral target object and generated with a stylized target expression base according to the video flag;

[0104] Specifically, a flag can be designed to assist in determining whether a video has undergone reconstruction of a stylized neutral target object and generation of a stylized target expression base. For example, the flag of a video is set to 0 or true by default; after the video undergoes the first reconstruction of a stylized neutral target object and generation of a stylized target expression base, the flag of the video is set to 1 or false. When the flag is 1 or false, it can be determined that the video has undergone reconstruction of a stylized neutral target object and generation of a stylized target expression base.

[0105] Step S1405, when the stylized neutral target object has not been reconstructed and the stylized target expression base has not been generated in the video, reconstructing the stylized neutral target face based on the video;

[0106] Specifically, refer to Fig.15 As shown, it can be implemented based on the following steps S1501 to S1511:

[0107] Step S1501, determining a neutral video frame in the video;

[0108] Specifically, the expression attributes of the target face can be determined according to the facial key point detection algorithm to assist in determining the neutral video frame;

[0109] Step S1503, capturing a neutral target face in a neutral video frame;

[0110] Specifically, a neutral target face can be captured according to a face detection algorithm;

[0111] In addition, the face detection algorithm can also be used to determine whether the size and other characteristic attributes of the neutral target face in the neutral video frame meet the corresponding threshold, and when it meets the corresponding threshold, the neutral target face is intercepted;

[0112] When intercepting, the position of the face frame can be determined according to the coordinate information of the face frame obtained by face detection, and the face center of the target face is used as a reference for interception. At the same time, in order to make the neutral target face fit the feature parameter model, the interception frame can be set to a square, and the side length is the longer side of the face frame.

[0113] Step S1505, preprocessing the neutral target face image, including image reconstruction processing and normalization processing.

[0114] Specifically, in order to further adapt the neutral target face to the feature parameter model, the captured neutral target face image is reconstructed, and the pixel value of each pixel on the neutral target face image is normalized to [0, 1] or [-1, 1].

[0115] Step S1507, calculating deformation parameters, namely, characteristic deformation parameters, for converting the neutral preset face into the neutral target face based on the characteristic parameter model.

[0116] Step S1509, adding the characteristic deformation parameter and the first deformation parameter to obtain a first stylized parameter;

[0117] Step S1511, deforming the neutral preset face using a first stylized parameter to obtain a stylized neutral target face;

[0118] Step S1407, when the stylized neutral target object has not been reconstructed and the stylized target expression base has not been generated in the video, a stylized target expression base is generated based on the video;

[0119] Specifically, refer to Fig.16 As shown, it can be implemented based on the following steps S1601 to S1605:

[0120] Step S1601, obtaining the characteristic deformation parameters calculated in step S1507;

[0121] Step S1603, adding the characteristic deformation parameter and the N groups of second deformation parameters to obtain N groups of second stylized parameters;

[0122] Step S1605, performing deformation transformation on the neutral preset face using each set of second stylized parameters to obtain N stylized target expression bases;

[0123] Step S1409, driving the reconstructed stylized neutral target face according to the video and the generated stylized target expression base;

[0124] Step S1411, when the stylized neutral target object has been reconstructed and the stylized target expression base has been generated in the video, the stylized target expression base and the stylized neutral target face corresponding to the video stored locally are directly read;

[0125] Step S1413: driving the read stylized neutral target face based on the video and the read stylized target expression base.

[0126] Specifically, in step S1409, the reconstructed stylized neutral target face is driven according to the video and the generated stylized target expression base, and in step S14,3, the read stylized neutral target face is driven based on the video and the read stylized target expression base, referring to Fig.17 As shown, it can be implemented based on the following steps S1701 to S1707:

[0127] Step S1701, performing key point detection on the target face in the current video frame by using a deep learning algorithm;

[0128] Step S1703, calculating the expression parameters and posture parameters of the target face in the current video frame according to the key point detection results of the target face.

[0129] Among them, the pose parameters include parameters corresponding to the rotation, translation and scale of the current face. The expression parameters include the weighting coefficients of each stylized target expression basis to achieve the expression of the target face in the current video frame;

[0130] Step S1705, using the expression parameter as a weighting coefficient, linearly weighted superposition is performed on the stylized target expression base and the stylized neutral target object according to the above formula (1) to obtain the expression driving result corresponding to the current video frame;

[0131] Step S1707, performing translation, rotation, and scale transformation on the expression driving result based on the posture parameters to obtain the driving result corresponding to the current video frame.

[0132] In summary, in this exemplary embodiment, on the one hand, based on the generated stylized target expression base, stylized face driving of "thousands of faces for thousands of people" is realized, avoiding the problem of poor driving effect caused by the lack of corresponding expression bases for conventional stylized faces; on the other hand, the production cost caused by the need to re-produce multiple sets of expression bases for multiple stylized faces is reduced; at the same time, for the same video, the process of reconstructing the stylized neutral target face and generating the stylized target expression base only needs to be executed once, and the amount of driving calculations performed per frame is low, with high real-time and portability; in addition, the scalability of stylized face driving can also be enhanced.

[0133] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0134] For further reference, Fig.18 As shown, in an exemplary embodiment of the present disclosure, a stylized object driving device 1800 is provided, comprising a parameter determination module 1810, a stylized reconstruction module 1820, an expression base generation module 1830 and an object driving module 1840. Among them:

[0135] The parameter determination module 1810 can be used to capture a neutral target object with a neutral expression in a video containing a target object, and calculate feature deformation parameters based on the neutral target object; wherein the feature deformation parameters include deformation parameters for converting a neutral preset object into a neutral target object.

[0136] The stylized reconstruction module 1820 may be used to reconstruct a stylized neutral target object based on a neutral preset object, characteristic deformation parameters and first deformation parameters; wherein the first deformation parameters include deformation parameters for converting the neutral preset object into a stylized neutral preset object.

[0137] The expression base generation module 1830 can be used to generate a stylized target expression base based on a neutral preset object, characteristic deformation parameters and a second deformation parameter; wherein the second deformation parameter includes a deformation parameter for converting the neutral preset object into a stylized preset expression base.

[0138] The object driving module 1840 may be used to drive the stylized neutral target object based on the video and the stylized target expression base.

[0139] In an exemplary embodiment, the object driving module 1840 can be used to perform key point detection on the current video frame in the video to determine the expression parameters and posture parameters of the target object in the current video frame; perform expression transformation on the stylized neutral target object according to the expression parameters and the stylized target expression base to obtain the expression driving result corresponding to the current video frame; perform posture transformation on the expression driving result based on the posture parameters to obtain the driving result corresponding to the current video frame.

[0140] In an exemplary embodiment, the object driving module 1840 may be used to perform weighted combination of the stylized target expression base and the stylized neutral target object using the expression parameters as weights to obtain an expression driving result corresponding to the current video frame.

[0141] In an exemplary embodiment, the stylized reconstruction module 1820 may be used to perform parameter combination on the characteristic deformation parameter and the first deformation parameter to obtain the first stylized parameter; and perform deformation transformation on the neutral preset object based on the first stylized parameter to obtain the stylized neutral target object.

[0142] In an exemplary embodiment, the expression base generation module 1830 can be used to combine the characteristic deformation parameter and the second deformation parameter to obtain the second stylized parameter; and perform deformation transformation on the neutral preset object based on the second stylized parameter to obtain the stylized target expression base.

[0143] In an exemplary embodiment, the parameter determination module 1810 may be used to input the neutral target object into the feature parameter model to obtain deformation parameters for converting the neutral preset object into the neutral target object.

[0144] In an exemplary embodiment, the parameter determination module 1810 can be used to perform target object detection on each video frame in the video to determine the characteristic attributes of the target object in each video frame; determine that the target object is a neutral video frame with a neutral expression based on the characteristic attributes, and capture the target object in the neutral video frame to obtain a neutral target object.

[0145] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0146] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0147] In an exemplary embodiment of the present disclosure, an electronic device for implementing a driving method for a stylized object is also provided, which may be Figure 1 The terminal device 101, 102, 103 or the server 105 in the embodiment of the present invention comprises at least a processor and a memory, wherein the memory is used to store executable instructions of the processor, and the processor is configured to execute the driving method of the stylized object by executing the executable instructions.

[0148] Below Fig.19 Taking the mobile terminal 1900 in the example, the structure of the electronic device in the embodiment of the present disclosure is exemplarily described. It should be understood by those skilled in the art that in addition to the components specifically used for mobile purposes, Fig.19 The structure in the figure can also be applied to fixed type devices. In other embodiments, the mobile terminal 1900 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure can be implemented in hardware, software, or a combination of software and hardware. The interface connection relationship between the components is only shown schematically and does not constitute a structural limitation of the mobile terminal 1900. In other embodiments, the mobile terminal 1900 may also adopt the same Fig.19 Different interface connection methods, or a combination of multiple interface connection methods.

[0149] like Fig.19As shown, the mobile terminal 1900 may specifically include: a processor 1910, an internal memory 1921, an external memory interface 1922, a Universal Serial Bus (USB) interface 1930, a charging management module 1940, a power management module 1941, a battery 1942, an antenna 1, an antenna 2, a mobile communication module 1950, a wireless communication module 1960, an audio module 1970, a speaker 1971, a receiver 1972, a microphone 1973, an earphone interface 1974, a sensor module 1980, a display screen 1990, a camera module 1991, an indicator 1992, a motor 1993, a button 1994, and a subscriber identification module (SIM) card interface 1995, etc. The sensor module 1980 may include a depth sensor 19801, a pressure sensor 19802, a gyroscope sensor 19803, etc.

[0150] The processor 1910 may include one or more processing units, for example, the processor 1910 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor and / or a neural network processor (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0151] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of mobile terminal 1900, such as image recognition, face recognition, voice recognition, text understanding, etc. In some embodiments, NPU can be used to train feature parameter models and obtain deformation parameters for converting neutral preset objects into neutral target objects based on the feature parameter models.

[0152] The processor 1910 is provided with a memory. The memory can store instructions for implementing six modular functions: detection instructions, connection instructions, information management instructions, analysis instructions, data transmission instructions, and notification instructions, and the execution is controlled by the processor 1910.

[0153] The mobile terminal 1900 implements the display function through a GPU, a display screen 1990, and an application processor. The GPU is a microprocessor for image processing, which is connected to the display screen 1990 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 1910 may include one or more GPUs that execute program instructions to generate or change display information. In some embodiments, the GPU, the display screen 1990, and the application processor can be used to implement the reminder that when recording a video, the user needs to collect a video frame with a neutral expression of the target object to ensure that a neutral target object can be obtained.

[0154] The mobile terminal 1900 can realize the shooting function through the ISP, the camera module 1991, the video codec, the GPU, the display screen 1990 and the application processor. Among them, the ISP is used to process the data fed back by the camera module 1991; the camera module 1991 is used to capture static images or videos; the digital signal processor is used to process digital signals, and in addition to processing digital image signals, it can also process other digital signals; the video codec is used to compress or decompress digital videos, and the mobile terminal 1900 can also support one or more video codecs. In some embodiments, the function of recording videos can be realized based on the ISP, the camera module 1991, the video codec, the GPU, the display screen 1990 and the application processor, so as to drive stylized objects based on videos.

[0155] In addition, the exemplary embodiments of the present disclosure further provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification, for example, the program product may be executed. Figures 2 to 3 , Figure 7 , Fig.11 , Figures 13 to 17 Any one or more steps in .

[0156] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0157] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0158] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0159] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0160] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for driving a stylized object, characterized in that: include: In a video containing a target object, a neutral target object with a neutral expression is intercepted, and characteristic deformation parameters are calculated based on the neutral target object; wherein the characteristic deformation parameters include deformation parameters for converting a neutral preset object into a neutral target object; Reconstructing a stylized neutral target object based on the neutral preset object, the characteristic deformation parameter and a first deformation parameter; wherein the first deformation parameter includes a deformation parameter for converting the neutral preset object into a stylized neutral preset object; Generate a stylized target expression base based on the neutral preset object, the characteristic deformation parameter and the second deformation parameter; wherein the second deformation parameter includes a deformation parameter for converting the neutral preset object into a stylized preset expression base; driving the stylized neutral target object based on the video and the stylized target expression base; The driving of the stylized neutral target object based on the video and the stylized target expression base includes: Performing key point detection on a current video frame in the video to determine expression parameters and posture parameters of the target object in the current video frame; Performing expression transformation on the stylized neutral target object according to the expression parameter and the stylized target expression base to obtain an expression driving result corresponding to the current video frame; The expression driving result is subjected to posture transformation based on the posture parameter to obtain the driving result corresponding to the current video frame.

2. The method according to claim 1, characterized in that The step of performing expression transformation on the stylized neutral target object according to the expression parameter and the stylized target expression base to obtain an expression driving result corresponding to the current video frame includes: The expression parameters are used as weights to perform a weighted combination on the stylized target expression base and the stylized neutral target object to obtain an expression driving result corresponding to the current video frame.

3. The method according to claim 1, characterized in that The reconstructing a stylized neutral target object based on the neutral preset object, the characteristic deformation parameter and the first deformation parameter comprises: Combining the characteristic deformation parameter and the first deformation parameter to obtain a first stylized parameter; The neutral preset object is deformed based on the first stylized parameter to obtain a stylized neutral target object.

4. The method according to claim 1, characterized in that: The step of generating a stylized target expression base based on the neutral preset object, the characteristic deformation parameter and the second deformation parameter comprises: Combining the characteristic deformation parameter and the second deformation parameter to obtain a second stylized parameter; The neutral preset object is deformed based on the second stylized parameter to obtain a stylized target expression base.

5. The method according to claim 1, characterized in that The calculating characteristic deformation parameters based on the neutral target object includes: The neutral target object is input into a feature parameter model to obtain deformation parameters for converting the neutral preset object into the neutral target object.

6. The method according to claim 1, characterized in that The method of capturing the target object as a neutral target object with a neutral expression in a video containing the target object includes: Performing target object detection on each video frame in the video to determine characteristic attributes of the target object in each video frame; The target object is determined to be a neutral video frame with a neutral expression based on the characteristic attribute, and the target object in the neutral video frame is intercepted to obtain a neutral target object.

7. A driving device for a stylized object, characterized in that: include: A parameter determination module, used to intercept a neutral target object with a neutral expression in a video containing a target object, and calculate characteristic deformation parameters based on the neutral target object; wherein the characteristic deformation parameters include deformation parameters for converting a neutral preset object into a neutral target object; A stylized reconstruction module, configured to reconstruct a stylized neutral target object based on the neutral preset object, the characteristic deformation parameter and a first deformation parameter; wherein the first deformation parameter comprises a deformation parameter for converting the neutral preset object into a stylized neutral preset object; An expression base generation module, used to generate a stylized target expression base based on the neutral preset object, the characteristic deformation parameter and the second deformation parameter; wherein the second deformation parameter includes a deformation parameter for converting the neutral preset object into a stylized preset expression base; An object driving module, configured to drive the stylized neutral target object based on the video and the stylized target expression base; Wherein, the object driver module is configured to execute: Performing key point detection on a current video frame in the video to determine expression parameters and posture parameters of the target object in the current video frame; Performing expression transformation on the stylized neutral target object according to the expression parameter and the stylized target expression base to obtain an expression driving result corresponding to the current video frame; The expression driving result is subjected to posture transformation based on the posture parameter to obtain the driving result corresponding to the current video frame.

8. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 6 by executing the executable instructions.

Images