A method, device, storage medium and equipment for rendering body special effects
The method and device for rendering limb special effects address the challenges of accuracy and real-time rendering by using neural networks for key point detection and human body parametric modeling, achieving realistic and three-dimensional limb effects on human bodies.
Patent Information
- Application Number
- CN202210412285.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-19
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-04-19
AI Technical Summary
Existing technologies lack effective methods for rendering body special effects, particularly on human limbs, due to difficulties in obtaining accurate human body models, real-time detection, and displaying special effects naturally and three-dimensionally, with few products available in the market.
A method and device for rendering limb special effects involving the acquisition of an image and a reference limb model, generating a three-dimensional model of the target object, determining a rendering result based on a mapping relationship, and applying camera parameters to accurately fit limb special effects to the target object's posture, using neural networks for key point detection and human body parametric modeling to reduce computational load.
The method achieves real-time rendering of limb special effects that accurately fit the target object's posture, providing a better three-dimensional sense and overcoming challenges of computational complexity and real-time performance.
Smart Images

Figure CN114863005B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a method, device, storage medium and equipment for rendering body special effects. Background Art
[0002] At present, there are many mature video or picture special effects on application platforms such as short videos, live broadcasts, and beauty cameras, such as expression special effects, headgear, stickers, etc. However, most of the special effects on the market are head special effects based on human faces or human head postures, while there are relatively few studies on special effects based on human limbs after a structural understanding of the human body, and there are almost no products that have been put into practice. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the present application provides a rendering method, device, storage medium and equipment for body special effects to solve the defects in the related art.
[0004] According to a first aspect of the present application, a method for rendering limb special effects is provided, the method comprising:
[0005] Acquire an image and a reference limb model of a target object, wherein the image includes a limb of the target object, the reference limb model is used to present a rendering result of a specified limb special effect on a reference three-dimensional model, and the reference limb model includes a mapping relationship between the limb special effect and the reference three-dimensional model;
[0006] Based on the image, generate a three-dimensional model corresponding to the target object, and obtain camera parameters corresponding to the image;
[0007] Determine, according to a mapping relationship between the limb special effect and the reference three-dimensional model, a rendering result of the limb special effect on the three-dimensional model of the target object, and generate a limb special effect model of the target object corresponding to the limb special effect;
[0008] The limb special effects are rendered in the image according to the limb special effects model and the camera parameters.
[0009] According to a second aspect of the present application, a rendering device for body special effects is provided, the device comprising:
[0010] An input module is used to obtain an image and a reference limb model of a target object, wherein the image includes a limb of the target object, the reference limb model is used to present a rendering result of a specified limb special effect on a reference three-dimensional model, and the reference limb model includes a mapping relationship between the limb special effect and the reference three-dimensional model;
[0011] A human body modeling module, used to generate a three-dimensional model corresponding to the target object based on the image, and obtain camera parameters corresponding to the image;
[0012] A special effects modeling module, used to determine the rendering result of the limb special effects on the three-dimensional model of the target object according to the mapping relationship between the limb special effects and the reference three-dimensional model, and generate a limb special effects model of the target object corresponding to the limb special effects;
[0013] A rendering module is used to render the limb special effects in the image according to the limb special effects model and the camera parameters.
[0014] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method described in any of the above embodiments is implemented.
[0015] According to a fourth aspect of the present application, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the above embodiments when executing the program.
[0016] The three-dimensional model of the target object generated in the method described in the present application has the same posture as the target object in the input image, and the limb special effects model obtained after combining the limb special effects with the three-dimensional model is also a limb special effect corresponding to the posture. Therefore, the limb special effects rendered based on the limb special effects model can better fit the target object on the image and have a better three-dimensional sense.
[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a flowchart of a method for rendering body special effects according to an embodiment of the present application.
[0019] Figure 2 It is a network structure diagram of a model prediction and camera prediction network shown in the present application according to one embodiment.
[0020] Figure 3 It is a block diagram of a rendering device for body special effects according to an embodiment of the present application.
[0021] Figure 4 It is a schematic diagram of the structure of a computing device hardware according to an embodiment of the present application. DETAILED DESCRIPTION
[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0023] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0024] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0025] In this application, limb effects refer to special effects made around human limbs, such as vest effects, tattoo effects, etc., which can follow the movement of human limbs to perform corresponding actions, and can be applied in a variety of scenarios. For example, in the fields of live broadcasts and short videos, limb effects can be used to increase the fun, and in the clothing industry, the limb effects of clothes can also be used to simulate the effect of users wearing specified clothes, so that users can choose clothes that fit. Limb characteristics include texture-type limb effects and limb characteristics based on joint binding types. Among them, texture-type limb characteristics refer to special effects that are expressed in the form of texture on a specified area of the limbs, such as using patterns of multiple colors or shapes to stick on the user. Different from limb effects based on joint binding types, texture-type limb effects require more accurate human body model prediction because they need to be attached to a specified position on a person's limbs, rather than a specified position on the image. However, it is still relatively difficult to obtain a relatively accurate human body model based on the structural posture prediction of the human body, and it is not easy to achieve real-time detection. In addition, it is difficult to display special effect objects naturally and beautifully for the predicted human body model. Moreover, for special effects such as stickers, it is also necessary to solve the problem of how to place stickers to ensure that they are different from simple two-dimensional stickers and have a certain three-dimensional sense. Therefore, there are almost no body special effects that can be well applied in various application software on the market.
[0026] In this regard, the present application provides a method for rendering body special effects to solve the above problems.
[0027] like Figure 1 As shown, Figure 1 This is a flowchart of a method for rendering limb special effects according to an embodiment of the present application, comprising the following steps:
[0028] Step S101, obtaining an image of a target object and a reference limb model of limb special effects;
[0029] Step S102, generating a three-dimensional model corresponding to the target object based on the image of the target object, and obtaining camera parameters corresponding to the image;
[0030] Step S103, determining the rendering result of the limb special effects on the three-dimensional model of the target object according to the mapping relationship between the limb special effects and the reference three-dimensional model, and generating a limb special effects model of the target object corresponding to the limb special effects;
[0031] Step S104, rendering the limb special effects in the image according to the limb special effects model and camera parameters.
[0032] In step S101, the acquired image, i.e., the input image, contains the limbs of the target object, so that the specified limb special effects can be rendered according to the current position and action of the limbs of the target object in the input image. The reference limb model refers to a three-dimensional model used to present the rendering result of the specified limb special effects on the reference three-dimensional model. Among them, the specified limb special effects refer to the limb special effects related to the specified limbs of the person designed by the special effects designer, which can be a texture-type limb special effect or a limb special effect based on a joint binding type, which is not limited in this application. In some embodiments, the specified limb special effects can be static limb special effects, such as a necklace special effect bound to the neck, a tattoo special effect drawn on the arm, etc., or a dynamic limb special effect, such as a flame special effect bound to the arm, etc., which is not limited in this application. The reference three-dimensional model refers to the standard three-dimensional human body model that the designer refers to when designing special effects. It is a standard human body model designed according to the human limb structure, rather than a human body model generated according to any target object, and does not have any user's limb characteristics. In some embodiments, in order to facilitate designers to intuitively observe the effects of the special effects they designed, the reference three-dimensional model can be a human body model displayed in a specified posture, for example, a T-shaped human body posture, that is, a posture with both feet combined and both arms spread horizontally. In some embodiments, the human body shape of the reference three-dimensional model can also be designed according to specific application scenarios. For example, when designing clothing special effects for people with larger bodies, the reference three-dimensional model can be a human body model of larger body size. The reference limb model contains a mapping relationship between limb special effects and the reference three-dimensional model, and the position where the limb special effects are displayed is bound to the specified area on the human body model, so that even if the current posture of the target object is different from the posture of the reference three-dimensional model, the corresponding position of the limb special effect in the current posture of the target object can be confirmed according to the mapping relationship between the limb special effect and the reference three-dimensional model. For example, on the reference three-dimensional model, when the limb special effect is displayed on the left arm, in the currently acquired input image, the rendered limb special effect should also be on the left arm of the target object.
[0033] In some embodiments, the input image obtained may be a single image or a video frame, i.e., a frame of an image in a video stream. For example, the above-mentioned body special effects rendering method may be applied in application scenarios such as live broadcast and short videos. In a live broadcast scenario, when the anchor activates a body special effect, each frame of the video in the live broadcast process will be rendered with the body special effect before being output to the client.
[0034] In step S102, a three-dimensional model corresponding to the target object can be generated according to the acquired input image, and the camera parameters corresponding to the input image can be acquired. Among them, the three-dimensional model has the same posture as the current posture of the target object in the input image, which can be realized by a three-dimensional reconstruction algorithm, and the present application does not impose any restrictions on this. The camera parameters refer to the position parameters of the camera corresponding to the input image in the coordinate system where the three-dimensional model is located. The three-dimensional model is observed by the camera parameters, and the position and posture of the target object in the generated image can be consistent with the target object in the input image, so that the final rendering result of the limb special effects can be accurately fitted to the pixels corresponding to the human body.
[0035] In some embodiments, a three-dimensional model of the target object can also be generated by inputting the limb key points of the target object in the image, that is, first obtaining the limb key points of the target object, then generating a three-dimensional model corresponding to the target object based on the limb key points, and obtaining the camera parameters corresponding to the input image. In some embodiments, the limb key points can be the joints of the human body as key points. For example, the limb key points of the target object can include 17 key points, which are: nose, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left palm, right palm, left hip, right hip, left knee, right knee, left ankle, right ankle, left toe, right toe, and these 17 key points can together form the current posture of the target object. If the human body model is predicted based on the image, it often requires a large amount of calculation and has high requirements for the computing environment. However, many users, such as anchors on live broadcast platforms, use mostly consumer-grade computer devices, so it is difficult to process visual algorithms with large amounts of calculation. Moreover, the large amount of calculation also means that the processing time of the entire special effects rendering will be relatively long, that is, it is difficult to achieve real-time rendering, and the effect is also relatively poor in scenarios with high real-time requirements such as live video broadcasting. However, if the human body model is predicted based on key points, since only a small number of key points need to be processed, the amount of calculation of the algorithm for generating the three-dimensional model can be significantly reduced, so that most consumer-grade devices can afford the rendering of body special effects, and the processing speed is faster, which can achieve real-time special effects rendering.
[0036] In some embodiments, the method for obtaining the limb key points of the target object can be to obtain the limb key points of the target object by inputting the input image into a specified neural network for training. The specified neural network refers to a neural network model for predicting key points, which can be an existing open source key point detection model, such as mobilenet v3, or other key point detection models, and this application does not limit this.
[0037] In some embodiments, a method for generating a three-dimensional model corresponding to a target object based on limb key points and obtaining camera parameters corresponding to an input image may be to first obtain the posture features of the target object based on the limb key points; then generate posture parameters, shape parameters and camera parameters corresponding to the input image of the target object based on the posture features; and finally generate a three-dimensional model corresponding to the target object through human body parametric modeling based on the posture parameters and shape parameters. Wherein, obtaining the posture features of the target object based on the limb key points and generating the posture parameters, shape parameters and camera parameters corresponding to the input image of the target object based on the posture features can be implemented by a neural network or one or more neural network layers in a neural network model. Wherein, the posture feature refers to a special effect vector used to describe the posture of the target object in the input image, which can be implemented by one or more neural network layers, for example, through a fully connected layer, the coordinates of each limb key point are input, and then a feature vector for representing the current limb posture can be output. In some embodiments, the number of limb key points is 17, and the input of the above-mentioned connection layer is a 34-dimensional data including the horizontal and vertical coordinates of the 17 limb key points, and the output of the above-mentioned connection layer can be a 256-dimensional posture feature. Specifically, in some embodiments, the posture feature can also be other dimensions, such as 128 dimensions or 512 dimensions. When the dimension of the posture feature is higher, generally speaking, the more refined the posture it can express, and the higher the accuracy of the human body model finally generated, but relatively, the amount of calculation in the process of generating the three-dimensional human body model is also higher, which affects the real-time performance of the entire rendering. Therefore, the dimension of the posture feature can be set according to actual needs, and this application does not impose any restrictions on this. The posture parameters and shape parameters refer to a set of parameters that can represent the current posture of the target object, such as the rotation angle of each joint, and a set of parameters that can represent the state of the target object, such as height, weight, head-to-body ratio and other features. Based on the above posture features, through three different sub-neural networks, the posture parameters, shape parameters, and camera parameters corresponding to the input image of the target object can be predicted respectively. In some embodiments, the three sub-neural networks that predict the posture parameters, shape parameters, and camera parameters corresponding to the input image of the target object can all be fully connected layers.
[0038] Human body parametric modeling refers to the process of generating a human body model object based on the input parameters such as posture parameters and shape parameters after analysis and processing by the program's internal logic. In human body parametric modeling, the human body can be understood as the sum of a basic model and the deformation based on the model. The basic model can be called a human body parametric model, and the shape parameters can control the deformation of the basic model, thereby controlling the shape of the human body model generated in the end. The posture of the human body model is controlled by the posture parameters. Among them, the posture of the human body can be represented by a motion tree composed of various joints of the human body. Among them, according to the connection relationship between the joints, each joint can be divided into a parent node and a child node. When the parent node moves, it will drive the child node to move together. For example, when the elbow joint moves, the wrist joint on the same arm will move with the movement of the elbow joint, and the elbow joint is the parent node of the wrist joint. In the motion tree, the rotation relationship between each joint and the parent joint can be represented by a rotation vector. The local rotation vector of each joint is combined together to represent the current posture of the human body, that is, the local rotation vector of each joint can constitute the posture parameter of the human body model. By changing the rotation vector of any joint in the posture parameter, the posture of the human body model can be changed. In some embodiments, the human body parameterized model used in the present application can be an SMPL parameterized model, and correspondingly, the deformation of the human body model is controlled by ten increments, that is, the morphological parameter is a 10-dimensional parameter, and the motion tree is composed of 24 joints. If the rotation vector of each joint point is represented by a 6D angle drive signal, the posture parameter is a 144-dimensional parameter.
[0039] Although different joints in the human body model are connected together, their movements will produce some linkage effects, especially the effects between joints that are closer. For example, the movement of the shoulder joint will affect the elbow joint and wrist joint, and the movement of the elbow joint will also affect the wrist joint, while there is almost no effect between joints that are farther away. For example, it is difficult for the elbow joint to affect the knee joint. Through the correlation between the joints, that is, the degree of influence of one joint on another joint, the posture of the human body can be more accurately described. Therefore, in some embodiments, after the posture features of the target object are acquired based on the limb key points, the local features of the target object can also be extracted based on the posture features of the target object, and the posture features can be optimized according to the correlation between the local features of the target object, so as to generate the posture parameters, shape parameters and camera parameters corresponding to the input image of the target object based on the optimized posture features, so that the three-dimensional human body model generated based on the posture parameters and shape parameters has higher accuracy.
[0040] In some embodiments, the above-mentioned steps of extracting local features of the target object based on the posture features of the target object, and optimizing the posture features according to the correlation between the local features of the target object can be implemented through an attention mechanism. The specific process can be that after the posture features of the target object are obtained based on the limb key points, the query features (Q), key features (K) and value features (V) of the posture features are first extracted through three sub-network feature extractors, and then the K features are transposed, and the Q features and the transposed K features are matrix multiplied to obtain the attention feature matrix, and then the softmax operator is used to normalize the attention feature matrix, and the normalized attention feature matrix is matrix multiplied with the V features, and finally the feature vector F with attention is obtained, that is, the optimized posture features. Specifically, the above process can be represented by the following equation:
[0041] F = softmax(QK T )V
[0042] Among them, compared with the posture features before optimization, the optimized posture features F fully utilize the local features of the target object through the attention mechanism. Therefore, the three-dimensional human body model generated by the posture parameters and shape parameters obtained based on the optimized posture features has higher accuracy. In the above process, the three sub-network feature extractors used to extract Q features, K features and V features can be respectively recorded as Q layers, K layers and V layers. These three layers can all be fully connected layers. When the input posture features are 256-dimensional, the Q features, K features and V features output by these three layers are also 256-dimensional. Moreover, in some embodiments, these three feature extractors can be explicitly trained through end-to-end training.
[0043] Since the level of detail of the human body three-dimensional model is not high when rendering the limb special effects model, in some embodiments, after the three-dimensional model corresponding to the target object is generated, the three-dimensional model can be downsampled at least once. Here, downsampling refers to uniformly selecting about half or a specified number of three-dimensional points in the three-dimensional model for retention, while discarding other three-dimensional points to reduce the number of three-dimensional points in the three-dimensional model, thereby reducing the amount of subsequent calculations. For example, if the three-dimensional model obtained by SMPL parameterized model reasoning has a total of 6890 three-dimensional points, two downsamplings can be performed at the end so that the final three-dimensional model only retains 1200 three-dimensional points.
[0044] In some embodiments, step S102 can be implemented by an overall neural network, namely, a model prediction and camera prediction network. In some embodiments, the network structure diagram of the model prediction and camera prediction network can be as follows: Figure 2As shown in the figure, the first layer of the network is a fully connected layer (FC), which inputs the horizontal and vertical coordinates of 17 limb key points, with a total of 34 dimensions, and outputs a posture feature of 256 dimensions. For the posture feature, three sub-network feature extractors, Q layer, K layer and V layer, are used to extract Q features, K features and V features respectively, and these three layers are all fully connected layers. The input posture feature is 256 dimensions, and the output Q features, K features and V features are also 256 dimensions. Then, for Q features and K features, first transpose the K features, and then perform matrix multiplication on the Q features (256*1) and the transposed K features (1*256) to obtain the (256*256) attention feature matrix. Then, the softmax operator is used to normalize the attention feature matrix, and the normalized attention feature matrix is matrix multiplied with the V features (256*1), and finally the posture feature with attention (256*1) is obtained. The above-mentioned attention process can be regarded as the second layer of the entire model prediction and camera prediction network, which is used to optimize the posture features. Subsequently, the third layer of the neural network consists of three sub-networks, which are also fully connected layers, respectively used to predict the posture parameters, shape parameters and camera parameters corresponding to the input image of the target object. Among them, the input of the sub-network FC1 used to predict the posture parameters is 256-dimensional, and the output is 144-dimensional, including the 6D rotation angle drive signal of 24 joints; the input of the sub-network FC2 used to predict the morphological parameters is 256-dimensional, and the output is 10-dimensional, including 10 incremental parameters for controlling the deformation degree of the basic model; and the input of the sub-network FC3 used to predict the camera parameters is 256-dimensional, and the output is 3-dimensional. Finally, the posture parameters and morphological parameters are inferred through the SMPL parameterized model to obtain the 3D model of the target object, including 6890 3D points. It is worth noting that in some embodiments, 17 limb key points are selected from various joints of the human body, so the 24 joint points in the posture parameters corresponding to the SMPL model also include 17 limb key points. In addition, in some embodiments, in order to reduce the size of the neural network, thereby reducing the amount of calculation and rendering time, Figure 2 Each layer in may include only one neural network layer, and in other embodiments, in order to obtain higher accuracy, Figure 2 Each layer in may also include multiple neural network layers, and this application does not limit this. Figure 2 It can be seen that the model prediction and camera prediction networks used in this application contain fewer layers, and the parameter dimension of each network layer is also relatively small. Therefore, the calculation scale of the entire neural network is relatively small and the processing speed is relatively fast. Therefore, the model prediction and camera prediction networks used in this application can achieve real-time prediction of three-dimensional human body models.
[0045] Since the three-dimensional model obtained in step S102 is obtained through neural network prediction, it is inevitable that there are some errors. In some embodiments, the three-dimensional model can be further optimized based on the limb key points of the target object to obtain a more accurate three-dimensional network result. The specific process can be, first, the two-dimensional coordinates of each limb key point of the target object obtained through the input image are used as the actual value (P pred ); then extract the three-dimensional coordinates of the three-dimensional points corresponding to each limb key point from the generated three-dimensional model, and use a projection algorithm, such as a weak perspective projection algorithm, to obtain the two-dimensional coordinates of the three-dimensional points corresponding to each limb key point taken out from the three-dimensional model after projection based on the camera parameters obtained in the above process, as the predicted value of the limb key point (P proj ), that is, the predicted positions of the key points of each limb when the three-dimensional model generated by step S102 is projected onto the original input image; and an energy equation is defined:
[0046] E(pose)=||P pred -P proj ||
[0047] Among them, pose is the predicted posture of the three-dimensional model, such as the posture parameters obtained in the above model prediction and camera prediction network, and the above energy equation is used to represent the key points obtained based on the input image, that is, the error between the limb key points predicted by the prediction model and the key points obtained by the three-dimensional model projection. Then, through optimization algorithms, such as the steepest descent optimization algorithm, this error value is continuously optimized and reduced to obtain a more accurate three-dimensional model.
[0048] In step S103, the reference three-dimensional model is generated in a similar manner to the three-dimensional model of the generated target object, so the two have the same structure. Therefore, according to the mapping relationship between the limb special effect and the reference three-dimensional model, the rendering result of the limb special effect on the three-dimensional model of the target object can be determined, and the limb special effect model corresponding to the target object of the limb special effect is generated, that is, the limb special effect model can follow the posture of the target object to make corresponding changes, so that the limb special effect looks more three-dimensional and has a better performance effect. For example, a limb special effect with a vest effect can be designed on a reference three-dimensional model (a standard model, such as a three-dimensional model of a T posture), and the vest special effect is bound to a three-dimensional point mapping relationship on the reference three-dimensional model, and the mapping relationship can be recorded as an f function, with the input being a three-dimensional model, and the output being the corresponding position of each three-dimensional point on the vest special effect and the input three-dimensional model. Therefore, through the f function, the parameters of all three-dimensional point coordinates in the three-dimensional model predicted in step S102 can be used as input, and the output of the f function is the corresponding position of each three-dimensional point on the vest special effect. Therefore, through step S103, the three-dimensional coordinate information of the limb special effect can be obtained for use in the final rendering step.
[0049] In step S104, after obtaining the three-dimensional coordinate information of the limb special effect, the limb special effect can be rendered at the corresponding position in the input image according to the camera parameters obtained in step S102, and finally an output image containing the limb special effect corresponding to the target object is obtained.
[0050] If rendering is performed only based on the limb special effects model, the entire projection of the limb special effects will be completely rendered in the output image during rendering. However, in some images, the limbs of the target object may block the special effects. If the blocked part of the special effects is still rendered, it may not be realistic enough. Therefore, in some embodiments, the three-dimensional model of the target object can be rendered using a transparent color, and the limb special effects model can be rendered using a preset color. According to the spatial relationship between the three-dimensional model of the target object and the limb special effects model, it is determined whether the three-dimensional model of the target object blocks the specified local area of the limb special effects model, and then the unblocked area of the limb special effects model is rendered, and the blocked area is not rendered. Among them, the purpose of using a transparent color to render the three-dimensional model of the target object is that the target that needs to be rendered should only be the limb special effects, that is, the three-dimensional model of the target object is not explicitly rendered, and the transparent three-dimensional model will still block the non-transparent limb special effects model, so that the rendered limb special effects can retain the three-dimensional effect of the blockage.
[0051] Corresponding to the embodiment of the method for rendering limb special effects described above, the present application also provides a device for rendering limb special effects.
[0052] like Figure 3 As shown, Figure 3This is a block diagram of a rendering device for body special effects according to an embodiment of the present application, including the following modules:
[0053] Input module 310: used to obtain an image of a target object and a reference limb model of limb effects;
[0054] Human body modeling module 320: used to generate a three-dimensional model corresponding to the target object based on the image of the target object, and obtain camera parameters corresponding to the image;
[0055] Special effects modeling module 330: used to determine the rendering result of the limb special effects on the three-dimensional model of the target object according to the mapping relationship between the limb special effects and the reference three-dimensional model, and generate a limb special effects model of the target object corresponding to the limb special effects;
[0056] Rendering model 340: used to render the limb effects in the image according to the limb effects model and camera parameters.
[0057] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0058] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art may understand and implement it without creative work.
[0059] The present application also provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in any of the aforementioned embodiments when executing the program.
[0060] Figure 4 A more specific hardware structure diagram of a computing device provided in the present application is shown, and the device may include: a processor 401, a memory 402, an input / output interface 403, a communication interface 404, and a bus 405. The processor 401, the memory 402, the input / output interface 403, and the communication interface 404 are connected to each other in communication within the device through the bus 405.
[0061] The processor 401 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., for executing relevant programs to implement the technical solution provided in this application. The processor 401 can also include a graphics card, which can be a Nvidia titan X graphics card or a 1080Ti graphics card, etc.
[0062] The memory 402 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 402 may store an operating system and other application programs. When the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401.
[0063] The input / output interface 403 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.
[0064] The communication interface 404 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).
[0065] The bus 405 comprises a pathway for transmitting information between the various components of the device (eg, the processor 401 , the memory 402 , the input / output interface 403 , and the communication interface 404 ).
[0066] It should be noted that, although the above device only shows the processor 401, the memory 402, the input / output interface 403, the communication interface 404 and the bus 405, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.
[0067] The present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in any of the aforementioned embodiments when the program is executed by a processor.
[0068] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.
[0069] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.
[0070] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.
[0071] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the scheme of this application. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without paying creative labor.
[0072] The above is only a specific implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for rendering body special effects, characterized in that: The method comprises: Acquire an image and a reference limb model of a target object, wherein the image includes a limb of the target object, the reference limb model is used to present a rendering result of a specified limb special effect on a reference three-dimensional model, and the reference limb model includes a mapping relationship between the limb special effect and the reference three-dimensional model; Based on the image, generate a three-dimensional model corresponding to the target object, and obtain camera parameters corresponding to the image; Determine, according to a mapping relationship between the limb special effect and the reference three-dimensional model, a rendering result of the limb special effect on the three-dimensional model of the target object, and generate a limb special effect model of the target object corresponding to the limb special effect; The limb special effects are rendered in the image according to the limb special effects model and the camera parameters.
2. The method according to claim 1, characterized in that: The step of generating a three-dimensional model corresponding to the target object based on the image and acquiring camera parameters corresponding to the image comprises: Acquire limb key points of the target object, generate a three-dimensional model corresponding to the target object based on the limb key points, and acquire camera parameters corresponding to the image.
3. The method according to claim 2, characterized in that The limb key points are obtained by inputting the image into a neural network for training.
4. The method according to claim 2, characterized in that: The steps of generating a three-dimensional model corresponding to the target object based on the limb key points and acquiring camera parameters corresponding to the image include: Acquire the posture features of the target object based on the limb key points; Based on the posture features, respectively generate posture parameters and shape parameters corresponding to the target object and camera parameters corresponding to the image; According to the posture parameters and shape parameters, a three-dimensional model corresponding to the target object is generated through human body parametric modeling.
5. The method according to claim 4, characterized in that After the step of acquiring the posture features of the target object based on the limb key points, the method further includes: Based on the posture features of the target object, local features of the target object are extracted, and according to the correlation between the local features of the target object, the posture features are optimized to generate posture parameters and shape parameters corresponding to the target object and camera parameters corresponding to the image based on the optimized posture features.
6. The method according to claim 5, characterized in that The step of extracting local features of the target object based on the posture features of the target object and optimizing the posture features according to the correlation between the local features of the target object is achieved through an attention mechanism.
7. The method according to claim 2, characterized in that: After the steps of generating a three-dimensional model corresponding to the target object and obtaining camera parameters corresponding to the image, the method further includes: Taking the two-dimensional coordinates of the limb key points as the actual values of the limb key points; Obtaining the three-dimensional coordinates of the three-dimensional points corresponding to the limb key points in the three-dimensional model; According to the three-dimensional coordinates and the camera parameters, a two-dimensional coordinate of the three-dimensional point after projection based on the camera parameters is obtained through a projection algorithm as a predicted value of the limb key point; By optimizing the algorithm, the error between the actual value and the predicted value of the limb key point is reduced to optimize the three-dimensional model.
8. The method according to claim 1, characterized in that After the steps of generating a three-dimensional model corresponding to the target object based on the image and obtaining camera parameters corresponding to the image, the method further includes: The three-dimensional model is downsampled at least once.
9. The method according to claim 1, characterized in that: The step of rendering the limb special effects in the image according to the limb special effects model and the camera parameters includes: In the image, the three-dimensional model is rendered using a transparent color, and the limb special effects model is rendered using a preset color, and based on the spatial relationship between the three-dimensional model and the limb special effects model, it is determined whether the three-dimensional model occludes a specified local area of the limb special effects model.
10. The method according to claim 1, characterized in that The images are video frames.
11. A rendering device for body special effects, characterized in that: The device comprises: An input module is used to obtain an image and a reference limb model of a target object, wherein the image includes a limb of the target object, the reference limb model is used to present a rendering result of a specified limb special effect on a reference three-dimensional model, and the reference limb model includes a mapping relationship between the limb special effect and the reference three-dimensional model; A human body modeling module, used to generate a three-dimensional model corresponding to the target object based on the image, and obtain camera parameters corresponding to the image; A special effects modeling module, used to determine the rendering result of the limb special effects on the three-dimensional model of the target object according to the mapping relationship between the limb special effects and the reference three-dimensional model, and generate a limb special effects model of the target object corresponding to the limb special effects; A rendering module is used to render the limb special effects in the image according to the limb special effects model and the camera parameters.
12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.
13. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 10 is implemented.
