A method, system, electronic device and storage medium for animating hand-painted avatars

Abstract

The embodiments of the present invention provide a method, system, electronic device and storage medium for animating a hand-drawn avatar. The method obtains in real time the newly added hand-drawn elements of each step when the user draws the avatar by hand, and generates in real time the corresponding animation results based on the newly added hand-drawn elements. The recognition results and the animation image can be updated in real time according to the overall content of the user's hand-drawing, and the user's hand-drawn elements can be animated in real time. The animation is done from scratch, and the user animates while hand-drawing the avatar. During the process of hand-drawing the avatar, the user can obtain in real time the animation results of the currently hand-drawn facial features and other elements, so that the animation style finally formed can be adjusted during the process of hand-drawing the avatar.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a method, system, electronic device and storage medium for animating a hand-drawn avatar. Background Art

[0002] Animation production is an emerging industry in our country, among which the design and application of avatar cartoons are very popular. At present, avatar animation design usually extracts facial features, retains organ features, obtains the avatar sketch portrait sample, and then processes the sample to obtain the avatar cartoon.

[0003] At present, the commonly used animation method is that the user uploads a headshot photo, the system recognizes the user's headshot and facial features, and animates it at one time. This is an animation based on an existing portrait image. That is, it is the result of overall animation based on an existing template, and its animation style is fixed, which makes it impossible for users to make adaptive modifications to the animation details and adjust the animation style. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method, system, electronic device and storage medium for animating a hand-drawn avatar, which solves the problem in the prior art that overall animation is performed based on an existing template, the animation style is fixed, and the user is unable to adaptively modify the animation details and adjust the animation style.

[0005] To solve the above problems, in a first aspect, an embodiment of the present invention provides a method for animating a hand-drawn avatar, comprising:

[0006] Detect hand-drawn avatars in hand-drawn screen areas;

[0007] If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time;

[0008] Based on the hand-drawn elements of the current hand-drawn step, an animation result of the hand-drawn elements of the current hand-drawn step is generated in real time in the animation display area.

[0009] Compared with the prior art, the embodiment of the present invention acquires the hand-drawn elements of each step when the user hand-draws the avatar in real time, and generates the corresponding animation result based on the hand-drawn elements of the current step in real time. It can update the recognition result and the animation image in real time according to the overall content of the user's hand-drawing, and animate the user's hand-drawn elements in real time. The animation is from scratch, and the user animates while hand-drawing the avatar. During the process of hand-drawing the avatar, the user can know the animation result of the current hand-drawn facial features and other elements in real time, so that the final animation style can be adjusted during the process of hand-drawing the avatar.

[0010] In a second aspect, an embodiment of the present invention provides a hand-drawn avatar animation system, including:

[0011] A hand-drawn element acquisition module is used to detect hand-drawn avatars in the hand-drawn screen area;

[0012] If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time;

[0013] The animation generation module is used to generate the animation result of the hand-drawn elements of the current step in real time in the animation display area based on the hand-drawn elements of the current step.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the hand-drawn avatar animation method as described in the embodiment of the first aspect of the present invention are implemented.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the hand-drawn avatar animation method as described in the embodiment of the first aspect of the present invention.

[0016] Optionally, after generating an animation result of the hand-drawn element of the current hand-drawn step in real time in the animation display area based on the hand-drawn element of the current hand-drawn step, the method further includes:

[0017] Based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, the animation result of the hand-drawn elements of the previous hand-drawn step is corrected.

[0018] Optionally, the step of generating an animation result of the hand-drawn elements of the current hand-drawn step in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step, and correcting the animation result of the hand-drawn elements of the previous hand-drawn step based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, specifically includes:

[0019] Input the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step into the trained animation generation model; if the hand-drawn element of the current hand-drawn step is the first hand-drawn element in the hand-drawn screen area, the hand-drawn element of the previous hand-drawn step is a blank drawing paper;

[0020] Based on the animation generation model, the animation result of the hand-drawn elements of the current hand-drawn step is generated in real time in the animation display area, and according to the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, the animation result of the hand-drawn elements of the previous hand-drawn step is corrected.

[0021] Optionally, before generating an animation result of the hand-drawn elements of the current hand-drawn step in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step, the method further includes:

[0022] Collect hand-drawn elements of different styles and the corresponding animation results;

[0023] A set of image data pairs with each hand-drawn element and the corresponding animation result;

[0024] The image data pairs of different hand-drawn elements are randomly paired in pairs to construct a training sample set. Two different hand-drawn elements are used as input, and the corresponding recognition results and animation results are used as output. Deep learning training is performed to obtain an animation generation model.

[0025] Optionally, it also includes:

[0026] All hand-drawn elements in the current hand-drawn avatar are sequentially input into the trained hand-drawn prediction model to predict the hand-drawn elements of the next hand-drawn step, and the recognition results and animation results of the predicted hand-drawn elements are recommended to the user.

[0027] Optionally, it also includes:

[0028] The order of hand-drawn elements in the drawing process of different hand-drawn portraits is collected, and the hand-drawn elements are divided into several groups of input-output pairs according to the order of the hand-drawn elements, and the types of hand-drawn elements in each group of input-output pairs are randomly matched to form training samples;

[0029] A neural network training is performed based on the training samples to obtain a hand-drawing prediction model for predicting the hand-drawing elements of the next hand-drawing step.

[0030] Optionally, after confirming that the user has finished hand-drawing the avatar, the color and size of each hand-drawn element corresponding to the animation result are adjusted to output the final animated avatar. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0032] Figure 1 A flow chart of a method for animating a hand-drawn avatar according to an embodiment of the present invention;

[0033] Figure 2 A schematic diagram of a process of animating hand-drawn elements according to a method for animating a hand-drawn avatar according to an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a recognition result and an animation result generated between hand-drawn elements in two adjacent hand-drawn steps according to an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of an animation generation model according to an embodiment of the present invention;

[0036] Figure 5 A schematic diagram of a user hand-drawn element feature extraction network according to an embodiment of the present invention;

[0037] Figure 6 A hand-drawn prediction model schematic diagram according to an embodiment of the present invention;

[0038] Figure 7 A structural block diagram of a hand-drawn avatar animation system according to another embodiment of the present invention;

[0039] Figure 8 A schematic diagram of a server according to yet another embodiment of the present invention. DETAILED DESCRIPTION

[0040] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0041] In the embodiments of the present application, the term "and / or" is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0042] The terms "first" and "second" in the embodiments of the present application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a system, product or device comprising a series of parts or units is not limited to the listed parts or units, but optionally also includes parts or units that are not listed, or optionally also includes other parts or units inherent to these products or devices. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0043] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0044] The commonly used animation method is that the user uploads a headshot photo, the system recognizes the user's headshot and facial features, and animates it all at once. This is an animation based on an existing portrait image. Therefore, the user needs to draw a whole headshot before knowing the animation effect. During the user's hand-drawn headshot process, it is impossible to predict the animation result of the current hand-drawn facial features and other elements in real time.

[0045] Therefore, the embodiments of the present invention provide a method, system, electronic device and storage medium for animating a hand-drawn avatar, which can obtain the hand-drawn elements of each hand-drawn step of the user's hand-drawn avatar in real time, and generate corresponding animation results in real time based on the hand-drawn elements. The recognition results and animation images can be updated in real time according to the overall content of the user's hand-drawn, and the user's hand-drawn elements can be animated in real time. The next hand-drawn element and its animation can be predicted according to the user's hand-drawn order and elements, and recommended to the user. The following will be explained and introduced through multiple embodiments.

[0046] Figure 1 A method for animating a hand-drawn avatar according to an embodiment of the present invention includes:

[0047] S1, detecting the hand-drawn avatar in the hand-drawn screen area;

[0048] If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time;

[0049] S2. Generate an animation result of the hand-drawn elements of the current hand-drawn step in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step.

[0050] Specifically, in this embodiment, the screen can be divided into two parts, namely the hand-drawn screen area and the animation display area. The hand-drawn screen area is used for the user to hand-draw the avatar, and the animation display area is used to display the animation result. Each newly added hand-drawn element is regarded as a hand-drawn step. The hand-drawn elements include eyes, nose, mouth, face, hair, ears, etc., and each hand-drawn step needs to be performed on the basis of all the previous hand-drawn steps. All the hand-drawn steps are displayed in separate lines, such as Figure 2As shown in , in actual application, for each hand-drawing step the user draws on the left side (i.e., the hand-drawn screen area), the corresponding position on the right side (i.e., the animation display area) will display the animation result. For example, if the user draws a head portrait on the left side, he or she can draw the eyes, nose, mouth, head and other hand-drawn elements in steps, and extract the shape, size and other features on the right side to recognize the user's hand-drawn content in real time, and update the recognition result and the animation image in real time according to the overall content of the user's hand-drawn content. For users who have no painting skills, the method of this embodiment can output high-quality animation while maintaining the style of the user's drawn head portrait.

[0051] When collecting hand-drawn elements, as an implementation method, the hand-drawn track of the user when drawing the avatar is obtained in real time in the hand-drawn screen area. If it is determined that the user does not continue to draw within a preset time after the end point of the hand-drawn track is drawn, or the user confirms the end of drawing, then it is confirmed that the hand-drawing of the current hand-drawn step is ended;

[0052] The hand-drawing process between the hand-drawing end time of the current hand-drawing step and the hand-drawing end time of the previous hand-drawing step is regarded as a hand-drawing step of the user's hand-drawn avatar;

[0053] All hand-drawing trajectories of each hand-drawing step are extracted as hand-drawing elements.

[0054] As another implementation, the start time and end time of each hand-drawing step may also be selected by the user.

[0055] Based on the above embodiment, as a preferred implementation manner, after the animation result of the hand-drawn element of the current hand-drawn step is generated in real time in the animation display area based on the hand-drawn element of the current hand-drawn step, it also includes:

[0056] Based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, the animation result of the hand-drawn elements of the previous hand-drawn step is corrected.

[0057] Specifically, in this embodiment, Figure 3 As shown in, according to the user's drawing habits, the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step are used as input for analysis, and the animation result of the current hand-drawn step and the animation result of the previous hand-drawn step can be obtained. For example, the first step is to draw a face, and the second step is to draw eyes. Then, in the animation process of the face, the blank drawing paper and the hand-drawn face are used as input for analysis, that is, the blank drawing paper is used as the hand-drawn element of the previous hand-drawn step, and the face is used as the hand-drawn element of the current hand-drawn step. The final analysis result is the recognition result and animation result of the face, corresponding to the animation result of the previous hand-drawn step and the animation result of the current hand-drawn step.

[0058] In this embodiment, during an animation process, the hand-drawn elements of the current hand-drawn step and the previous hand-drawn step need to be processed simultaneously. In this way, the recognition result and animation result of the hand-drawn elements of the previous hand-drawn step can be corrected according to the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step. At the same time, each hand-drawn element needs to be recognized and animated twice in succession, which increases the reliability of recognition and animation. The relationship between different hand-drawn elements includes relative position relationship and size ratio relationship.

[0059] Based on the above embodiment, as a preferred implementation mode, the animation result of the hand-drawn elements of the current hand-drawn step is generated in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step, and the animation result of the hand-drawn elements of the previous hand-drawn step is corrected based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, specifically including:

[0060] Input the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step into the trained animation generation model; if the hand-drawn element of the current hand-drawn step is the first hand-drawn element in the hand-drawn screen area, the hand-drawn element of the previous hand-drawn step is a blank drawing paper;

[0061] Based on the animation generation model, the animation result of the hand-drawn elements of the current hand-drawn step is generated in real time in the animation display area, and according to the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, the animation result of the hand-drawn elements of the previous hand-drawn step is corrected.

[0062] In this embodiment, based on two consecutive recognition results of the hand-drawn elements in the previous hand-drawn step, the animation result of the hand-drawn elements in the previous hand-drawn step is corrected;

[0063] If it is determined that the two recognition results of the hand-drawn element in the previous hand-drawn step are different, the previous recognition result is corrected based on the latter recognition result, and the previous animation result is corrected based on the latter animation result.

[0064] Specifically, in the present embodiment, since the hand-drawn elements of the current hand-drawn step and the previous hand-drawn step are analyzed each time, the hand-drawn elements of the previous hand-drawn step have been analyzed once, that is, they have been recognized and animated once. Therefore, if the recognition results of the same hand-drawn element are different twice, correction is required. For example, the first step is to draw the face, and the second step is to draw the eyes. In the animation process of the eyes, the hand-drawn face and eyes are used as input, and the recognition results of the face and eyes are output. If the recognition result of the face this time is the same as the recognition result of the face last time, only the eyes are animated, otherwise the animation of the face is changed while the eyes are animated. This hand-drawn step is to enable the model to have the ability to intelligently learn the relationship between different hand-drawn elements and use this relationship to correct the recognition results online to prevent the face from being recognized as eyes. If the face is recognized as eyes in the first time, the recognition result of the face can be corrected in the second time.

[0065] Since there is no possibility of repeated drawing of hand-drawn elements, and there will be more and more hand-drawn elements in the avatar, therefore, in this embodiment, by comparing the previous and subsequent recognition results, and each time comparing the recognition result of the previous hand-drawn step with the recognition result of the current hand-drawn step, the relationship between the user's hand-drawn elements and elements is automatically identified, and according to the relationship between the hand-drawn elements of the previous hand-drawn step and the current hand-drawn step, the recognition result of the previous hand-drawn step is corrected, the previous recognition can be verified, and in each recognition process, the previous recognition result is corrected based on the next recognition result, and the animation result is adaptively corrected, thereby increasing the reliability of recognition and animation.

[0066] Based on the above embodiment, as a preferred implementation, it also includes:

[0067] Collect hand-drawn elements of different styles and the corresponding animation results; the same hand-drawn element has different painting styles and corresponds to multiple animation results. For the same hand-drawn element, collect its different styles of painting and multiple animation results corresponding to each style of painting, making the sample set more comprehensive.

[0068] A set of image data pairs with each hand-drawn element and the corresponding animation result;

[0069] The image data pairs of different hand-drawn elements are randomly paired in pairs to construct a training sample set. Two different hand-drawn elements are used as input, and the corresponding recognition results and animation results are used as output. Deep learning training is performed to obtain an animation generation model.

[0070] The animation generation model is used to generate the animation result of the hand-drawn elements of the current hand-drawn step in real time based on the hand-drawn elements of the current hand-drawn step, and based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, correct the recognition result of the hand-drawn elements of the previous hand-drawn step, and correct the animation result of the hand-drawn elements of the previous hand-drawn step.

[0071] Specifically, in this embodiment, it is necessary to pre-train an animation generation model, which is an encoding-decoding structure and is used to animate the user's hand-drawn elements in real time.

[0072] When collecting data, it is necessary to find hand-drawn style pictures of different face shapes. The collected data includes: hand-drawn avatar pictures and anime avatar pictures. According to the order of hand-drawing, they are split into basic elements such as eyes, nose, mouth, face, hair, ears, etc., and scaled to the same size (such as 64*64), and a batch of image data pairs such as <hand-drawn eyes, anime eyes>, <hand-drawn nose, anime nose>, <hand-drawn mouth, anime mouth>, <hand-drawn face, anime face>, <hand-drawn hair, anime hair>, <hand-drawn ears, anime ears> are obtained, which are used as the input and output of the model respectively. The basic components of the avatar mainly include face, eyes, nose, mouth, ears, and hair. According to people's painting habits, when collecting training data, the basic elements of different styles are considered. For example, when users hand-draw faces, face shapes are involved, including almond-shaped face, oval face, round face, oblong face, square face, rectangular face, diamond face and triangular face. The shapes of the eyes include standard eyes, phoenix eyes, slender eyes, round eyes, and squinting eyes. Nose shapes include fleshy nose, hooked nose, and lion nose. When collecting hand-drawn and anime pictures, you need to collect pictures of the above-mentioned different styles of elements to increase richness.

[0073] When building the model, in this embodiment, the animation generation model is based on a convolutional neural network as the basic structure and an encoding-decoding as the basic framework. Figure 4As shown in the figure, the basic structure of the model is composed of convolutional neural networks (CNN) in deep learning. The parameters involved in CNN are as follows: Since the model processes the user's hand-drawn elements, their size will not be too large, so a 3*3 convolution kernel is used. The format of the convolution kernel is convolution kernel length * convolution kernel width * number of input channels * number of output channels. For example, a 3*3*3*16 convolution kernel means: the convolution kernel size is 3*3, the number of input channels is 3, and the number of output channels is 16. The stride and image padding used in the convolution layer are shown in the figure. The upsampling layer uses the bilinear interpolation method to enlarge the feature map, and the enlarged size Scale = 2. 1*1 convolution can integrate features and convert dimensions. Therefore, 4 groups of convolutions can reduce the size of the feature map to 1 / 16 of the original image, thereby extracting low-level and high-level features of the input image. The entire model can be expressed as:

[0074] Input->Conv(3,3,3,16)->Conv(3,3,16,32)->Conv(3,3,32,64)->Conv(3,3,64,128)->Ups

[0075] am(2)->Conv(1,1,128,64)->Upsam(2)->Conv(1,1,64,32)->Upsam(2)->Conv(1,1,32,16)->

[0076] Upsam(2)->Conv(1,1,16,3)->Output;

[0077] The parameters of the convolution operation Conv are (convolution kernel width, convolution kernel height, input channel, output channel), and the parameters of upsampling Upsam are (upsampling multiple). The loss function of the model is the mean square error between the model output and the real animated element.

[0078] During model training, in this embodiment, according to the drawing habits of people, the current hand-drawing step and the previous hand-drawing step are used as input, and the animation style is used as output. By minimizing the loss function, the trainable parameters of the model are adjusted by the gradient descent method until the model converges. (For example, the first step is to draw a face, and the second step is to draw eyes. In the animation process of the face, the blank drawing paper and the hand-drawn face are used as input, corresponding to Figure 4 The hand-drawn elements of the user's previous hand-drawn step and the hand-drawn elements of the user's current hand-drawn step output the recognition result and animation result of the face, corresponding to Figure 4The animation results of the hand-drawn elements of the previous hand-drawn step and the animation results of the hand-drawn elements of the current hand-drawn step are shown in the figure. In the animation process of the eyes, the hand-drawn face and eyes are used as input, and the recognition results of the face and eyes are output. If the recognition result of the face this time is the same as the recognition result of the face last time, only the eyes are animated, otherwise the animation of the face is changed while the eyes are animated. This step is to enable the model to have the ability to intelligently learn the relationship between different hand-drawn elements and use this relationship to correct the recognition results online to prevent the face from being recognized as eyes. If the face is recognized as an eye for the first time, the recognition result of the face can be corrected for the second time.

[0079] After the above model training is completed, the model needs to be inferred, specifically including: extracting the hand-drawn elements of the user's current hand-drawn step and the previous hand-drawn step as the input of the trained model, the model automatically identifies the relationship between the user's hand-drawn elements and the elements, and corrects the recognition result of the previous hand-drawn step according to the relationship between the previous and current hand-drawn elements, and outputs the animation result of the current hand-drawn step. For example, after starting the inference, the user hand-draws the first element I1, then the blank canvas (I0) and the first hand-drawn element are respectively used as the previous and current hand-drawn elements, which are input into the model to obtain the animation result, A0, A1, that is, ({A0, A1} = Gen ({I0, I1})); then, the user hand-draws the second element I2, and I1 and I2 are respectively used as the previous and current hand-drawn elements, which are input into the model, that is, ({A1, A2} = Gen ({I1, I2})). Since the loaded model is already a converged model, the model at this time has the ability to correct the recognition result according to the relationship between the hand-drawn elements. According to the relationship between I1 and I2, the model will automatically correct the animation result of I1, and A1 at this time is the corrected result.

[0080] In this embodiment, it is only necessary to use the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step as the input of the animation generation model, so as to simultaneously realize the above-mentioned embodiments of generating the animation result of the hand-drawn elements of the current hand-drawn step in real time based on the hand-drawn elements of the current hand-drawn step, and generating the recognition result of the hand-drawn elements of the current hand-drawn step and the recognition result of the hand-drawn elements of the previous hand-drawn step based on the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step;

[0081] Based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, the recognition result of the hand-drawn elements of the previous hand-drawn step is corrected, and the animation result of the hand-drawn elements of the previous hand-drawn step is corrected.

[0082] Based on the above embodiments, as a preferred implementation, Figure 1 As shown in , it also includes:

[0083] S3. Based on the order of the hand-drawn elements in all current hand-drawn steps, predict the hand-drawn elements of the next hand-drawn step, and recommend the recognition result and animation result of the predicted hand-drawn elements to the user.

[0084] Specifically, in this embodiment, in addition to the real-time animation of the hand-drawn avatar, it is also necessary to provide guiding suggestions for the user's hand-drawing, and guide the hand-drawn elements and styles of the next hand-drawing step. The animated hand-drawn elements are obtained, and the predicted hand-drawn elements and animated elements are flashed on the left and right sides of the screen respectively. By providing the user with predicted hand-drawn elements and animated elements, the hand-drawing difficulty of the user can be reduced, and even non-professional painting users can do hand-drawing. At the same time, it also provides guidance for the user's painting, and can enable the user to choose the style of the drawn image by showing the animation effect to the user in advance.

[0085] Based on the above embodiment, as a preferred implementation manner, before predicting the hand-drawn elements of the next hand-drawn step based on the order of the hand-drawn elements in all current hand-drawn steps, the method further includes:

[0086] The order of hand-drawn elements in the drawing process of different hand-drawn portraits is collected, and the hand-drawn elements are divided into several groups of input-output pairs according to the order of the hand-drawn elements, and the types of hand-drawn elements in each group of input-output pairs are randomly matched to form training samples;

[0087] A neural network training is performed based on the training samples to obtain a hand-drawing prediction model for predicting the hand-drawing elements of the next hand-drawing step.

[0088] Specifically, in this embodiment, the hand-drawing prediction model is a sequence-to-sequence structure, which collects the painting processes and sequences of different hand-drawings, and decomposes the painting process. For example, the face, eyes, nose, mouth, ears, eyebrows, eyelashes, and hair are encoded as numbers 1-8. Taking the hand-drawing sequence of 1-2-7-6-3-4-5-8 as an example, it can be divided into input-output pairs such as <1,2-7-6-3-4-5-8>, <1-2,7-6-3-4-5-8>, and <1-2-7,6-3-4-5-8>. Different types of element combinations (such as oval face with phoenix eyes) are collected to form training samples for training the hand-drawing prediction model.

[0089] Specifically, in this embodiment, the hand-drawn prediction model is based on a recurrent neural network (RNN) as the basic structure, a sequence-to-sequence basic framework, and takes the user's hand-drawn element features as input to output the predicted value of the next element. Figure 5As shown in the figure, in order to obtain the feature vector of the element, a fully connected layer is added on the basis of the animation generation model encoding network. The number of neurons in the fully connected layer is 128, that is, the feature dimension of the user's hand-drawn element is 128. The entire model can be expressed as:

[0090] Input->Conv(3,3,3,16)->Conv(3,3,16,32)->Conv(3,3,32,64)->Conv(3,3,64,128)->

[0091] Flatten->FC(128)

[0092] The parameters of the convolution operation Conv represent (convolution kernel width, convolution kernel height, input channel, output channel), Flatten represents flattening the feature map into a vector, and the parameters of the fully connected operation FC represent (the number of neurons in the fully connected layer). In this way, through the feature extraction network, the picture of the hand-painted element can be converted into a 128-dimensional feature vector as the input of the RNN. The dimensions of the output and hidden layer of the RNN are also set to 128.

[0093] Specifically, in this embodiment, the hand-drawn prediction model network structure is as follows: Figure 6 As shown in the figure, the user's previous hand-drawn steps are extracted respectively, and the features are extracted as the input of the RNN. The last RNN is used to predict the next element. RNN(128,128)->RNN(128,128)->RNN(128,128)->RNN(128,128)->RNN(128,6), where the parameters of the RNN operation are expressed as (number of hidden layer neurons, number of output layer neurons). The number of neurons in the output layer of the last RNN is 6, which represents the predicted values ​​of basic elements such as eyes, nose, mouth, face, hair, and ears. The element with the largest predicted value is taken as the final predicted element, i.e., o i+1 =pred(o1,o2,...,o i ).

[0094] The loss function is the cross entropy between the predicted value and the true value: L = Cross(o i+1 ,o' i+1 )=o' i+1 log(o i+1 ).

[0095] The user's hand-drawing steps are extracted and input into the hand-drawing prediction model, and the predicted elements of the next hand-drawing step are output and input into the animation generation model to obtain the animated elements. The predicted hand-drawn elements and the animated elements are flashed and displayed on the left and right sides of the collection screen respectively.

[0096] Based on the above embodiments, as a preferred implementation, Figure 1 As shown in , it also includes:

[0097] S4. After confirming that the user's hand-drawn avatar is finished, adjust the color and size of each hand-drawn element corresponding to the animation result, and output the final animated avatar.

[0098] Specifically, in this embodiment, the user hand-draws the avatar on the left side, draws the eyes, nose, mouth, head and other elements step by step, extracts the shape, size and other features on the right side, recognizes the user's hand-drawn content in real time, and updates the recognition results and the cartoon image in real time according to the overall content of the user's hand-drawing. After the user finishes hand-drawing, the overall color and size of the cartoon avatar are adjusted, and the final result is output.

[0099] The method of this embodiment can display the results of animation of the user's hand-drawn content in real time, and can correct the recognition results and animation results of the hand-drawn elements in the previous step. At the same time, it can also predict the next hand-drawn element and animate it according to the order of the hand-drawn elements in the previous hand-drawn steps, and provide guiding suggestions for the user. Compared with the prior art that relies on animating the entire avatar at one time, it can maintain the style of the avatar drawn by the user while outputting higher quality animation.

[0100] The embodiment of the present invention further provides a hand-drawn avatar animation system, based on the hand-drawn avatar animation method in the above embodiments, such as Figure 7 As shown in , it includes a hand-drawn element acquisition module 10, an animation generation module 20 and a hand-drawn prediction module 30, wherein:

[0101] A hand-drawn element acquisition module 10 detects the hand-drawn avatar in the hand-drawn screen area;

[0102] If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time;

[0103] The animation generation module 20 generates the animation result of the hand-drawn elements of the current hand-drawn step in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step, and corrects the animation result of the hand-drawn elements of the previous hand-drawn step based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step.

[0104] The hand-drawing prediction module 30 is used to predict the hand-drawing elements of the next hand-drawing step based on the order of the hand-drawing elements in all current hand-drawing steps, and recommend the animation results of the predicted hand-drawing elements to the user.

[0105] Specifically, in this embodiment, the system mainly includes an animation generation model (Gen) and a hand-drawn prediction model (Pred). The images required by the two models include the hand-drawn elements of each step of the user, the hand-drawn elements of the user's current hand-drawn step, and the hand-drawn elements of the user's previous step. These images can be stored in a memory first, and the training method is as shown in the above embodiments.

[0106] Based on the same concept, the embodiment of the present invention also provides a server schematic diagram, such as Figure 8 As shown, the server may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the steps of the hand-drawn avatar animation method described in the above embodiments. For example, it includes:

[0107] S1, detecting the hand-drawn avatar in the hand-drawn screen area;

[0108] If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time;

[0109] S2. Generate an animation result of the hand-drawn elements of the current hand-drawn step in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step.

[0110] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0111] Based on the same concept, an embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores a computer program, which includes at least one code segment, and the at least one code segment can be executed by a main control device to control the main control device to implement the steps of the hand-drawn avatar animation method described in the above embodiments. For example, it includes:

[0112] S1, detecting the hand-drawn avatar in the hand-drawn screen area;

[0113] If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time;

[0114] S2. Generate an animation result of the hand-drawn elements of the current hand-drawn step in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step.

[0115] Based on the same technical concept, an embodiment of the present application also provides a computer program, which, when executed by a main control device, is used to implement the above method embodiment.

[0116] The program may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor.

[0117] Based on the same technical concept, the embodiment of the present application further provides a processor, which is used to implement the above method embodiment. The above processor can be a chip.

[0118] In summary, the embodiments of the present invention provide a method, system, electronic device and storage medium for animating a hand-drawn avatar, which can obtain the newly added hand-drawn elements of each step when the user draws the avatar in real time, and generate the corresponding animation results in real time based on the newly added hand-drawn elements. It can also update the recognition results and the animation image in real time according to the overall content of the user's hand-drawing, and animate the user's hand-drawn elements in real time. The animation is done from scratch, and the user animates while hand-drawing the avatar. During the process of hand-drawing the avatar, the user can obtain the animation results of the currently hand-drawn facial features and other elements in real time, so that the final animation style can be adjusted during the process of hand-drawing the avatar.

[0119] The various embodiments of the present invention can be combined arbitrarily to achieve different technical effects.

[0120] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidStateDisk), etc.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for animating a hand-drawn avatar. It is characterized in that include: Detect hand-drawn avatars in hand-drawn screen areas; If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time; The hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step are input into a pre-trained animation generation model, wherein hand-drawn elements of different styles and corresponding animation results are collected, a set of image data pairs of each hand-drawn element and the corresponding animation result are used, and the image data pairs of different hand-drawn elements are randomly paired in pairs to construct a training sample set, and two different hand-drawn elements are used as input, and the corresponding recognition results and animation results are used as output, and deep learning training is performed to obtain an animation generation model; The animation generation model generates the animation result of the hand-drawn elements of the current hand-drawn step in real time in the animation display area based on the hand-drawn elements of the current hand-drawn step, generates the recognition result of the hand-drawn elements of the current hand-drawn step and the recognition result of the hand-drawn elements of the previous hand-drawn step based on the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, and based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, corrects the recognition result of the hand-drawn elements of the previous hand-drawn step, and corrects the animation result of the hand-drawn elements of the previous hand-drawn step.

2. The method for animating a hand-drawn avatar according to claim 1, It is characterized in that After the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step are input into the trained animation generation model, the method further includes: If the hand-drawn element of the current hand-drawn step is the first hand-drawn element in the hand-drawn screen area, the hand-drawn element of the previous hand-drawn step is a blank drawing paper.

3. The method for animating a hand-drawn avatar according to claim 1, It is characterized in that Also includes: All hand-drawn elements in the current hand-drawn avatar are sequentially input into the trained hand-drawn prediction model to predict the hand-drawn elements of the next hand-drawn step, and the recognition results and animation results of the predicted hand-drawn elements are recommended to the user.

4. The method for animating a hand-drawn avatar according to claim 3, It is characterized in that Also includes: The order of hand-drawn elements in the drawing process of different hand-drawn portraits is collected, and the hand-drawn elements are divided into several groups of input-output pairs according to the order of the hand-drawn elements, and the types of hand-drawn elements in each group of input-output pairs are randomly matched to form training samples; A neural network training is performed based on the training samples to obtain a hand-drawing prediction model for predicting the hand-drawing elements of the next hand-drawing step.

5. The method for animating a hand-drawn avatar according to claim 1, It is characterized in that Also includes: After confirming that the user's hand-drawn avatar is finished, adjust the color and size of each hand-drawn element corresponding to the animation result, and output the final animated avatar.

6. A hand-drawn avatar animation system, It is characterized in that include: A hand-drawn element acquisition module is used to detect hand-drawn avatars in the hand-drawn screen area; If a new hand-drawing step is detected, the hand-drawing elements of the current hand-drawing step are obtained in real time; An animation generation module is used to input the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step into a pre-trained animation generation model, wherein hand-drawn elements of different styles and corresponding animation results are collected, a set of image data pairs of each hand-drawn element and the corresponding animation result are used, and the image data pairs of different hand-drawn elements are randomly paired in pairs to construct a training sample set, and two different hand-drawn elements are used as input, and the corresponding recognition results and animation results are used as output, and deep learning training is performed to obtain an animation generation model; The animation generation model generates the animation result of the hand-drawn elements of the current step in real time in the animation display area based on the hand-drawn elements of the current step, generates the recognition result of the hand-drawn elements of the current hand-drawn step and the recognition result of the hand-drawn elements of the previous hand-drawn step based on the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, and based on the relationship between the hand-drawn elements of the current hand-drawn step and the hand-drawn elements of the previous hand-drawn step, corrects the recognition result of the hand-drawn elements of the previous hand-drawn step, and corrects the animation result of the hand-drawn elements of the previous hand-drawn step.

7. A terminal / electronic device / server, It is characterized in that include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the hand-drawn avatar animation method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the steps of the method for animating a hand-drawn avatar as claimed in any one of claims 1 to 5 are implemented.

Images