Method for generating animation through figure stick figures based on YoloV8

Abstract

The invention discloses a method for generating an animation by using figure stick figures based on YoloV8. The method can effectively generate the animation by using a static figure stick figure. Detecting a figure in the stick figure through YoloV8 to obtain the position of the drawing figure; extracting figures in the picture through OpenCV; estimating a figure posture in the picture through the MMPose; figure texture materials are manufactured through algorithms such as triangular segmentation and are applied to action track data obtained through recognition of deep learning algorithms such as Alpha Pose, and therefore animation generation of figure stick figures is achieved. According to the invention, the drawing becomes more vivid and interesting, brand new creation experience and fun are brought to the user, and the infinite imagination and creativity of the user are stimulated.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a method for applying deep learning technologies such as YoloV8 and MMPose to artistic creation. Background Art

[0002] With the development of artificial intelligence technology, especially in the field of computer vision, more and more applications are beginning to be integrated into all aspects of daily life. Especially in the field of children's digital education and digital entertainment, artificial intelligence technology is changing children's learning methods and entertainment experience. Children's paintings are not only a way for them to express their ideas, but also an important carrier to show their unlimited imagination. However, children's paintings are mainly simple sketches, lacking dynamic forms of expression. With the development of deep learning technology, it is now possible to convert children's simple sketches into animations. Through image detection and segmentation technology, the character elements in the painting can be extracted from the image, and the motion estimation technology can be used to make these characters that were originally stationary in the painting move. This technology not only makes the painting more vivid and interesting, but also brings children a new creative experience and fun, and stimulates children's unlimited imagination and creativity. This method is based on such a technical background, and aims to make static simple sketches of characters move through this method. Summary of the invention

[0003] The patent of this invention provides a method for generating animations from simple character drawings based on YoloV8, the purpose of which is to realize the conversion of simple character drawings into dynamic animations through the combination of artificial intelligence technology, especially deep learning models and computer vision technology.

[0004] The present invention provides a method for generating animation based on YoloV8's stick figure drawings, characterized in that it comprises the following steps: Step S1: inputting stick figure drawings through various methods, including hand-drawing, photographing and importing from photo albums on a mobile device, and pre-processing the pictures; Step S2: Based on the YoloV8 target detection model, the image input in step S1 is detected, the bounding box of the painted figure is obtained, and the image is cropped; Step S3: Extracting the painted figures from the image obtained in step S2 based on OpenCV; Step S4: perform pose estimation on the image in step S3 based on MMPose; Step S5: making a suitable action template and applying it to the picture; Step S6: Generate an animation by processing the person in the painting image extracted in step S3, the posture skeleton of the person in the painting extracted in step S4, and the action template obtained in step S5.

[0005] Further, step S2 includes: Step S2.1: Use Labelimg to annotate the collected stick figures, identify the positions of the figures, and create a data set; Step S2.2: Input the labeled data set produced in step S2.1 into the YoloV8 network for training to obtain a target detection model; Step S2.3: Based on the object detection model trained in step S2.2, the input image is detected, the bounding box of the painted person is obtained, and the image is cropped; Further, step S3 includes: Step S3.1: Binarize the image obtained in step S2 using an adaptive threshold algorithm; Step S3.2: removing noise from the binary image obtained in step S3.1 by a closing operation; Step S3.3: Fill the image obtained in step S3.2 by flood filling to ensure that there are no holes in the closed area; Step S3.4: Perform contour detection on the closed area obtained in step S3.3, calculate the area of ​​each closed area, retain the largest area, and obtain a mask image of the painted figure; Step S3.5: Apply the mask of step S3.4 to the image of step S2 to complete the extraction of the painted figure; Further, step S5 includes: Step S5: Create a suitable action template and apply it to the image, including: Step S5.1: Upload a video containing human dancing and separate the video and audio through FFmpeg; Step S5.2: Estimating the pose of the video in step S5.1 based on Alpha Pose to obtain the skeleton of the dancing person; Step S5.3: Convert the skeleton in step S5.2 into bvh format by processing the data format, and package and store it with the audio in step S5.1 to complete the production of the action template; Further, step S6 includes: Step S6.1: generate a 2D mesh by using Delaunay triangulation on the person mask image extracted in step S3, and apply it to the original image in step S1 to texture it; Step S6.2: Applying the action template in step S5 to the person in the painting picture, specifically comprising: Step S6.2.1: By performing principal component analysis on the skeleton of step S5 across the time dimension, a plane with the most intense movement is obtained as the projection plane of the action; Step S6.2.2: Subtract the X and Z coordinates of the root joint from the skeleton in step S5 so that the root joint is located at the origin of the coordinate system. Then, rotate along the Y axis so that the vector of the average value of the vector connecting the shoulder joint and the vector connecting the hip joint is perpendicular to the X axis; Step S6.2.3: for the bones of the lower arm, upper limb, lower limb, neck and spine of the human skeleton in step S6.2.2, respectively calculate their positions on the projection planes obtained in step S6.2.1 and rotate these bones; Step S6.3: Use the ARAP method to draw the texture material in step S6.2 onto the skeleton in step S6.2.3, so as to generate the corresponding animation from the painted character.

[0006] Through the above steps, the present invention realizes the conversion from static figure sketches to animations, making the drawings more vivid and interesting, thereby more intuitively expressing the creativity and emotions of the painter.

[0007] The beneficial effect of the present invention is that the given method can accurately extract the character elements of the stick figure drawing, and give life to these elements through the posture estimation technology, so that the originally static picture can be animated, bringing a new creative experience and fun to the creator. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram of the overall process of the present invention; Figure 2 The figure is a schematic diagram of the process of extracting characters from paintings using OpenCV in the present invention; Figure 3 The figure is a flow chart of making action templates according to the present invention. DETAILED DESCRIPTION

[0009] The present invention provides a method for generating animations from stick figures of characters based on YoloV8. The present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0010] Figure 1 The overall process of the present invention is shown. First, the position of the target person is detected by YoloV8; the painted person is extracted based on OpenCV; the posture of the picture person is estimated based on MMPose; and the extracted action template is applied and mapped to the picture person to complete the animation rendering.

[0011] Figure 2 The present invention shows the process of extracting characters from paintings by using a series of algorithms such as adaptive threshold, closing operation, flood filling, and contour detection through OpenCV.

[0012] Figure 3 It is shown that the present invention separates the audio and video of the input video, estimates the posture of the video, converts the format, obtains the trajectory data of the character's movement, and packages the audio and movement trajectory data to make an action template.

[0013] In summary, the present invention discloses a method for generating animation from a simple drawing of a person based on YoloV8, which can effectively generate an animation from a drawing of a person.

[0014] The above describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the protection scope of the present invention in any form, and all technical solutions obtained by equivalent replacement and the like fall within the protection scope of the present invention.

[0015] The parts not involved in the present invention are the same as the prior art or can be implemented by using the prior art.

Claims

1. A method for generating animation based on YoloV8's stick figure drawing, characterized in that: The following steps are involved: Step S1: inputting stick figure drawings through various methods, including hand-drawing, photographing and importing from photo albums on a mobile device, and pre-processing the pictures; Step S2: Based on the YoloV8 target detection model, the image input in step S1 is detected, the bounding box of the painted figure is obtained, and the image is cropped, which specifically includes: Step S2.1: annotate the collected stick figure pictures, identify the positions of the characters, and create a data set; Step S2.2: Input the labeled data set produced in step S2.1 into the YoloV8 network for training to obtain a target detection model; Step S2.3: Based on the object detection model trained in step S2.2, the input image is detected, the bounding box of the painted person is obtained, and the image is cropped; Step S3: Extracting the painted figures from the image obtained in step S2 based on OpenCV, specifically including: Step S3.1: Binarize the image obtained in step S2 using an adaptive threshold algorithm; Step S3.2: performing a closing operation to remove noise from the binary image obtained in step S3.1; Step S3.3: Fill the image obtained in step S3.2 by flood filling to ensure that there are no holes in the closed area; Step S3.4: Perform contour detection on the closed area obtained in step S3.3, calculate the area of ​​each closed area, retain the largest area, and obtain a mask image of the painted figure; Step S3.5: Apply the mask of step S3.4 to the image of step S2 to complete the extraction of the painted figure; Step S4: perform pose estimation on the image in step S3 based on MMPose; Step S5: Create a suitable action template and apply it to the image, including: Step S5.1: Upload a video containing human dancing and separate the video and audio through FFmpeg; Step S5.2: Estimating the pose of the video in step S5.1 based on Alpha Pose to obtain the skeleton of the dancing person; Step S5.3: Convert the skeleton in step S5.2 into bvh format by processing the data format, and package and store it with the audio in step S5.1 to complete the production of the action template; Step S6: Generate an animation by processing the person in the painting image extracted in step S3, the posture skeleton of the person in the painting extracted in step S4, and the action template obtained in step S5, specifically including: Step S6.1: generate a 2D mesh by using Delaunay triangulation on the person mask image extracted in step S3, and apply it to the original image in step S1 to texture it; Step S6.2: Applying the action template in step S5 to the person in the painting picture, specifically comprising: Step S6.2.1: By performing principal component analysis on the skeleton of step S5 in the time dimension, a plane with the most intense movement is obtained as the projection plane of the action; Step S6.2.2: Subtract the X and Z coordinates of the root joint from the skeleton in step S5 so that the root joint is located at the origin of the coordinate system. Then, rotate along the Y axis so that the vector of the average value of the vector connecting the shoulder joint and the vector connecting the hip joint is perpendicular to the X axis; Step S6.2.3: for the bones of the lower arm, upper limb, lower limb, neck and spine of the human skeleton in step S6.2.2, respectively calculate their positions on the projection planes obtained in step S6.2.1 and rotate these bones; Step S6.3: Use the ARAP method to draw the texture material in step S6.2 onto the skeleton in step S6.2.3, so as to generate the corresponding animation from the painted character.

Images