A method for performing three-dimensional animation texture special effects on a mobile terminal

By using skeleton map and face recognition technology on the mobile terminal, downsampling videos and establishing human-body relationships before and after frames, the problem of large amount of calculation and consistency of three-dimensional animation maps in the existing technology is solved, and a fast and accurate three-dimensional animation map special effect is achieved. Maintains the consistency of the action in the video.

CN118570349BActive Publication Date: 2024-11-19JIANGSU AUSTIN OPTRONICS TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411055954.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-19
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

The prior art cannot quickly and efficiently perform three-dimensional animation map special effects on mobile terminals, and the existing algorithms have a large amount of calculations, which can easily lead to inter-frame discontinuity and human body occlusion problems, making it difficult to maintain consistency in video.

Method used

Using a method based on skeleton map and face recognition, the human body relationship between the front and back frames is established by downsampling videos, and the human body recognition and motion analysis is performed using OpenPose and YOLOfastest models to reduce the amount of calculations and ensure the three-dimensional dynamic changes and consistency of animation maps.

Benefits of technology

It realizes the three-dimensional animation map special effects in the mobile terminal within minute-level time, maintains the timing consistency of human body movement, reduces the computing resource requirements, and improves the accuracy of human body correspondence between front and back frames.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118570349B_ABST
    Figure CN118570349B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for performing three-dimensional animation sticker special effects on a mobile terminal, characterized in that it includes the following steps: downsampling a video V1 to be processed to obtain a downsampled video V2; the starting frame picture in the video V1 to be processed is the same as the starting frame picture in the downsampled video V2; human body recognition is performed on each frame picture in the video V2, an outer frame mark is added to the recognized human body, the outer frame picture of each frame is converted into skeleton data, and a one-to-one correspondence relationship between the same human body between two adjacent frames is established according to the skeleton data; the human body skeleton data is converted into a 3D human body skeleton posture; a 3D animation sticker is added to the 3D human body skeleton posture, and the 3D animation sticker added to the same human body is the same. The sticker is realized to perform 3D dynamic changes with human body movements; the relationship between the personnel of the previous and next frames is established based on the skeleton diagram, which greatly reduces the amount of calculation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a method for performing three-dimensional animation texture special effects on a mobile terminal. Background Art

[0002] In group activities, videos of collective actions are often shot, such as kindergarten children dancing together, company employee team building, etc. In the later stage, some special effects can be added to these actions, such as adding bunny ears to the children's heads, or Nezha's Hot Wheels, etc., to increase the fun of the video.

[0003] However, at present, there are two main shortcomings of traditional special effects on existing platforms:

[0004] [1]. These existing special effects are 2D and do not change dynamically in 3D with human body movements. For example, they simply put bunny ears on children’s heads statically and do not change with the children’s left and right movements. This is because they do not analyze the changes in human body movements, especially head movements, in 3D posture.

[0005] [2] These existing special effects lack consistency in the video. Sometimes they suddenly disappear and appear as the action changes, and sometimes they become larger or smaller. This is because the existing algorithms do not take into account the consistency of action videos.

[0006] Of course, the advantages of these traditional special effects algorithms are that they require little computation and can achieve real-time effects, and are particularly suitable for processing on mobile terminals.

[0007] If the real-time requirements are not high, that is, after shooting the video, you can return to the mobile phone or computer for analysis and processing; or upload it to the server and let the server process it, then you can use computing power to design new algorithms to make up for the two main shortcomings mentioned above.

[0008] However, at present, the existing post-processing generation algorithms tend to be relatively large models such as GAN and Diffusion, which perform pixel-level reconstruction of the entire image. The advantage of this type of solution is that it can achieve an overall realistic effect, but the disadvantages are also obvious. It not only requires a large amount of computing resources at the server level with a GPU, but also because it is a full pixel-level reconstruction, the original real information of children or team-building workers is likely to be lost. In addition, most of them do not take into account the continuity between frames, which is prone to a "flickering" effect.

[0009] In addition, since there are often many people in team-building videos and they are prone to occluding each other, the common SlowFast algorithm is better at calculating the correspondence between the human bodies in the previous and next frames, but the amount of calculation is too large and is not suitable for calculation on a mobile phone.

[0010] Therefore, if there is an algorithm that can be processed on a mobile terminal within minutes in the later stage to achieve three-dimensional animation mapping along with the movement of the human body and can take into account the consistency of the action video, then it will have great application prospects and practical significance. Summary of the invention

[0011] Based on the existing artificial intelligence technology, the present invention proposes a method for performing 3D animation mapping based on a mobile terminal and realizing special effects for video actions. The method is characterized in that the method can be processed by a mobile terminal within minutes in the later stage of video shooting to achieve 3D animation mapping with the movement of the human body, and can take into account the effect of consistency of the action video, and pay attention to maintaining timing consistency with the movement of the human body. The core idea is to make full use of the resources in each calculation step, try to use addition and subtraction, and reduce the computing power requirements.

[0012] A method for performing three-dimensional animation texture effects on a mobile terminal of the present invention is characterized by comprising the following steps:

[0013] Downsample the video V1 to be processed to obtain a downsampled video V2;

[0014] The starting frame image in the video V1 to be processed is the same as the starting frame image in the downsampled video V2;

[0015] Perform human body recognition on each frame of the video V2, add an outer frame mark to the recognized human body, and obtain the outer frame map of each frame. The outer frame map of the human body numbered hi in the fi-th frame is recorded as IMK(hi,fi);

[0016] Convert the outer frame image of each frame into skeleton data, and record the human skeleton data numbered hi in the fi-th frame as Data1 (hi, fi);

[0017] According to the skeleton data, a one-to-one correspondence between the same human body in two adjacent frames is established;

[0018] Convert human skeleton data into 3D human skeleton posture;

[0019] Add 3D animation textures to 3D human skeleton postures. The 3D animation textures added to the same human body are the same.

[0020] Furthermore, the method of the present invention also includes calculating the center of mass position of each human body in each frame based on human skeleton data, and defining the position of the chest in the skeleton data as the center of mass position of the human body; before establishing a one-to-one correspondence between the same human body between two adjacent frames based on the skeleton data, first determine whether there is occlusion between human bodies in each frame of video V2, and when occlusion between human bodies occurs, perform face recognition on the frame; and establish a one-to-one correspondence with the human body in the initial frame image by means of face recognition.

[0021] Furthermore, when there is no human body occlusion in the two adjacent frames and a one-to-one correspondence can be established for the same human body in the two adjacent frames, the movement speed of each human body in the two adjacent frames, that is, the movement speed of the center of mass of each human body, is calculated, and the center of mass movement speed is judged. When the center of mass movement speed is greater than the threshold T2, face recognition is performed on the frame located at the rear of the two adjacent frames, and a one-to-one correspondence is established with the human body in the initial frame image through face recognition.

[0022] Furthermore, the OpenPose method is used to convert the outer frame image into skeleton data.

[0023] Furthermore, the center of mass position of the human body is the center of mass position of the human body after the jitter is removed and normalized;

[0024] The average centroid hzt(fi) of all people in the fi frame is expressed as:

[0025]

[0026] hz(hi,fi) represents the centroid data of the human body numbered hi in the fi-th frame;

[0027] The calculation formula for the center of mass position hzg1 of the human body after removing jitter and normalization is:

[0028] hzg1(hi,fi)=hz(hi,fi)-hzt(fi)

[0029] In the same frame of the image, it is determined whether the distance between adjacent centroids is less than the threshold Th1. If it is less than the threshold Th1, it is determined to be human body occlusion.

[0030] Furthermore, based on the skeleton data, a one-to-one correspondence between the same human body in two adjacent frames is established, specifically:

[0031] For the centroid hzg1(hi,fi) in the fi-th frame, find the centroid hzg1(hi,fi-1) closest to the centroid in the fi-1 frame and establish a one-to-one correspondence between the centroids, that is, establish a one-to-one correspondence between the same human body between two adjacent frames.

[0032] Furthermore, when a one-to-one correspondence between the human body of two adjacent frames cannot be established according to the aforementioned method, a one-to-one correspondence is established between the two adjacent frames and the human body in the initial frame image through face recognition, and then a human body correspondence between the two adjacent frames is established.

[0033] For example, when the sixth frame and the seventh frame cannot directly establish a one-to-one correspondence, a one-to-one correspondence is established between the two frames and the human body in the initial frame through face recognition, and then a human body correspondence is established between the two frames;

[0034] Furthermore, the motion speed Ve(hi,fi) of the human body numbered hi in the fi-th frame is calculated using the following formula:

[0035] Ve(hi,fi)=hzg1(hi,fi)-hzg1(hi,fi-1).

[0036] Furthermore, a one-to-one correspondence is established with the human body in the initial frame image by means of face recognition, as follows:

[0037] The YOLOfastest model M2 is used to perform face detection on the frame to be recognized, and the face feature matrix Fo(hi,sfi) of the sfi frame is obtained; sfi is the number of the frame that needs to be recognized;

[0038] Compare the facial feature matrix Fo(hi,sfi) with the facial feature matrix Fo(hi,1) of the starting frame image, and establish a one-to-one correspondence with the human body in the starting frame image.

[0039] Furthermore, 3D animation textures are added to the 3D human skeleton posture, that is, 3D animation textures are added to each frame in video V2; the frames deleted due to downsampling are respectively inserted into video V2; the insertion position is the same as the position in video V1; 3D animation textures are added to the inserted frames, and the 3D animation textures are the same as the 3D animation textures of the frames before the insertion position. For example, two frames are inserted after the frame numbered 5 in video V2, and 3D animation textures are added to the two frames, and the added 3D animation textures are the same as the 3D animation textures added to the frame numbered 5.

[0040] Furthermore, the starting frame image refers to a selected image in the video that contains a frontal image of a human face and contains all the characters to be added with special effects;

[0041] Furthermore, the method of the present invention further comprises respectively calculating the total number of pixels of each human frame in the starting frame image; if the total number of pixels of a single human frame is lower than a set threshold, the human frame is deleted, and subsequent corresponding relationship establishment and face recognition are no longer performed on the human body;

[0042] Beneficial effects: The method provided by the present invention for performing three-dimensional animation mapping special effects on a mobile terminal realizes three-dimensional dynamic changes with human body movements; the relationship between the human body in the previous and next frames is established based on the skeleton diagram, which greatly reduces the amount of calculation, and at the same time combines face recognition to improve the accuracy of establishing the corresponding relationship between the human body in the previous and next frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention requires a precondition that the field of view of the shooting camera is facing the entire team building personnel, and the shooting camera remains basically motionless. On this basis, the present invention is a method for performing three-dimensional animation mapping special effects on a mobile terminal, which mainly includes preprocessing, establishing the relationship between the front and back frames of personnel based on a skeleton diagram, establishing the relationship between the front and back frames of personnel based on face recognition, 3D posture analysis of the human body, 3D mapping and other steps.

[0045] Preprocessing refers to controlling the resolution and frames per second of the original video to meet the needs of later analysis and reduce computing workload.

[0046] Among them, the relationship between the previous and next frames of the skeleton graph is established. When facing the video, the human body area is first discovered and the image including the human body area is segmented. Then the corresponding relationship between the human body needs to be found between consecutive frames, and different mapping task requirements are achieved for different characters. For example, the mapping effect of child A is rabbit ears, and the mapping effect of child B is Little Red Riding Hood, etc. It is characterized by being fast, but it may cause errors.

[0047] Among them, the establishment of the relationship between the previous and next frames based on face recognition is to use the face recognition algorithm to make up for the tracking errors that may occur when finding the continuous relationship between the previous and next frames, when there is occlusion, when there are one-to-many people in the previous and next frames, when the movement speed exceeds the human limit, and when there is a fixed interval frame (in this implementation, it is every 30 frames in V2, that is, once every 3 seconds), so as to make up for the tracking errors that may occur when finding the continuous relationship between the previous and next frames, that is, mistaking child A for child B. Its characteristics are accuracy, but slow speed.

[0048] Among them, the role of human body 3D posture analysis is to use the time series relationship to convert 2D skeleton human body posture into 3D human body posture, that is, 2D+ skeleton posture into 3D human body skeleton posture.

[0049] Among them, the role of 3D mapping is to convert the special effects posture of animation actions and combine them with videos.

[0050] A method for performing three-dimensional animation texture effects on a mobile terminal of the present invention specifically comprises the following steps:

[0051] Step 1. Preprocessing

[0052] Some videos shot by users have high or low resolutions, and some have high frames per second. Fortunately, when building a team, we can ask users for the minimum requirements for this issue. We need to analyze the situation where the resolution is very low and affects the later analysis, and ask users for the requirements. For videos with high frame rates, we can downsample them to reduce the amount of calculation in the later stage.

[0053] Step 1.1 Select the first frame of the video as the starting frame image for a detection, analyze the pixel size of a single human action area, and determine whether it meets the minimum pixel requirement of the analysis. Then the user can select different mapping effects for different objects.

[0054] Considering that in the process of team-building video, the front and rear frame face angles and expressions will produce various changes, and the first frame at the beginning is generally static, and the expression is generally positive and neutral. Therefore, in order to improve stability, when the amount of calculation is not large, the first frame is selected as the starting frame picture of this method to start processing. The present invention compares the front and rear frame faces that need to be detected with the first frame, and uses the first frame as an intermediary to establish the corresponding relationship of the front and rear frame faces. In the embodiment of the present invention, the YOLOfastest algorithm is adopted, which has the characteristics of lightweight and fast speed, and can quickly detect the human body. Based on the YOLOfastest model M1 trained on the AVA data set, the first frame picture is input, and the human body area in the picture can be obtained, and then the outer frame of all areas is marked to obtain the outer frame picture of each human body. Then the total number of pixels of each human body picture detected is counted, and if there is a situation lower than the threshold value Th1, then the user is warned, indicating that the pixel resolution of a single human body is low, which may affect the development of later work.

[0055] At this moment, if the user feels that this person is not important, or there is an error, then delete the problematic human body area and stop tracking it, and continue with the next work, selecting the animation special effects for each person, for example, some children may add rabbit ears, while others may add Hot Wheels.

[0056] At the same time, feature detection is performed on the face of each person to be tracked. Similarly, based on the YOLOfastest model, the YOLOfastest model M2 trained on the public data set CASIA-FACEV5 is used to detect the face, save the label number hi, and save the feature parameters of the 17th, 20th, and 23rd feature extraction layers of the face image corresponding to each face number hi on the YOLOfastest model M2 to form a feature matrix Fo(hi,fi). At this moment, fi=1, that is, the first frame.

[0057] If the user thinks this person is important, then a request is made to reselect the starting frame image until the requirements of this step are met.

[0058] Step 1.2 Generally speaking, 3D animation does not require a very high temporal resolution, so for videos with a high frame rate, you can downsample them, and then directly add the texture effect of the previous frame in the previous and next frames, thereby reducing the amount of calculation in the later steps of human body recognition and segmentation, facial posture analysis, motion skeleton analysis, motion prediction, etc.

[0059] In a specific embodiment of the present invention, the number of frames per second in the video (originally generally 30 frames or 60 frames) is downsampled to 10 frames, and then the texture effect of the first frame of the 10 frames is added to the original number of frames. Specifically, for example, assuming that in the original video V1 with 30 frames per second, the image numbers of the 30 frames contained in 1 second are:

[0060] V1={I1,I2,I3,……I30}

[0061] After this round of scheme, it is reduced to 10 frames, so 1 second contains 10 frames of images, and its relationship with the above image is:

[0062] V2={I1,I4,I7,……,I28}

[0063] Then, the present invention processes the images in V2, including human body recognition and segmentation, facial posture analysis, motion skeleton analysis, motion prediction, etc., but in the final 3D mapping module, V1 is mapped, wherein the mapping effects of I2 and I3 refer to I1 in V2; the mapping effects of I5 and I6 refer to I4 in V2. In this way, the final 30-frame effect is still presented to the user, but the calculation work on human body recognition and segmentation, facial posture analysis, motion skeleton analysis, motion prediction, etc. for each frame is reduced to 2 / 3 of the total number of video frames.

[0064] Step 2. Establish the relationship between the previous and next frames of the person based on the skeleton graph

[0065] When faced with a video, the first step is to discover the human body areas and segment their images. Then, it is necessary to find the corresponding relationship between the human body between consecutive frames in V2 to meet different mapping task requirements. For example, the mapping effect of child A is rabbit ears, and the mapping effect of child B is Little Red Riding Hood, etc.

[0066] In this task, a specific problem to be solved is that the participants in the video are moving, for example, there may be occlusion between moving children, and the present invention hopes to run on a mobile terminal. Therefore, here, the present invention needs to complete the human body recognition and segmentation tasks under the condition of limited computing resources as much as possible.

[0067] In this module, the main idea is to detect the human skeleton and extract the skeleton area and the face area; secondly, analyze the skeleton area and use the movement speed relationship to find the front-back continuity relationship;

[0068] Because the work of human skeleton detection and skeleton area search is what must be done in the task process to be solved in the present invention, and the computing resource demand is less relative to face recognition.Based on this premise, the present invention has designed the scheme based on this prerequisite, i.e. mainly utilizing the skeleton front and back frame continuous relationship to carry out human tracking, the computing resources required for this tracking scheme are seldom, and the original resources are fully utilized, but misjudgment may occur under the blocking condition.So, the tracking algorithm module part of this scheme is a kind of hybrid algorithm, which is realized by combining the face recognition algorithm with high accuracy and the skeleton speed relation with low accuracy, i.e. only in the front and back frames where blocking occurs, and on the fixed interval frame (for regular quality inspection), the face recognition algorithm with high accuracy is used to make up for the misjudgment of the tracking algorithm only by the speed relation.

[0069] In this step, the difficulties that need to be solved are:

[0070] [1]. Sometimes the photographer uses a mobile phone to shoot handheld, so the video is likely to have a small range of shaking.

[0071] [2]. Since the pixel-level movement speed of a person close to the camera will be larger than that of a person far from the camera at the same movement speed, the movement speed needs to be normalized according to its depth relative to the camera.

[0072] Step 2.1 video segment extraction, wherein the video segment refers to the human body frame image in each frame;

[0073] The YOLOfastest algorithm is adopted, which is lightweight and fast. It can quickly detect human bodies. Based on the YOLOfastest model M1 trained on the AVA dataset, in each frame of the video V2, the human body area in the picture is obtained, and then the outer frames of all areas are marked to obtain and save the outer frame image IMK(hi,fi) of each human body hi, where hi represents the human body number, from 1 to himax, himax is the maximum number of people confirmed by the user in step 1.1, and fi represents the fith frame, from 1 to fimax, and fimax is the maximum number of frames in the video.

[0074] Step 2.2 Skeleton graph conversion.

[0075] Using the OpenPose method, the outer frame image in each frame obtained in step 2.1 is converted into skeleton data Data1 (hi) for further analysis, where hi is the hi-th person, which has been confirmed by the user in step 1.1. Those that are not needed, and the remaining selections are the animation effects for each person.

[0076] OpenPose is an open source library of human posture recognition algorithms developed by Carnegie Mellon University (CMU) in the United States based on convolutional neural networks and supervised learning and using Caffe as the framework. It can realize posture estimation of human 2D actions, facial expressions, finger movements, etc. Its description and source code can be found at the website https: / / github.com / CMU-Perceptual-Computing-Lab / openpose. In order to increase the speed and achieve lightweight deployment, the present invention only cuts out and uses the skeleton map recognition module of human 2D actions, and does not use its facial expression and finger movement modules.

[0077] Step 2.3 Human body matching and anomaly detection based on the centroid of the previous and next frames of the skeleton image

[0078] The position of the chest in the skeleton image calculated by OpenPose (in the original video image, i.e., the V2 video in the previous step 1.2) is used as the center of mass of the person hz(hi,fi)=(x,y), where hi represents the hi-th person, from 1 to himax, i.e., the maximum number of people, fi represents the fi-th frame, and x, y represent the positions on the horizontal and vertical coordinates of the image, respectively.

[0079] Then, the average center of mass of all people in each frame hzt(fi) is calculated as follows:

[0080]

[0081] Then, by subtracting the average value of the center of mass of the frame hzt(fi) from the center of mass of each person in the frame, the motion deviation caused by small-scale jitter can be roughly eliminated, and a normalized value hzg1 relative to the movement of the entire team can be obtained, that is:

[0082] hzg1(hi,fi)=hz(hi,fi)-hzt(fi)

[0083] Then, the fi-1 frame is de-jittered to obtain the centroid hzg1(hi,fi-1) of each person in the fi-1 frame;

[0084] Here, the threshold Th1 is used to determine whether there is another center of mass position of each hi in the center of mass hzg1 in two adjacent frames, and the distance between it and hi is less than Th1. If so, it is determined that occlusion occurs, and then the face recognition module is used to enter step 3.

[0085] If not, then find the centroid hzg1(hi,fi) in the fi frame and the centroid hzg1(hi,fi-1) closest to the centroid in the fi-1 frame, and establish their relationship mapping, that is, the personnel relationship of the previous and next frames is corresponding. At this point, if there is a many-to-1 state between the centroids of the previous and next frames, this is questionable, and the face recognition module is also used, and enter step 3.

[0086] At this moment, if it is a 1 to 1 result, then calculate the movement speed of each person, that is:

[0087] Ve(hi,fi)=hzg1(hi,fi)-hzg1(hi,fi-1)

[0088] Ve(hi,fi) represents the movement speed of the human body numbered hi in the fi-th frame;

[0089] Step 2.4: Detect anomaly of human body matching between front and back frames based on the change of speed of front and back frames of skeleton graph

[0090] The distance between the head point and the chest, the distance between the chest and the hip bone, the distance between the hip bone and the knee, and the distance between the knee and the ankle are added together to form the approximate height hh(hi,fi) of the person in each frame. Note that the approximate height of each frame is different. The reason for designing this parameter here is that the movement limit of each person is related to his or her height, and generally does not exceed a certain multiple of his or her height. In this way, we can divide the movement pixel distance by the height distance, thereby roughly controlling the inconsistent effect caused by the distance from the camera. The movement distance is normalized according to its depth relative to the camera.

[0091] Veg(hi,fi)=Ve(hi,fi) / hh(hi,fi)

[0092] hh(hi,fi) represents the height of the person numbered hi.

[0093] Then, use the threshold Th2 to judge each Veg(hi,fi). If there is a value greater than Th2, it means that the person's movement exceeds the limit of human movement and is suspicious, and go to step 3.

[0094] At the same time, keep hh(hi,fi), which will be used to calculate the 3D texture and body proportion relationship in the following step 4.

[0095] Step 3. Establishing the relationship between people in the previous and next frames based on face recognition

[0096] This step uses a face recognition algorithm to compensate for possible tracking errors that may occur when looking for the continuous relationship between the front and back frames when there is occlusion, when there are one person in the front and back frames and when the movement speed exceeds the human limit and the frames are fixed at intervals (in this implementation, every 30 frames in V2, i.e. once every 3 seconds), that is, mistaking child A for child B.

[0097] Assume that the number of frames required for face recognition detection is sfi-1, sfi, and its relationship with fi is known during the running process in step 2.

[0098] Step 3.1 Here, call the outer frame image IMK(hi,fi) of each human body hi corresponding to the sfi-1 and sfi frames in step 2.1. Similarly, based on the YOLOfastest model, use the YOLOfastest model M2 trained on the public dataset CASIA-FACEV5 to detect faces. Extract the feature parameters of the 17th, 20th, and 23rd feature extraction layers of the face image corresponding to each face number hi on the YOLOfastest model M2 to form the face feature matrix Fo(hi,sfi-1) and Fo(hi,sfi).

[0099] Step 3.2 calls the saved tag number hi in step 1.1, and saves the feature parameters of the 17th, 20th, and 23rd feature extraction layers of the face image corresponding to each face number hi on the YOLOfastest model M2 to form the face feature matrix Fo(hi,fi). At this moment, fi=1, that is, the first frame.

[0100] Step 3.3 compares the facial feature matrix Fo(hi,sfi) of the sfi frame with the facial feature matrix Fo(hi,1) of the first frame; for each person in the sfi frame, find the person who is most similar to him in the first frame, and update the number of the person in the sfi frame so that the number of the person in the sfi frame is the same as the number of the most similar person in the first frame; that is, in the sfi frame and the first frame, the numbers of similar people are the same;

[0101] According to the different hi of the two frames, the facial feature matrices Fo(hi,sfi-1) and Fo(hi,1) are cross-subtracted in turn, and then the sum of the absolute values ​​of the differences in each feature parameter in the matrix is ​​calculated to obtain the similarity Sim(hi,hi) between each face in the sfi-1 frame and each face in the first frame.

[0102] For example, subtract the feature matrix of the first person in the sfi-1 frame from the feature matrix of the first person in the 1st frame, and then calculate the sum of the absolute values ​​of the differences in each feature parameter in the matrix to obtain Sim(1,1). The first 1 is the first person in the sfi-1 frame, and the second 1 is the first person in the 1st frame.

[0103] Subtract the feature matrix of the second person in the sfi-1 frame from the feature matrix of the first person in the first frame, and then calculate the sum of the absolute values ​​of the differences in each feature parameter in the matrix to obtain Sim(2,1). The first 2 is the second person in the sfi-1 frame, and the second 1 is the first person in the first frame.

[0104] Then, based on the principle of minimum Sim value, find the correspondence between all people in the sfi-1 frame and all people in the 1st frame.

[0105] Using the same method, find the correspondence between all people in the sfi frame and all people in the first frame.

[0106] Then, taking the person in frame 1 as the median, find the correspondence between all persons in the sfi-1 frame and all persons in the sfi frame.

[0107] In this way, through steps 1 to 3, the human body relationship pairing of the previous and next frames can be achieved with very little computing resources, so as to achieve the effect of tracking the movements of multiple people in team-building videos on a mobile phone.

[0108] Step 4. Human 3D posture analysis

[0109] The human skeleton image obtained by OpenPose is a 2D human posture, not a 3D human posture. However, the present invention is oriented to 3D animation mapping, so it is necessary to use the timing relationship to convert the 2D skeleton human posture into a 3D human posture, that is, convert the 2D skeleton posture into a 3D human skeleton posture.

[0110] The present invention adopts part of the model in the paper "MonoPerfCap: Human Performance Capture from MonocularVideo", which is from the journal ACM Transactions on Graphics, Vol. 9, No. 4, 2018, No. 39; in this embodiment, the preprocessing of Part 1 and the mapping of Part 3 in the paper model are removed, and only Part 2 is used. This is because the calculation amount of Parts 1 and 3 of the model is very large, so they are not adopted. In addition, the part of extracting 2D+skeletal points through the trajectory point subspace matching in Part 2, since the present invention has obtained the 2D points matched in time sequence in the previous steps 1 to 3, it can directly form the skeleton point trajectory point subspace graph, so the module of 2D+skeletal points through the trajectory point subspace matching is also deleted. Only the Batch-based pose estimation module of Part 2 in the MonoPerfCap model, which converts 2D+skeletal posture to 3D human skeleton posture, is retained, so the calculation amount is greatly reduced compared with the original LiveCap calculation amount, and it can be run on a mobile phone.

[0111] Step 5. 3D Animated Texture

[0112] After obtaining the 3D human skeleton posture, the present invention adjusts the posture of the 3D animation map according to the skeleton posture, so that the rabbit ears animation is consistent with the direction of the face, the Hot Wheels animation is consistent with the posture of the straight line of the knee + ankle, etc.

[0113] Then, the size of the 3D animation map is adjusted according to the approximate height hh(hi,fi) of the person in each frame stored in step 2.4 as a ratio.

[0114] Then, according to the 3D human skeleton posture, the plane formed by each person's chest + foot bones is taken as the layer where the person is located, and the human body outer frame IMK(hi,fi) obtained in step 2.1 is used as the content space of the plane layer. In addition, the previous 3D posture analysis and sized 3D animation map are also used as the content space of this layer. A simple mask calculation work is performed to remove the layers of occlusions from the distant objects, and the final image is obtained, which corresponds to each frame of V2 at this moment.

[0115] V3={UI1, UI4, UI7,...,UI28}.

[0116] Among them, UI1 indicates that the picture with 3D animation stickers added in the first frame

[0117] Then, U1=UI1-I1, U4=UI4-I4, and so on are subtracted in sequence to obtain the actual added 3D animation map content in addition to the content.

[0118] U={U1, U4, U7,..., U28}

[0119] Finally, return to step 1.2 and add these special effects to the original downsampled frames in turn to get:

[0120] UV1={UI1, I2+U1, I3+U1, UI4, I5+U4, I6+U4, ....UI28,

[0121] I29+U28,I30+U28}

[0122] This is the final result.

[0123] In general, the present invention uses OpenPose and MonoPerfCap for cropping, utilizes the YOLOfastest algorithm, and realizes the matching and tracking of people in the previous and next frames through addition and subtraction, replacing the massive convolution modules in the first and third stages of the existing algorithms such as MonoPerfCap. Its advantages are fast calculation speed and can be oriented to mobile terminals. Its disadvantages are that it integrates multiple models and has a relatively large storage memory requirement, but it is acceptable to general mobile phones.

[0124] By implementing the present invention, a method for realizing 3D animation texture special effects for team-building videos based on existing mobile terminal resources is realized. The method has the advantage that under the existing mobile terminal computing resources, it can pay attention to the movement of the human body and maintain timing consistency.

Claims

1. A method for performing three-dimensional animation texture effects on a mobile terminal, characterized in that: The steps include: Downsample the video V1 to be processed to obtain a downsampled video V2; The starting frame image in the video V1 to be processed is the same as the starting frame image in the downsampled video V2; Perform human body recognition on each frame of the video V2, add an outer frame mark to the recognized human body, and obtain the outer frame map of each frame. The outer frame map of the human body numbered hi in the fi-th frame is recorded as IMK(hi,fi); Convert the outer frame image of each frame into skeleton data, and record the human skeleton data numbered hi in the fi-th frame as Data1 (hi, fi); Calculate the center of mass position of each human body in each frame based on the human skeleton data, and define the position of the chest in the skeleton data as the center of mass position of the human body; determine whether there is occlusion between human bodies in each frame of video V2; The center of mass position of the human body is the center of mass position of the human body after the jitter is removed and normalized; The average centroid hzt(fi) of all people in the fi frame is expressed as: hz(hi,fi) represents the centroid data of the human body numbered hi in the fi-th frame; The calculation formula for the center of mass position value hzg1 of the human body after removing jitter and normalization is: hzg1(hi,fi)=hz(hi,fi)-hzt(fi); In the same frame of the image, determine whether the distance between adjacent centroids is less than the threshold Th1. If it is less than the threshold Th1, it is determined to be human occlusion; When there is occlusion between human bodies, face recognition is performed on the frame; a one-to-one correspondence is established with the human body in the starting frame image through face recognition; For frames without human body occlusion, a one-to-one correspondence between the same human body in two adjacent frames is established based on the skeleton data, specifically: For the centroid hzg1(hi,fi) in the fi-th frame, find the centroid hzg1(hi,fi-1) closest to the centroid in the fi-1 frame, and establish a one-to-one correspondence between the centroids, that is, establish a one-to-one correspondence between the same human body in two adjacent frames; Convert human skeleton data into 3D human skeleton posture; Add 3D animation textures to the 3D human skeleton posture, that is, add 3D animation textures to each frame in video V2. The 3D animation textures added to the same human body are the same. The frames deleted due to downsampling are then inserted into video V2 respectively; The insertion position is the same as that in video V1; A 3D animation map is added to the inserted frame, wherein the 3D animation map is the same as the 3D animation map of the frame before the insertion position.

2. According to claim 1, a method for performing three-dimensional animation texture effects on a mobile terminal is characterized in that: When there is no human body occlusion in the two adjacent frames and a one-to-one correspondence can be established between the same human body in the two adjacent frames, the movement speed of each human body in the two adjacent frames, that is, the movement speed of the center of mass of each human body, is calculated, and the center of mass movement speed is judged. When the center of mass movement speed is greater than the threshold T2, face recognition is performed on the frame located at the rear of the two adjacent frames, and a one-to-one correspondence is established with the human body in the initial frame image through face recognition.

3. According to claim 1, a method for performing three-dimensional animation texture effects on a mobile terminal is characterized in that: The OpenPose method is used to convert the outer frame image into skeleton data.

4. According to claim 1, a method for performing three-dimensional animation texture effects on a mobile terminal is characterized in that: When a one-to-one correspondence between the human body of two adjacent frames cannot be established according to the method of claim 1, a one-to-one correspondence is established between the two adjacent frames and the human body in the initial frame image by means of face recognition, and then a human body correspondence between the two adjacent frames is established.

5. According to claim 2, a method for performing three-dimensional animation texture effects on a mobile terminal is characterized in that: Calculate the movement speed Ve(hi,fi) of the human body numbered hi in the fi frame. The calculation formula is: Ve(hi,fi)=hzg1(hi,fi)-hzg1(hi,fi-1).

6. According to claim 1, a method for performing three-dimensional animation texture effects on a mobile terminal is characterized in that: A one-to-one correspondence is established with the human body in the initial frame image through face recognition, as follows: Use the YOLOfastest model M2 to perform face recognition on the human frame in the starting frame image, and obtain the face feature matrix Fo(hi,1) of the starting frame image; The YOLOfastest model M2 is used to perform face detection on the frame to be recognized, and the face feature matrix Fo(hi,sfi) of the sfi frame is obtained; sfi is the number of the frame that needs to be recognized; Compare the facial feature matrix Fo(hi,sfi) with the facial feature matrix Fo(hi,1) of the starting frame image, and establish a one-to-one correspondence with the human body in the starting frame image.