A method for generating transition frames of sign language videos based on skeleton migration

Abstract

The present invention relates to a template- and entity-based sign language video combination method, comprising the following steps: skeleton information extraction: continuous frame skeleton extraction is performed on each video frame, and skeleton coordinates are stored as intermediate data for use in subsequent steps; blank frame removal: according to different situations of splicing positions, blank frames and semantic frames of video frames are marked by using an inter-frame difference method, and unnecessary blank frames are removed; for each splicing, corresponding splicing frames 1, 2 and reference frames 1, 2 are selected according to different splicing positions; S4 for each splicing, a cubic Bezier curve is used to generate transition skeleton coordinates; S5 a transition RGB image frame is generated; S6 video synthesis: the entity video with blank frames removed, the generated transition frames and the template video with blank frames removed are combined into complete data.

Description

Technical Field

[0001] The invention belongs to the fields of computer vision and machine learning and is a method for generating transition frames of sign language videos based on skeleton migration. Background Art

[0002] Surveys show that by 2016, there were about 27.8 million hearing-impaired people in my country, accounting for 24% of the six types of disabilities, ranking first. Worldwide, nearly 5% of the population suffers from disabling hearing loss. It is difficult for hearing-impaired people to obtain information, communicate with others, and perceive the environment, which brings great obstacles to their lives, studies, and work. At present, the task of sign language recognition has attracted the attention of many researchers, but sign language recognition technology is far from meeting the actual application needs of hearing-impaired people. The main reason is the lack of sign language data sets and the small data scale. The largest Chinese continuous sign language data set currently has only 25,000 videos, covering 100 sentences. In view of the difficulty and high cost of collecting and annotating sign language data sets, the present invention proposes a sign language video combination method based on templates and entities. By dynamically combining different sign language videos, the effective sample size of sign language is expanded, thereby expanding the scale of sign language data sets.

[0003] The background technologies involved in the present invention are:

[0004] (1) Sign language dataset and sign language recognition algorithm

[0005] (a) Sign language dataset

[0006] Up to now, sign language datasets are still scarce and the data size is small. The largest Chinese continuous sign language recognition dataset, CSL, was collected under laboratory conditions, with a total of 100 sign sentences and a vocabulary of 178. (A total of 25,000 videos, more than 100 hours).

[0007] (b) Sign language recognition method

[0008] In the early days of sign language recognition, people still used more traditional methods, which were roughly divided into four stages: image preprocessing, hand segmentation, feature extraction, and classification recognition. After the HMM model was widely used in the field of speech recognition, HMM was also introduced into sign language recognition[1], achieving good results. On this basis, Dreuw[2] et al. proposed a complete set of sign language recognition methods. After the introduction of deep learning, neural networks were also applied to sign language recognition methods[3]. The model based on deep networks has greatly improved the accuracy of sign language recognition in terms of performance.

[0009] However, the construction of sign language datasets started relatively late compared to other deep learning fields. Unlike speech recognition, data collection and annotation for sign language is expensive and non-scalable, which poses a significant problem for recognition. Therefore, recent work on continuous sign language recognition has addressed this problem in a weakly supervised manner and adopted a network structure consisting of a feature extractor and an alignment model. The feature extractor extracts visual information from image sequences, and the alignment model searches for possible alignments between visual features and corresponding labels. Previous work used HMMs to update frame state labels and train feature extractors [4]. Graves et al. [5] provided connectionist temporal classification (CTC) to align predictions and labels by maximizing the sum of the probabilities of feasible alignments.

[0010] However, due to the problem of overfitting, the original end-to-end method leads to insufficient training of the feature extraction model. To solve this problem, Min et al. [6] proposed a visual alignment constraint, which makes the CSLR network end-to-end trainable by enforcing the use of the feature extractor for prediction and provides more alignment supervision, namely two new metrics to evaluate the different behaviors of the feature extractor and the alignment model.

[0011] (2) Human pose estimation

[0012] OpenPose[7] is one of the most popular methods for human pose estimation. Like many bottom-up methods, OpenPose first detects parts (keypoints) belonging to each person in the image and then assigns the parts to different individuals. The human pose skeleton represents the orientation of a person in a graphical format. In essence, it is a set of coordinates that are connected to each other to describe the posture of a person. Each coordinate in the skeleton is called a part, or joint or keypoint. A valid connection between two parts is called a pair of limbs. In sign language recognition, skeleton information is also one of the most important features in sign language recognition.

[0013] (3) Human joint skeleton information extraction

[0014] BlazePose[8], a lightweight convolutional neural network architecture for human pose estimation. The network generates 33 body keypoints for a person. Some studies have shown that the stacked hourglass architecture can greatly improve the prediction accuracy even with a small number of parameters. BlazePose extends this idea and uses an encoder-decoder network architecture to predict heatmaps for all joints.

[0015] MediaPipe Hands[9] proposes a real-time on-device hand tracking solution that can predict the human hand skeleton from a single RGB image. The solution includes:

[0016] A. Palm detector: It locates the palm by scanning the whole image and using a directional hand localization frame.

[0017] B. Hand coordinate estimation model: It returns high-fidelity 2.5D coordinates by operating on the hand positioning box cropped by the palm detector.

[0018] MediaPipe Hands uses the non-maximum suppression (NMS) algorithm to solve the problem of mutual occlusion or self-occlusion of hands, and uses a feature pyramid network (FPN) codec feature extractor to adapt to the detection of small objects such as fingers.

[0019] (4) RGB reconstruction based on skeleton information:

[0020] Pose Transfer

[10] proposed a new generative adversarial network to transfer the pose of a person. Given a person in a target pose, the network's generator consists of a series of pose-attention transfer modules, each of which transfers certain areas of its attention and gradually generates a person image. The person image generated by this method has better appearance consistency and shape consistency with the input image, so it looks more realistic. Two discriminators are designed, called appearance discriminator D A and shape discriminator D S To judge P g The people and P included c Are the people in P similar? g Is the posture consistent with S t same.

[0021] References

[0022] [1]Wen Gao,Gaolin Fang,Debin Zhao,and Yiqiang Chen.A chinese signlanguage recognition system based on sofm / srn / hmm.Pattern Recognition,37(12):2389–2402,2004.

[0023] [2]Philippe Dreuw,David Rybach,Thomas Deselaers,Morteza Zahedi,andHermann Ney.Speech recognition techniques for a sign language recognitionsystem.In Eighth Annual Conference of the International Speech CommunicationAssociation,2007.

[0024] [3]Runpeng Cui,Hu Liu,and Changshui Zhang.A deep neural framework forcontinuous sign language recognition by iterative training.IEEE Transactionson Multimedia,21(7):1880–1891,2019.

[0025] [4]Oscar Koller,Cihan Camgoz,Hermann Ney,and Richard Bowden.Weaklysupervised learning with multi-stream cnn lstm-hmms to discover sequentialparallelism in sign language videos.IEEE Transactions on Pattern Analysis andMachine Intelligence,2019.

[0026] [5]Alex Graves,Santiago Fernández,Faustino Gomez,and JürgenSchmidhuber.Connectionist temporal classification:labelling unsegmentedsequence data with recurrent neural networks.In Proceedings of the 23rdInternational Conference on Machine learning,pages 369–376,2006.

[0027] [6]Min Y,Hao A,Chai X,et al.Visual Alignment Constraint for Continuous Sign Language Recognition[J].2021.

[0028] [7]Z.Cao, G.Hidalgo Martinez, T.Simon, S.Wei, and YASheikh.Openpose1.1.0benchmark.

[0029] [8]Bazarevsky V,Grishchenko I,Raveendran K,et al.BlazePose:On-deviceReal-time Body Pose tracking[J].2020.

[0030] [9]Zhang F, Bazarevsky V, Vakunov A, et al. MediaPipe Hands: On-deviceReal-time Hand Tracking[J]. 2020.

[0031]

[10] J Zhang, Liu X, Li K. Human Pose Transfer by Adaptive HierarchicalDeformation[J]. Computer Graphics Forum, 2020.

[0032]

[11] Roy, Prasun et al. "Multi-scale Attention Guided Pose Transfer." ArXiv abs / 2202.06777(2022):n.pag. Summary of the invention

[0033] The purpose of the present invention is to provide a method for generating transition frames of sign language videos based on skeleton migration, in view of the fact that the construction of current sign language datasets is relatively cumbersome, the data volume required is huge, and the shooting cost is high. The present invention adopts a new dataset construction idea to realize the construction of a large-scale sign language dataset with controllable cost. To this end, the present invention adopts the following technical solutions:

[0034] A method for generating transition frames of sign language video based on skeleton migration comprises the following steps:

[0035] S1 Skeleton information extraction: Continuous frame skeleton extraction is performed on each video frame, and the skeleton coordinates are stored as intermediate data for use in subsequent steps;

[0036] S2 removes blank frames: According to different situations of splicing positions, the blank frames and semantic frames of the video frames are marked by using the inter-frame difference method to remove unnecessary blank frames;

[0037] S3 selects corresponding splicing frames 1 and 2 and reference frames 1 and 2 for each splicing according to different splicing positions, and the method is as follows:

[0038] ① The splicing position is the beginning: select the last frame of the entity video semantic frame and the first frame of the template video semantic frame as splicing frames 1 and 2 respectively; select the first frame of the blank frame at the end of the entity video as reference frame 1, and the last frame of the blank frame at the beginning of the template video as reference frame 2;

[0039] ② The splicing position is in the middle, involving two splicing: the first splicing selects the previous frame of the template video splicing position and the first frame of the entity video semantic frame as splicing frames 1, 2, and the next frame of the template video splicing position and the last frame of the blank frame at the beginning of the entity video as reference frames 1, 2; the second splicing selects the last frame of the entity video semantic frame and the next frame of the template video splicing position as splicing frames 1, 2, and the first frame of the blank frame at the end of the entity video and the previous frame of the template video splicing position as reference points 1, 2;

[0040] ③ The splicing position is the end: select the last frame of the semantic frame of the template video and the first frame of the semantic frame at the beginning of the entity video as the splicing frames 1 and 2; select the first frame of the blank frame at the end of the template video and the last frame of the blank frame at the beginning of the entity video as the reference frames 1 and 2;

[0041] S4 generates transition skeleton coordinates using cubic Bezier curves for each splicing;

[0042] S5 generates transitional RGB image frames

[0043] S51 establishes a human posture conversion network and pre-trains it;

[0044] S52, for each generated transition skeleton coordinate, the corresponding spliced ​​frame is used as the guiding frame, the skeleton coordinates of the guiding frame are the guiding skeleton coordinates, the guiding frame, the guiding skeleton coordinates and the transition skeleton coordinates are input into the pre-trained human posture conversion network, and the transition RGB image frame corresponding to each transition skeleton coordinate is calculated;

[0045] S6 video synthesis: Combine the entity video with blank frames removed, the generated transition frames, and the template video with blank frames removed into complete data.

[0046] Preferably, in S1, MediaPipe Pose is used to perform continuous frame skeleton extraction on each video frame.

[0047] Preferably, the method of S2 is as follows:

[0048] S21 calculates the inter-frame difference: the nth frame and the n-1th frame image in the video sequence are denoted as f n and f n-1 , the grayscale value of the corresponding pixel in the two frames is recorded as f n (x,y) and f n-1 (x, y), subtract the grayscale values ​​of the corresponding pixels of the two frames of images, and take their absolute values ​​to obtain the difference matrix D n ; The difference matrix D n Take the average value as a description of the difference between the nth frame and the previous frame;

[0049] S22 saves the average difference between all frames and the previous frame as a scatter curve;

[0050] S23 sets a smoothing window, finds all the maximum points after smoothing; sets a threshold; detects all the maximum points whose average difference values ​​are greater than the threshold, and regards the part between the first maximum point and the last maximum point detected as a semantic frame, and the rest as a blank frame;

[0051] S24 has different splicing positions for different templates, and removes blank frames in the following three situations:

[0052] ① The splicing position is the beginning: remove the blank frames at the end of the entity video and the blank frames at the beginning of the template video;

[0053] ② The splicing position is in the middle: remove the blank frames at the beginning and end of the entity video;

[0054] ③ The splicing position is the end: remove the blank frames at the beginning of the entity video and the end of the template video.

[0055] Preferably, in S23, the smoothing window is set to 20; and the threshold is set to 1.5.

[0056] Preferably, the method of S4 is:

[0057] The skeleton coordinates of the spliced ​​frame 1 are P1, and the skeleton coordinates of the reference frame 1 are P2; the skeleton coordinates of the reference frame 2 are P3, and the skeleton coordinates of the spliced ​​frame 2 are P4;

[0058] For the splicing frame coordinates P1, P4 and the reference frame coordinates P2, P3, the parametric equation of the transition curve is:

[0059] B(t)=P1(1-t) 3 +3P2t(1-t) 2 +3P3t2 (1-t)+P4 3 t 3 ,t∈[0,1]

[0060] Where B(t) is the coordinate of a point on the curve, t is a parameter,

[0061] Nine points on the transition curve are selected equidistantly to generate the intermediate transition skeleton, that is, let t be 0.1, 0.2, ..., 0.9 respectively, and 9 B(t) values ​​are obtained, which are the transition skeleton coordinates.

[0062] Preferably, the method of S52 is:

[0063] ① For the first five frames in the transition RGB image frame, the splicing frame 1 is selected as the guide frame, and the splicing frame 2 is selected as the guide frame for the last four frames in the transition RGB image frame;

[0064] ② The input guide frame and transition skeleton coordinates are first encoded into corresponding features through convolution operations, and the attention mask M is calculated using the correlation between the guide skeleton features and the target skeleton features. t ,participate in the feature conversion from the guidance frame features to the generated transition RGB image frame;

[0065] ③Decode the converted features and generate a transitional RGB image frame.

[0066] The beneficial effects of the technical solution provided by the present invention are:

[0067] (1) High practicability. The method provided by the present invention can effectively reduce the cost of sign language data collection. In the shooting work of constructing a data set, it is only necessary to shoot N1+N2 data to obtain N1×N2 data. In addition, the skeleton information generated in the middle can be directly used as the feature preprocessing result of subsequent sign language recognition or action detection, thereby saving computing power and time costs.

[0068] (2) High accuracy. The present invention generates intermediate transition frames based on skeletons and RGB images, which are closer to real action transitions, more coherent and smoother in visual perception, and the generated data effect is less different from real-life shooting, and performs well in sign language recognition models. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 :Flowchart of splicing method

[0070] Figure 2 :Skeleton information selection

[0071] Figure 3 : Smoothed mean difference scatter curve

[0072] Figure 4 : Intermediate skeleton generation result

[0073] Figure 5 : Intermediate transition frame generation process

[0074] Figure 6 : Intermediate transition frame generation result. In the figure, note: the black border is the original video frame and its skeleton, and the gray border is the generated transition skeleton and its RGB frame DETAILED DESCRIPTION

[0075] The present invention belongs to the field of computer vision and machine learning, and provides a new idea for constructing a sign language dataset to achieve the construction of a large-scale sign language dataset with controllable cost. Figure 1 As shown:

[0076] The invention comprises two parts: a new skeleton-based transition frame generation and a skeleton migration-based video frame generation.

[0077] 1. Skeleton-based transition frame generation: For the input template video and entity video, the skeleton information of the intermediate transition frame is generated through the Bezier curve. The specific process is divided into the following steps:

[0078] (1) Skeleton information extraction: Use MediaPipe Pose to extract skeletons from consecutive frames and store the skeleton information as intermediate data for use in subsequent steps. A total of 57 skeleton information points are selected, including 21 for each of the finger joints and 15 for the body joints. Figure 2 shown.

[0079] (2) Removing blank frames: Since the video sequence captured by the camera is continuous, if there is no moving target in the scene, the change between consecutive frames is very slight. If there is a moving target, there will be obvious changes between consecutive frames. Therefore, according to the different situations of the splicing position (here the splicing position refers to the specific position where the entity video is inserted into the template video), the inter-frame difference method is used to mark the blank frames and semantic frames of the video frames, and unnecessary blank frames are removed to make the transition action smoother and more coherent.

[0080] The frame difference calculation method is:

[0081] D(x,y)=|f n (x,y)-f n-1 (x,y)| (1)

[0082] Among them, the nth frame and the n-1th frame image in the video sequence are denoted as f n and f n-1 , the grayscale value of the corresponding pixel in the two frames is recorded as f n (x,y) and f n-1(x, y), subtract the grayscale values ​​of the corresponding pixels of the two frames of images, and take their absolute values ​​to obtain the difference matrix D n The difference matrix is ​​averaged as a description of the difference between the frame and the previous frame, and the average difference between all frames and the previous frame is saved as a scatter curve, such as Figure 3 As shown. Set the smoothing window size to 20, find all the maximum points after smoothing, and these points are regarded as the significant movement of the characters in the video. Set the threshold to 1.5 to detect all the maximum points whose average difference value is greater than the threshold, and regard the part between the first maximum point and the last maximum point detected as the semantic frame, and the rest as blank frames. For different templates, the splicing positions are different, which can be divided into the following three cases:

[0083] ① The splicing position is the beginning: remove the blank frames at the end of the entity video and the beginning of the template video.

[0084] ② The splicing position is a certain frame in the middle: remove the blank frames at the beginning and end of the entity video.

[0085] ③ The splicing position is the end: remove the blank frames at the beginning of the entity video and the end of the template video.

[0086] (3) According to different stitching positions, stitching frames 1 and 2 and reference frames 1 and 2 are selected as follows:

[0087] ① The splicing position is the beginning: select the last frame of the entity video semantic frame and the first frame of the template video semantic frame as splicing frames 1 and 2 respectively. Select the first frame of the blank frame at the end of the entity video as reference frame 1, and the last frame of the blank frame at the beginning of the template video as reference frame 2.

[0088] ② The splicing position is in the middle: This situation involves two splicings: the first time, the previous frame of the template video splicing position and the first frame of the entity video semantic frame are selected as splicing frames 1, 2, and the next frame of the template video splicing position and the last frame of the blank frame at the beginning of the entity video are selected as reference frames 1, 2; the second time, the last frame of the entity video semantic frame and the next frame of the template video splicing position are selected as splicing frames 1, 2, and the first frame of the blank frame at the end of the entity video and the previous frame of the template video splicing position are selected as reference frames 1, 2.

[0089] ③ The splicing position is the end: the last frame of the semantic frame of the template video and the first frame of the semantic frame at the beginning of the entity video are selected as splicing frames 1 and 2. The first frame of the blank frame at the end of the template video and the last frame of the blank frame at the beginning of the entity video are selected as reference frames 1 and 2.

[0090] (4) Generate transition skeleton: Use cubic Bezier curve to generate the coordinates of the intermediate transition skeleton.

[0091] The skeleton coordinates of the spliced ​​frame 1 are denoted as P1, and the skeleton coordinates of the reference frame 1 are denoted as P2; the skeleton coordinates of the reference frame 2 are denoted as P3, and the skeleton coordinates of the spliced ​​frame 2 are denoted as P4.

[0092] For the splicing frame coordinates P1, P4 and the reference frame coordinates P2, P3, the parametric equation of the transition curve is:

[0093] B(t)=P1(1-t) 3 +3P2t(1-t) 2 +3P3t 2 (1-t)+P4 3 t 3 ,t∈[0,1] (2)

[0094] Where B(t) is the coordinate of a point on the curve and t is the parameter.

[0095] Each time the transition is performed, 9 points on the curve are selected at equal intervals to generate the intermediate transition skeleton. Let t be 0.1, 0.2, ..., 0.9, and 9 B(t) values ​​are obtained, which are the coordinates of the transition skeleton. Figure 4 shown.

[0096] (5) Generate transition frame

[0097] For each generated transition skeleton coordinate, the corresponding spliced ​​frame is used as the guidance frame, and the skeleton coordinates of the guidance frame are the guidance skeleton coordinates. The guidance frame, the guidance skeleton coordinates and the transition skeleton coordinates are input into the pre-trained human posture conversion network to calculate the transition RGB image frame corresponding to each transition skeleton coordinate. The process is as follows: Figure 5 shown.

[0098] ① For the first five frames in the transition frame, splicing frame 1 is selected as the guide frame, and for the last four frames in the transition frame, splicing frame 2 is selected as the guide frame. The guide skeleton coordinates contain 57 skeleton information points, including 21 for each of the finger joints and 15 for the body joints.

[0099] ② The input guidance image and skeleton coordinates are first encoded into corresponding features through convolution operations, and the attention mask M is calculated using the correlation between the guidance skeleton features and the target skeleton features. t , involved in the transformation from guiding image features to generated image features.

[0100] ③ The human posture conversion network used in this embodiment is the PATNs network mentioned in reference 10. This embodiment uses the PATNs network to complete the progressive human posture conversion. The training data set of the network is a plurality of randomly selected image pairs after the entity and template data sets are mixed. In each pair of images, one image is used as a guide frame and provides guide skeleton coordinates, and the other image provides target skeleton coordinates and is used to judge the quality of the generated image to guide the optimization of network parameters.

[0101] ④The features of the generated image will be decoded into an RGB image.

[0102] Those skilled in the art should understand that although this example uses the PATNs network as a posture converter, the method can actually use any convolutional neural network to complete this process as long as it has posture consistency and appearance consistency above a specified threshold. For example, the posture conversion network mentioned in reference 11 can also be used to complete this process.

[0103] (6) Video Synthesis

[0104] The entity video with blank frames removed, the generated transition frames, and the template video with blank frames removed are combined into complete data.

Claims

1. A method for generating transition frames of a sign language video based on skeleton migration, comprising the following steps: S1 Skeleton information extraction: Continuous frame skeleton extraction is performed on each video frame, and the skeleton coordinates are stored as intermediate data for use in subsequent steps; S2 removes blank frames: According to different situations of splicing positions, the blank frames and semantic frames of the video frames are marked by using the inter-frame difference method to remove unnecessary blank frames; S3 selects corresponding splicing frames 1 and 2 and reference frames 1 and 2 for each splicing according to different splicing positions, and the method is as follows: ① The splicing position is the beginning: select the last frame of the entity video semantic frame and the first frame of the template video semantic frame as splicing frames 1 and 2 respectively; select the first frame of the blank frame at the end of the entity video as reference frame 1, and the last frame of the blank frame at the beginning of the template video as reference frame 2; ② The splicing position is in the middle, involving two splicing: the first splicing selects the previous frame of the template video splicing position and the first frame of the entity video semantic frame as splicing frames 1, 2, and the next frame of the template video splicing position and the last frame of the blank frame at the beginning of the entity video as reference frames 1, 2; the second splicing selects the last frame of the entity video semantic frame and the next frame of the template video splicing position as splicing frames 1, 2, and the first frame of the blank frame at the end of the entity video and the previous frame of the template video splicing position as reference points 1, 2; ③ The splicing position is the end: select the last frame of the semantic frame of the template video and the first frame of the semantic frame at the beginning of the entity video as the splicing frames 1 and 2; select the first frame of the blank frame at the end of the template video and the last frame of the blank frame at the beginning of the entity video as the reference frames 1 and 2; For each splicing, S4 uses a cubic Bezier curve to generate transition skeleton coordinates as follows: The skeleton coordinates of the spliced ​​frame 1 are P1, and the skeleton coordinates of the reference frame 1 are P2; the skeleton coordinates of the reference frame 2 are P3, and the skeleton coordinates of the spliced ​​frame 2 are P4; For the splicing frame coordinates P1, P4 and the reference frame coordinates P2, P3, the parametric equation of the transition curve is: B(t)=P1(1-t) 3 +3P2t(1-t) 2 +3P3t 2 (1-t)+P4 3 t 3 ,t∈[0,1] Where B(t) is the coordinate of a point on the curve, t is a parameter, Select 9 points on the transition curve at equal intervals to generate the intermediate transition skeleton, that is, let t be 0.1, 0.2, ..., 0.9 respectively, and get 9 B(t) values, which are the transition skeleton coordinates; S5 generates a transitional RGB image frame by: S51 establishes a human posture conversion network and pre-trains it; S52 For each generated transition skeleton coordinate, the corresponding spliced ​​frame is used as the guiding frame, and the skeleton coordinates of the guiding frame are the guiding skeleton coordinates. The guiding frame, the guiding skeleton coordinates and the transition skeleton coordinates are input into the pre-trained human posture conversion network to calculate the transition RGB image frame corresponding to each transition skeleton coordinate.

2. The method according to claim 1, characterized in that In S1, MediaPipePose is used to extract continuous frame skeletons from each video frame.

3. The method according to claim 1, characterized in that The method of S2 is as follows: S21 calculates the inter-frame difference: the nth frame and the n-1th frame image in the video sequence are denoted as f n and f n-1 , the grayscale value of the corresponding pixel in the two frames is recorded as f n (x,y) and f n-1 (x, y), subtract the grayscale values ​​of the corresponding pixels of the two frames of images, and take their absolute values ​​to obtain the difference matrix D n ; The difference matrix D n Take the average value as a description of the difference between the nth frame and the previous frame; S22 saves the average difference between all frames and the previous frame as a scatter curve; S23 sets a smoothing window, finds all the maximum points after smoothing; sets a threshold; detects all the maximum points whose average difference values ​​are greater than the threshold, and regards the part between the first maximum point and the last maximum point detected as a semantic frame, and the rest as a blank frame; S24 has different splicing positions for different templates, and removes blank frames in the following three situations: ① The splicing position is the beginning: remove the blank frames at the end of the entity video and the blank frames at the beginning of the template video; ② The splicing position is in the middle: remove the blank frames at the beginning and end of the entity video; ③ The splicing position is the end: remove the blank frames at the beginning of the entity video and the end of the template video.

4. The method according to claim 3, characterized in that In S23, the smoothing window is set to 20 and the threshold is set to 1.

5.

5. The method according to claim 1, characterized in that The method of S52 is: ① For the first five frames in the transition RGB image frame, the splicing frame 1 is selected as the guide frame, and the splicing frame 2 is selected as the guide frame for the last four frames in the transition RGB image frame; ② The input guide frame and transition skeleton coordinates are first encoded into corresponding features through convolution operations, and the attention mask M is calculated using the correlation between the guide skeleton features and the target skeleton features. t ,participate in the feature conversion from the guidance frame features to the generated transition RGB image frame; ③Decode the converted features and generate a transitional RGB image frame.

Images