Human motion similarity measurement method

Abstract

The present invention discloses a human motion similarity measurement method, comprising the following steps: step 1, extracting key action frames and action intervals from a video to be tested to obtain a data set; step 2, using skeletal coordinate information to regularize the data set to obtain a skeletal graph data set; step 3, building a twin neural network model; step 4, setting a template to be tested T and a reference template R, using the trained twin neural network model to perform similarity measurement on single-frame images in the test template T and the reference template R, and obtaining the dissimilarity of the single-frame images; step 5, using the DTW algorithm to calculate the dissimilarity of the single-frame images obtained in step 3, and obtaining the final similarity measurement result. The present invention solves the problem in the prior art that the similarity calculated based on features such as the distance and angle between bones is too linear and lacks the concept of "action", resulting in low accuracy of similarity measurement.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and machine learning, and relates to a method for measuring similarity of human motions. Background Art

[0002] Human motion similarity measurement is based on the two-dimensional image coordinate sequence of each joint of the human body obtained by existing motion capture technology, and obtains the similarity between the skeleton model and the motion sequence in a single frame sequence through pattern recognition methods. Human motion similarity measurement aims to form a unified motion evaluation standard through the relatively impressive learning ability and powerful computing power of today's machines, provide a reference for expert evaluation, and reduce the workload of expert evaluation. Related content involves athlete motion evaluation in sports, patient mobility evaluation in the medical field, etc.

[0003] At present, human motion evaluation mainly relies on expert experience, that is, the quality of human motion (such as correctness, fluency, etc.) is evaluated and measured by the experience and professional knowledge of experts. With the development of computer technology and data analysis technology, there are more and more automated and intelligent analysis methods based on pattern recognition, which makes it possible to assist expert evaluation and reduce the burden on experts. The template matching method extracts feature templates from the reference action sequence and the test action sequence to directly match them, but it is sensitive to noise and motion intervals; the probability-based method defines the state representation of a single-frame action, and then defines the transition probability of the state between actions to evaluate the similarity between the test action and the reference action. The emergence of deep learning methods has greatly improved the feature extraction and guarantee performance of data. Among them, convolutional neural network is a widely used model that can extract accurate and effective image representations and directly output judgment results in an end-to-end form, which can reduce errors before data input. Convolutional neural network performs well in single-frame action recognition, but for sequence behavior recognition, trajectory information needs to be considered. Summary of the invention

[0004] The purpose of the present invention is to provide a method for measuring the similarity of human motions, which uses dynamic time warping (DTW) and deep learning methods to achieve similarity measurement of human target motions, thereby solving the problem in the prior art that the similarity calculated based on features such as the distance and angle between bones is too linear and lacks the concept of "motion", resulting in low accuracy of similarity measurement.

[0005] The technical solution adopted by the present invention is:

[0006] The human motion similarity measurement method includes the following steps:

[0007] Step 1: extract key action frames and action intervals from the test video to obtain a data set;

[0008] Step 2, using the bone coordinate information to regularize the data set to obtain a bone graph data set;

[0009] Step 3: Build a twin neural network model and use the twin neural network model to train the skeleton image dataset;

[0010] Step 4: Set the template to be tested T and the reference template R, and use the trained twin neural network model to measure the similarity of the single-frame images in the test template T and the reference template R to obtain the dissimilarity of the single-frame images;

[0011] Step 5: Use the DTW algorithm to calculate the dissimilarity of the single-frame image obtained in step 3 to obtain the final similarity measurement result.

[0012] The present invention is also characterized in that:

[0013] The specific method for extracting key action frames in step 1 is:

[0014] Step 1.1.1, disassemble the test video frame by frame and convert it into grayscale image;

[0015] Step 1.1.2 performs inter-frame differential operations on the grayscale image frame by frame. Assume that there are K frames in the video, the previous frame is curr, and the previous frame is pre, and the differential matrix diff is obtained. The operation here is:

[0016] diff=cur-pre (1)

[0017] Sum=sum(diff) (2)

[0018]

[0019] Where diff is the matrix after matrix subtraction between frames, Sum is the cumulative sum of the differences between pixels between frames, that is, the difference intensity between frames; Mean is the average of Sum according to the number of pixels, that is, the average difference intensity between frames;

[0020] Step 1.1.3, all frames of the video are differentially accumulated and averaged and then saved in the array Diff;

[0021] Step 1.1.4, perform smoothing operation on Diff, and expand Diff by (window_len-1) length to ensure that the values ​​that need to be convolved can be fully convolved. At this time, we have:

[0022] Diff=[f 0 ,f 1 ,........f K+2*window_len-3 ] (5)

[0023] Diff is the average inter-frame differential intensity without smoothing;

[0024] Step 1.1.5, from f 0 Start by aligning the midpoint of the convolution array and performing convolution operations on the values ​​in Diff in turn:

[0025]

[0026] Diff′ is the average inter-frame differential intensity after smoothing;

[0027] Step 1.1.6: After the operation of step 1.1.5 is completed, the previously expanded part is removed, and the average inter-frame differential intensity for smoothing is obtained. Finally, the local maximum value is selected according to the average inter-frame differential intensity to obtain the key frame actions and their subscripts with relatively uniform distribution.

[0028] The method for extracting the action interval is:

[0029] Step 1.2.1, set an upper threshold and a lower threshold to determine whether the current frame is in motion, that is, the interval of the test action. The upper threshold is used to determine the start of the action, and the lower threshold is used to determine the end of the action. Using two thresholds can achieve more accurate capture. The condition for judging the start of the action is the appearance of an average inter-frame differential intensity greater than the upper threshold. The end position is searched in the subsequent interval. The end of the search is marked by the continuous appearance of at least x average inter-frame differential intensities below the lower threshold.

[0030] Step 1.2.2, detect the length of the interval searched in step 1.2.1, filter out the interval outside the threshold, find the key frame position with the maximum value below the start position and the minimum value above the end position on the key frame, and then expand the left and right equal lengths to complete the extraction of the test action interval.

[0031] Step 2 is as follows:

[0032] Step 2.1, define joint[0]~joint

[14] as the joint positions of the skull, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, coccyx, right hip, right knee, right ankle, left hip, left knee, and left ankle, respectively. The two-dimensional joint coordinate information of a person is obtained from a human action video using motion capture technology, which is based on the human joint coordinate information extracted frame by frame from the video.

[0033] Step 2.2, uniformly scale the spine lengths of jiont[1] and jiont[8] to length t. Assume that the coordinates of jiont[1] are (jiont[1][0], jiont[1][1]), and the coordinates of jiont[8] are (jiont[8][0], jiont[8][1]) (the second dimension coordinate 0 represents the horizontal coordinate x, and 1 represents the vertical coordinate y), then the spine scaling ratio r is:

[0034]

[0035] Then the coordinates representing the joint points are mapped from the QT coordinate system to the rectangular coordinate system, so that the center position of the human spine coincides with the origin of the coordinate system, and then each bone is uniformly scaled like the origin according to the scaling ratio r;

[0036] Step 2.3, use drawing tools to connect each joint point in an orderly manner to generate a skeleton diagram that can reflect human body movements.

[0037] In step 3, the twin neural network model is a twin neural network model including three convolutional layers and three linear layers connected in sequence; the Constractive Loss function is used as the loss function of the twin neural network model, and the formula is:

[0038]

[0039] Where P represents the dimension of the feature vector, Y is a label indicating whether two samples match (similar), where Y = 1 means that the two samples are similar or matched, and Y = 0 means they are not similar or matched; margin is a preset threshold, N represents the number of samples, and D W D W (X 1 ,X 2 ), representing two sample feature vectors X 1 and X 2 The Euclidean distance.

[0040] When the twin neural network model is trained in step 3, each image in the training set is a skeleton image processed in step 2; each sample in the training set is a labeled image consisting of a test image and a standard image in pairs, and each group of images corresponds to a label. Two skeleton images of the same type are positive samples, and the label is 1, and two skeleton images of different types are negative samples, and the label is 0.

[0041] In step 4, the reference template R and the test template T are defined as:

[0042] R={R(1),R(2),.....R(m),.....R(M)} (10)

[0043] T={T(1),T(2),......T(n),......T(N)} (11)

[0044] Where m is the time sequence number of the training action frame, m=1 is the starting action frame, m=M is ​​the ending action frame, M is the number of action frames contained in the template, and the reference template R is a set of standard action frames extracted through key frame extraction and action interval extraction;

[0045] By using the trained twin neural network model to operate on the reference template R and the test template T, the dissimilarity of a single frame image can be obtained, denoted as d[n,m], which is the dissimilarity between the nth frame action of the test template T and the mth action of the reference template R.

[0046] Step 5 is as follows:

[0047] The test T and reference R are expanded two-dimensionally along the horizontal and vertical axes respectively, and the plane is divided into N*M grids. Two values ​​are recorded at the grid (n, m), one is the distance d[T(n), R(m)] between the test and reference frames, that is, the weight, and the other is the minimum cumulative distance D[n, m] from the starting point to the grid (n, m), that is, the cumulative weight. The calculation formula is:

[0048] D[n,m]=d[T(n),R(m)]+min{D(n-1,m),D(n-1,m-1),D(n,m-1)} (14)

[0049] The path that satisfies formula (14) is recorded as the path with the minimum cumulative weight, and D[n,m] is the dissimilarity of the action sequence to be tested. Thus, the measurement of human action similarity is completed.

[0050] The beneficial effects of the present invention are:

[0051] The present invention is an estimation of the standardization of human target actions based on the similarity measurement between standard and test actions, in which the calculation of similarity is carried out by relying on the DTW algorithm and deep learning, and is used on continuous action sequences by imitating the calculation of similarity of continuous sound sequences. Compared with the traditional method of only considering the relationship between bone coordinates and distances and angles between skeletons, deep learning makes the distance calculation more sensitive to "action" through the learning and training of samples (the traditional method is too linear and the recognition effect of similar but different actions will be relatively poor), which is closer to the standard of human action recognition, breaks through the limitation of linear calculation, improves the accuracy of similarity, and reduces the accumulation of errors when comparing continuous action sequences. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 It is a definition diagram of human skeleton joints in the present invention;

[0054] Figure 3 It is a structural diagram of the twin neural network model in the present invention;

[0055] Figure 4 is the result of the single frame test of the 2885th frame of the standard action and the 2378th frame of the test action in implementation 1 of the present invention;

[0056] Figure 5 is the result of the single frame test of the 2911th frame of the standard action and the 2396th frame of the test action in implementation 1 of the present invention;

[0057] Figure 6 It is the result of the single frame test of the 2924th frame of the standard action and the 2404th frame of the test action in implementation 1 of the present invention. DETAILED DESCRIPTION

[0058] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] The human body action similarity measurement method of the present invention is as follows: Figure 1 , including the following steps:

[0060] Step 1: extract key action frames and action intervals from the test video to obtain a data set;

[0061] Step 2, using the bone coordinate information to regularize the data set to obtain a bone graph data set;

[0062] Step 3: Build a twin neural network model and use the twin neural network model to train the skeleton image dataset;

[0063] Step 4: Set the template to be tested T and the reference template R, and use the trained twin neural network model to measure the similarity of the single-frame images in the test template T and the reference template R to obtain the dissimilarity of the single-frame images;

[0064] Step 5: Use the DTW algorithm to calculate the dissimilarity of the single-frame image obtained in step 3 to obtain the final similarity measurement result.

[0065] The specific method for extracting key action frames in step 1 is:

[0066] Step 1.1.1, disassemble the test video frame by frame, with a frame rate of 25 frames per second, and then perform simple preprocessing (convert the color image to grayscale);

[0067] Step 1.1.2 performs frame-by-frame differential operations. Assume that there are K (0 to K-1) frames in the video, the current frame is curr, and the previous frame is pre. The differential matrix diff is obtained. Here is the matrix operation:

[0068] diff=cur-pre (1)

[0069] Sum=sum(diff) (2)

[0070]

[0071] Where diff is the matrix after matrix subtraction between frames, Sum is the cumulative sum of the differences between pixels between frames, that is, the difference intensity between frames; Mean is the average of Sum according to the number of pixels, that is, the average difference intensity between frames.

[0072] Step 1.1.3: All video frames (0 to K-1) are differentially accumulated and averaged and then stored in the array Diff.

[0073] Step 1.1.4, perform smoothing operation on Diff. Regarding the smoothing operation, a one-dimensional array with a length of window_len is used to perform a moving convolution operation on Diff. The value of the convolution array is a mapping of discrete integer points of a Gaussian function or a cosine window in the interval [0, window_len), which is used to highlight or filter a certain object. Indow_len is usually set to an odd number:

[0074] W=[w 0 ,w 1 ,....w window-len-1 / 2 .....w window_len-1 ] / sum(W) (4)

[0075] Where W is the convolution array, W i is the function value of the Gaussian function at x=i;

[0076] Diff is expanded before and after by (window_len-1) length to ensure that the values ​​that need to be convolved can be fully convolved, then we have:

[0077] Diff=[f 0 ,f 1 ,........f K+2*window_len-3 ] (5)

[0078] Diff is the average inter-frame differential intensity without smoothing;

[0079] Step 1.1.5, from f 0 Start by aligning the midpoint of the convolution array and performing convolution operations on the values ​​in Diff in turn:

[0080]

[0081] Diff′ is the average inter-frame differential intensity after smoothing;

[0082] Step 1.1.6: After the operation of step 1.1.5 is completed, the previously expanded part is removed, and the average inter-frame differential intensity for smoothing is obtained. Finally, the local maximum value is selected according to the average inter-frame differential intensity to obtain the key frame action and its subscript with relatively uniform distribution. The work of this part is to effectively reduce the number of frames to reduce the amount of calculation while retaining the integrity of the action, so the method of extracting action key frames is adopted. Assuming that the action is 20 frames, the result of action key frame extraction can be simply expressed as:

[0083] [1,3,5,7,9,11,13,15,17,19]

[0084] The method for extracting the action interval in step 1 is:

[0085] Step 1.2.1, set an upper threshold (up) and a lower threshold (down) to determine whether the current frame is in motion (i.e., the interval of the test action). The upper threshold is used to determine the start of the action, and the lower threshold is used to determine the end of the action. Using two thresholds can achieve more accurate capture. The condition for judging the start of the action is the appearance of an average inter-frame differential intensity greater than the upper threshold. The end position is searched in the interval thereafter (there will be no new start before the end of an action search). The end of the search is marked by the appearance of at least x consecutive average inter-frame differential intensities below the lower threshold.

[0086] Step 1.2.2, detect the length of the intervals searched in step 1.2.1, filter out the intervals that are too long or too short, find the key frame position with the maximum value lower than the start position and the minimum value higher than the end position on the key frame, and then expand the left and right equal lengths to complete the extraction of the test action interval. The reason for extracting the action interval is mainly to deal with the fact that there are multiple groups of actions in a video in actual situations and there are obvious pauses between the actions. With the help of interval extraction, these actions can be separated. This part will be explained with the key frame extraction results of step 1.1.6 as an example: Assuming that there are two actions in the 20 frames, the two actions obtained after the action interval extraction are located at 4 to 8 frames and 15 to 16 frames respectively. In order to improve the accuracy of the interval boundary, the action is expanded forward and backward, and the expanded ones are 3 to 9 frames and 13 to 18 frames. Finally, combined with the results of key frame extraction, action 1 is [3,5,7,9], and action 2 is [13,15,17,19]. The final action is the action template entered in step 4.

[0087] Step 2 is to regularize the human skeleton involved in the measurement according to the length of the spine due to the differences in body shape between different test subjects. Specifically:

[0088] Step 2.1, define joint[0]~joint

[14] as the joint positions of the skull, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, coccyx, right hip, right knee, right ankle, left hip, left knee, and left ankle, respectively. Use motion capture technology to obtain the two-dimensional joint coordinate information of a person from a human action video. The present invention is based on the extraction of human joint coordinate information from the video frame by frame.

[0089] Step 2.2, uniformly scale the spine lengths of jiont[1] and jiont[8] to length t. Assume that the coordinates of jiont[1] are (jiont[1][0], jiont[1][1]), and the coordinates of jiont[8] are (jiont[8][0], jiont[8][1]) (the second dimension coordinate 0 represents the horizontal coordinate x, and 1 represents the vertical coordinate y), then the spine scaling ratio r is:

[0090]

[0091] Then the coordinates representing the joint points are mapped from the QT coordinate system to the rectangular coordinate system, so that the center position of the human spine coincides with the origin of the coordinate system, and then each bone is uniformly scaled like the origin according to the scaling ratio r;

[0092] Step 2.3, use drawing tools to connect each joint point in an orderly manner to generate a skeleton diagram that can reflect human body movements.

[0093] The structure of the twin neural network model in step 3 is:

[0094] In the present invention, the structure of the twin neural network is a twin neural network model including three convolutional layers and three linear layers connected in sequence. Before calculating the similarity of the skeleton graphs of the standard action frame and the test action frame, the shared weight network will be used to extract features from them respectively. Figure 3 .

[0095] When the twin neural network model is trained in step 3, each sample in the training set is a labeled image of a test image and a standard image in pairs, and each image in the training set is a skeleton image processed in step 2; by referring to and corresponding to the action video, the same actions completed by different testers are classified into one category, thereby integrating multiple action frame sets. For example, a simple gymnastics action can be divided into two 28 beats, and the actions on each beat are different. Each beat can correspond to 5 to 10 skeleton images. First, understand the standard action of the action, and then collect the corresponding beats of the test group and the "reasonable" actions made in the transition process between beats from the tester's actions, and classify the skeleton images of the same actions made by different testers or similar "reasonable" actions made by the same tester at adjacent times into one category, and so on to collect more actions to make a training set of skeleton images about human body actions.

[0096] Then, the samples are collected and labeled in a random manner. A frame is randomly selected from multiple frames, and then a positive sample or a negative sample is randomly determined (two skeleton images of the same type are positive samples, and the label is 1, and two skeleton images of different types are negative samples, and the label is 0). At the same time, the ratio of the number of positive and negative samples is kept near 1:1. For example, the first randomly selected image is the first skeleton image in the action 1-the first shot class, and then it is randomly selected as a positive sample or a negative sample. If it is a positive sample, another skeleton image is taken from the class and labeled 1. Otherwise, other types of actions except this class are randomly selected and labeled 0. The parameters for training the network input are a sample (two skeleton images) and a label.

[0097] When training the twin neural network model in step 3, the Constractive Loss function is used as the loss function of the model:

[0098] Constractive Loss is mainly used in dimensionality reduction, that is, the originally similar samples, after dimensionality reduction (feature extraction), in the feature space, the two samples are still similar; and the originally dissimilar samples, after dimensionality reduction, in the feature space, the two samples are still dissimilar. Similarly, this loss function can also well express the matching degree of paired samples.

[0099] The expression of contrastive loss is as follows:

[0100]

[0101] Where P represents the dimension of the feature vector, Y is a label indicating whether two samples match (similar), where Y = 1 means that the two samples are similar or matched, and Y = 0 means they are not similar or matched; margin is a preset threshold, N represents the number of samples, and D W D W (X 1 ,X 2 ), representing two sample feature vectors X 1 and X 2 The Euclidean distance is:

[0102]

[0103] By observing the expression of the above contrastive loss, we can find that this loss function can well express the matching degree of paired samples and can also be well used to train feature extraction models.

[0104] In step 4, the reference template R and the test template T are defined as:

[0105] R={R(1),R(2),.....R(m),.....R(M)} (10)

[0106] T={T(1),T(2),......T(n),......T(N)} (11)

[0107] Where m is the timing label of the training action frame, m=1 is the starting action frame, m=M is ​​the ending action frame, M is the number of action frames contained in the template, and the reference template R is a set of standard action frames (skeleton graphs) extracted through key frame extraction and action interval extraction.

[0108] The similarity measurement result of a single frame image can be obtained by using the trained twin neural network model to operate on the reference template R and the test template T:

[0109] output0,output1=net(img0,img1) (12)

[0110] dissimilarity=pair_distance(output0,output1) (13)

[0111] net(): neural network feature extraction operation, output0,1: feature vectors of dimension 5, representing img0,1 respectively, pair_distance(): pixel-level Euclidean distance between feature maps, dissimilarity: single-frame action dissimilarity, single-frame action dissimilarity is denoted as d[n,m], that is, the dissimilarity between the n-th frame action of the reference action template T and the m-th action of the test template R.

[0112] In step 5, the DTW algorithm is used to combine the single-frame results to measure the similarity of human movements. The DTW (Dynamic Time Warping) algorithm is a relatively prominent similarity (or distance) function of time series data. It is the simplest and most effective method in processing isolated word speech recognition. Considering that there are many similarities between isolated word speech and action concepts (both are continuous sequences composed of a certain static state), the single-frame matching results obtained by deep learning are considered to replace the traditional Euclidean distance in DTW to achieve the combination of the two. Finally, it is found that the two have a relatively good fit.

[0113] Step 4 is as follows: expand T and R two-dimensionally along the horizontal and vertical axes respectively, divide the plane into N*M grids, and record two values ​​at the grid (n, m). One is the distance d[T(n), R(m)] between the test and reference frames, i.e., the weight. The other is the minimum cumulative distance D[n, m] from the starting point to the grid (n, m), i.e., the cumulative weight. That is, the distance D[N, M] between them needs to be calculated, i.e., the dissimilarity. The smaller D[N, M] is, the more similar the templates T and R are, which can be expressed by the following formula:

[0114] D[n,m]=d[T(n),R(m)]+min{D(n-1,m),D(n-1,m-1),D(n,m-1)} (14)

[0115] The path that satisfies formula (12) is recorded as the path with the minimum cumulative weight, and D[n,m] is the dissimilarity of the action sequence to be tested. Thus, the measurement of human action similarity is completed.

[0116] The DTW algorithm uses the idea of ​​dynamic programming to find an irreversible path with the minimum cumulative weight from the starting grid (1,1) to the destination grid (N,M). The shortest path can be described in sequence as (n1,m1),...,(ni,mj),...,(nN,mM), where (n1,m1) = (1,1), (nN,mM) = (N,M). When the path has passed through the grid point (n,m), the next grid point (nextn,nextm) can only be one of the following three situations:

[0117] (nextn, nextm)=(n+1,m)or(n+1,m+1)or(n,m+1) (15)

[0118] The above constraints are represented by boundary. The problem of finding the best path can be reduced to finding the best path when the constraint boundary is satisfied, so that the accumulation of weights along the path, i.e. the distance value, is minimized.

[0119] The method for searching the path is as follows: starting from the point (n, m), several paths that meet the boundary can be developed. For the point (n, m), its previous grid (pren, prem) can only come from the grids (n-1, m-1), (n, m-1), and (n-1, m). Therefore, the cumulative weight (distance) at (n, m) is

[0120] Example 1

[0121] According to steps 1 to 5, a video sequence is measured for similarity, and the result is as follows: Figures 4 to 6 The left side shows the picture in the standard action sequence, and the right side shows the picture in the test action sequence. The figure shows the results of single-frame tests, which are the single-frame test results of the 2885th frame of the standard action and the 2378th frame of the test action, the 2911th frame of the standard action and the 2396th frame of the test action, and the 2924th frame of the standard action and the 2404th frame of the test action. The final dissimilarity result of this embodiment is 114.2, and the smaller the final value, the more similar the actions are.

[0122] The method of the present invention is used to measure similarity. Single-frame similarity realizes the comparison of static action similarity. DTW completes the frame-by-frame correspondence between standard action and test action. It combines the two more important indicators of action accuracy and time matching in action similarity measurement, combines the respective advantages of deep learning method and DTW algorithm, and has unique superiority in human action similarity measurement.

Claims

1. A human motion similarity measurement method, characterized in that: The following steps are involved: Step 1: extract key action frames and action intervals from the test video to obtain a data set; Step 2, using the bone coordinate information to regularize the data set to obtain a bone graph data set; Step 3: Build a twin neural network model and use the twin neural network model to train the skeleton image dataset; The twin neural network model is a twin neural network model including three convolutional layers and three linear layers connected in sequence. The Constractive Loss function is used as the loss function of the twin neural network model, and the formula is: （8） in represents the dimension of the feature vector, is a label indicating whether two samples match, where Y=1 means the two samples are similar or match, and Y=0 means they are not similar or match; margin is a preset threshold, N is the number of samples, for The abbreviation of , representing two sample feature vectors and The Euclidean distance of Step 4: Set the template to be tested T and the reference template R, and use the trained twin neural network model to measure the similarity of the single-frame images in the test template T and the reference template R to obtain the dissimilarity of the single-frame images; The reference template R and the test template T are defined as: （10） （11） Where m is the time sequence number of the training action frame, m=1 is the starting action frame, m=M is ​​the ending action frame, M is the number of action frames contained in the template, and the reference template R is a set of standard action frames extracted through key frame extraction and action interval extraction; The trained twin neural network model is used to operate the reference template R and the test template T to obtain the dissimilarity of a single frame image, which is recorded as , that is, the dissimilarity between the n-th frame action of the test template T and the m-th action of the reference template R; Step 5: Use the DTW algorithm to calculate the dissimilarity of the single-frame image obtained in step 3 to obtain the final similarity measurement result, which is specifically: The test T and reference R are expanded two-dimensionally along the horizontal and vertical axes respectively, and the plane is divided into N*M grids. Two values ​​are recorded at the grid (n, m), one is the distance between the test and reference frames. , that is, the weight, and search for another minimum cumulative distance from the starting point to the grid (n, m) , that is, the cumulative weight, the calculation formula is: D[n, m]=d[T(n),R(m)]+min{D(n-1,m),D(n-1,m-1),D(n,m-1)} （14） The path that satisfies formula (14) is recorded as the path with the minimum cumulative weight, and D[n,m] is the dissimilarity of the action sequence to be tested. Thus, the measurement of human action similarity is completed.

2. The human motion similarity measurement method according to claim 1, characterized in that: The specific method for extracting the key action frame in step 1 is: Step 1.1.1, disassemble the test video frame by frame and convert it into grayscale image; Step 1.1.2 performs inter-frame differential operations on the grayscale image frame by frame. Assume that there are K frames in the video, the previous frame is curr, and the previous frame is pre, and the differential matrix diff is obtained. The operation here is: （1） （2） （3） Where diff is the matrix after matrix subtraction between frames, Sum is the cumulative sum of the differences between pixels between frames, that is, the difference intensity between frames; Mean is the average of Sum according to the number of pixels, that is, the average difference intensity between frames; Step 1.1.3, all frames of the video are differentially accumulated and averaged and then saved in the array Diff; Step 1.1.4, perform smoothing operation on Diff, and expand Diff by (window_len-1) length to ensure that the values ​​that need to be convolved can be fully convolved. At this time, we have: （5） Diff is the average inter-frame difference intensity without smoothing; Step 1.1.5, from Start by aligning the midpoint of the convolution array and performing convolution operations on the values ​​in Diff in turn: （6） Diff′ is the average inter-frame difference intensity after smoothing; Step 1.1.6: After the operation of step 1.1.5 is completed, the previously expanded part is removed, and the average inter-frame differential intensity for smoothing is obtained. Finally, the local maximum value is selected according to the average inter-frame differential intensity to obtain the key frame actions and their subscripts with relatively uniform distribution.

3. The human motion similarity measurement method according to claim 1, characterized in that: The method for extracting the action interval in step 1 is: Step 1.2.1, set an upper threshold and a lower threshold to determine whether the current frame is in motion, that is, the interval of the test action. The upper threshold is used to determine the start of the action, and the lower threshold is used to determine the end of the action. Using two thresholds can achieve more accurate capture. The condition for judging the start of the action is the appearance of an average inter-frame differential intensity greater than the upper threshold. The end position is searched in the subsequent interval. The end of the search is marked by the continuous appearance of at least x average inter-frame differential intensities below the lower threshold. Step 1.2.2, detect the length of the interval searched in step 1.2.1, filter out the interval outside the threshold, find the key frame position with the maximum value below the start position and the minimum value above the end position on the key frame, and then expand the left and right equal lengths to complete the extraction of the test action interval.

4. The human motion similarity measurement method according to claim 1, characterized in that: The step 2 is specifically as follows: Step 2.1, define joint[0]~joint[14] as the joint positions of the skull, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, coccyx, right hip, right knee, right ankle, left hip, left knee, and left ankle, respectively. The two-dimensional joint coordinate information of a person is obtained from a human action video using motion capture technology, which is based on the human joint coordinate information extracted frame by frame from the video. Step 2.2, uniformly scale the spine lengths of jiont[1] and jiont[8] to length t. Assume that the coordinates of jiont[1] are (jiont[1][0], jiont[1][1]), and the coordinates of jiont[8] are (jiont[8][0], jiont[8][1]). The second dimension coordinate 0 represents the horizontal coordinate x, and 1 represents the vertical coordinate y. Then the spine scaling ratio r is: （7） Then the coordinates representing the joint points are mapped from the QT coordinate system to the rectangular coordinate system, so that the center position of the human spine coincides with the origin of the coordinate system, and then each bone is uniformly scaled like the origin according to the scaling ratio r; Step 2.3, use drawing tools to connect each joint point in an orderly manner to generate a skeleton diagram that can reflect human body movements.

5. The human motion similarity measurement method according to claim 1, characterized in that: When the twin neural network model is trained in step 3, each picture in the training set is a skeleton image processed in step 2; each sample in the training set is a labeled picture of a test picture and a standard picture in pairs, each group of pictures corresponds to a label, two skeleton images of the same type are positive samples, and the label is recorded as 1, and two skeleton images of different types are negative samples, and the label is recorded as 0.

Images