Multi-camera Time Synchronization Method Based on Human Posture

Abstract

The present invention belongs to the field of computer vision. To determine the time difference during video shooting with small error and simple operation, the present invention provides a multi-camera time synchronization method based on human pose, and the steps are as follows: Step 1: Given videos V 1 and V 2 shot by two cameras with different perspectives of the same space; Step 2: Use the HRNet human 2D pose estimation method to obtain the 2D poses P 1 and P 2 of the human body in each frame of V 1 and P 2 ; Step 3: Use a pre-trained feature embedding network to obtain the embedded features F 1 and F 2 of each frame of P 1 and F 2 ; Step 4: According to the similarity between F 1 and F 2 , obtain the matching matrix X; Step 5: According to the matching matrix X, obtain the matching vectors O between frames in V 1 and V 2 ; Step 6: Based on the matching vectors O, determine the time difference during the shooting of V 1 and V 2 to achieve the goal of video synchronization. The present invention is mainly applied to video processing and synchronization scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to a multi-view video synchronization method based on human pose. It can be used to synchronize the videos captured by multi-view cameras in the same space where there are people moving, accurately find the time difference between the two videos during shooting, and specifically relates to a multi-camera time synchronization method based on human pose. Background Art

[0002] (1) Human 2D pose estimation method (see reference [1]): Human 2D pose estimation, as an important part of data preprocessing, has been widely applied to various fields such as action recognition, human-computer interaction, and virtual reality. The methods for 2D pose estimation based on deep learning can be divided into two categories, namely top-down methods and bottom-up methods. The former first detects the human body in the image and then performs pose estimation. The latter first detects the human body joint points and then connects the detected joint points to obtain the human pose. Currently, the 2D pose estimation method based on deep learning already has relatively high accuracy. The present invention uses the pose estimation method HRNet to obtain the 2D pose of the human body in the video.

[0003] (2) Multi-view camera system: With the in-depth development of the intelligent society and the popularization of mobile devices, there are a large number of multi-view camera systems in life and production. They can capture the same space from different angles and provide more comprehensive information. The present invention aims to solve the synchronization problem of the videos captured by multi-view cameras. Because only after the multi-view videos are synchronized can the characteristics of their comprehensive information be better utilized for collaborative analysis, such as cooperative tracking, multi-view target association, three-dimensional reconstruction based on multi-views, etc.

[0004] (3) Camera pose (see reference [2]): Camera pose refers to the three-dimensional spatial position and three-dimensional spatial attitude of the camera in a certain reference coordinate system.

[0005] (4) Graph convolutional neural network (see reference [3]): The graph structure has great advantages in modeling the relationships between multiple nodes. The graph convolutional neural network applies convolutional operations to graph-structured data and has very strong feature learning ability and relationship representation ability. The present invention uses the graph convolutional neural network ST-GCN that builds a graph structure for human body joint points and makes certain adjustments to its network structure to be applicable to video synchronization. The adjusted ST-GCN in this article is called the feature embedding network.

[0006] For the actual work requirements, there is an important difficulty: the pose of the camera during video shooting is unknown, and accurate camera pose estimation is relatively complex and difficult to achieve in some special scenarios. Therefore, the present invention needs to be implemented under the condition of unknown camera pose. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, the present invention aims to propose an effective time synchronization method for videos shot by multi-view cameras. Based on the consistency of human posture at different perspectives at the same time, a relatively simple and effective strategy is used to determine the time difference between video shots. This method has the advantages of small error and simple operation. To this end, the technical solution adopted by the present invention is a multi-camera time synchronization method based on human posture, which includes the following steps:

[0008] Step 1: Given two videos captured by cameras with different viewpoints of the same space, denoted as V 1 and V 2 ;

[0009] Step 2: Use HRNet human 2D pose estimation method to get V 1 and V 2 The 2D posture of the human body in each frame is represented by P 1 and P 2 ;

[0010] Step 3: Use the pre-trained feature embedding network to get P 1 and P 2 The embedded features of each frame are denoted as F 1 and F 2 ;

[0011] Step 4: According to F 1 and F 2 The similarity between them is used to obtain the matching matrix X;

[0012] Step 5: According to the matching matrix X, get V 1 and V 2 In the example, the matching vector O between frames;

[0013] Step 6: Determine V based on the matching vector O 1 and V 2 The time difference when shooting is used to achieve the goal of video synchronization.

[0014] Step 2: Extract V using pre-trained HRNet 1 and V 2 The human body posture of each frame is represented by P 1 ∈R M×K×D and P 2 ∈R N×K×D , where M and N are V 1 and V 2 The length of , K is the number of human joints detected, and D is the feature dimension of each joint.

[0015] The purpose of calculating the matching matrix is to obtain V1 and V 2 The corresponding relationship between frames in

[0016] (1) Input P 1 and P 2 obtained from the previous operation into the feature embedding network to obtain the embedding features of each frame, denoted as F 1 ∈R M×C and F 2 ∈R N×C , where C is the dimension of the embedding features of each frame. The feature embedding network is trained in a self-supervised manner according to F 1 and F 2 ;

[0017] (2) Calculate the similarity matrix S ∈ R 1 [ 2 1 and F 2 between them, and define each similarity score in the matrix as: M×N

[0018]

[0019] ]>[

[0019] where S m，n represents the similarity score between the m-th frame in F 1 and the n-th frame in F 2 ;

[0020] (3) Perform a row-wise Softmax operation on the above S to obtain the matching matrix X ∈ R M×N , and each value in the matrix is:

[0021]

[0022] where, X m，n ∈ [0,1], and the larger the value, the more matching the m-th frame in F 1 and the n-th frame in F 2 .

[0023] Calculate the matching vector:

[0024] Perform an Argmax operation on each row of the matching matrix obtained in the previous step to obtain the matching vector O ∈ R M×1 : M×1

[0025] O = [o1, o2, o3, …, o M M

[0026] where the index of each element in O represents the corresponding frame in V 1 , and the value represents the corresponding matching frame in V 2 .

[0027] Determine the shooting time difference:

[0028] Determine V using the matching vector 1 and V 2 The time difference at the start of shooting is obtained by calculating the time difference between all matching frame pairs:

[0029]

[0030] Among them, O f To determine V 1 and V 2 The time difference at the start of shooting, Med() is the median operation.

[0031] The characteristics and beneficial effects of the present invention are:

[0032] 1. This invention synchronizes videos based on the 2D pose of the human body in the video, rather than the appearance features of the image. On the one hand, human 2D pose extraction is relatively mature, and it is easy to obtain relatively accurate human 2D pose from images for video synchronization. On the other hand, compared with the appearance features of the entire image, the embedded features based on the human 2D pose are more likely to extract consistency between different viewpoints at the same time, thereby achieving better synchronization performance.

[0033] 2. Many existing methods rely on the posture of the camera during video shooting. In contrast, the present invention does not rely on the posture of the camera during video shooting, and has no requirements on whether the shooting camera is a fixed camera or a mobile camera, thereby improving the adaptability of the method.

[0034] 3. The feature embedding network in this invention is trained in a self-supervised manner, eliminating the difficulty of training data labeling. Combining the self-supervised training method proposed in this method with the strategy for determining the time difference between video shots, it successfully solves the problem of multi-view video synchronization in specific scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 : Schematic diagram of images taken by cameras with different perspectives in the same space.

[0036] Figure 2 : Operation flow chart of multi-view video time synchronization method. DETAILED DESCRIPTION

[0037] The technical solution adopted by this invention is a multi-camera time synchronization method based on human posture. It is mainly explained from three aspects: first, the technical solution adopted to solve practical problems, and second, the complete implementation process based on the above solution.

[0038] 1. Technical solution: The technical solution of the present invention is listed. Figure 2 shown.

[0039] Step s1: Given videos captured by two cameras with different perspectives of the same space, denoted as V 1 and V 2 ;

[0040] Step s2: Using the HRNet human 2D pose estimation method, obtain the 2D poses of the human body in each frame of V 1 and V 2 , denoted as P 1 and P 2 ;

[0041] Step s3: Using a pre-trained feature embedding network to obtain the embedded features of each frame of P 1 and P 2 , denoted as F 1 and F 2 ;

[0042] Step s4: According to the similarity between F 1 and F 2 , obtain the matching matrix X;

[0043] Step s5: According to the matching matrix X, obtain the matching vectors O between frames in V 1 and V 2 ;

[0044] Step s6: Based on the matching vector O, determine the time difference when V 1 and V 2 are captured to achieve the goal of video synchronization.

[0045] II. Implementation of the multi-view video synchronization method

[0046] (I) Extraction of human poses

[0047] The input of the feature embedding network is the human pose of each frame in the video, so human pose extraction needs to be carried out first. The specific method is: using the pre-trained HRNet to extract the human poses of each frame in V 1 and V 2 , denoted as P 1 ∈R M ×K×D and P 2 ∈R N×K×D , where M and N are the lengths of V 1 and V 2 respectively, K is the number of detected human joint points, and D is the feature dimension of each joint.

[0048] (II) Calculation of the matching matrix

[0049] The purpose of calculating the matching matrix is to obtain V 1 and V2 The correspondence between the middle frame and the frame, the specific steps are as follows:

[0050] (1) The P obtained in the previous step 1 and P 2 Input into the feature embedding network to get the embedding of each frame, denoted as F 1 ∈R M×C and F 2 ∈R N×C , where C is the dimension of the embedded features of each frame. The feature embedding network is based on F 1 and F 2 Trained in a self-supervised manner.

[0051] (2) Calculate F 1 and F 2 The similarity matrix S∈R M×N , define each similarity score in the matrix as:

[0052]

[0053] Among them S m，n F 1 The mth frame and F 2 The similarity score of the nth frame in .

[0054] (3) Perform row Softmax operation on the above S to obtain the matching matrix X∈R M×N , each value in the matrix is:

[0055]

[0056] Among them, X m，n ∈[0,1], the larger the value, the higher the F 1 The mth frame and F 2 The closer the nth frame is to the match.

[0057] (3) Calculating matching vectors

[0058] Perform Argmax operation on each row of the matching matrix obtained in the previous step to obtain the matching vector O∈R M×1 :

[0059] O=[o1,o2,o3,…,o M ]

[0060] The index of each element in O represents V 1 The corresponding frame in V 2 The corresponding matching frames in .

[0061] (4) Determine the shooting time difference

[0062] Determine V using the matching vector 1 and V 2 The time difference at the start of shooting. The specific method is to obtain the time difference between videos based on the time differences between all pairs of matching frames:

[0063]

[0064] where O f is the determined time difference between V 1 and V 2 at the start of shooting, and Med() is the median operation.

[0065] As described above, this is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

[0066] References

[0067] [1]Sun K,Xiao B,Liu D,et al.Deep high-resolution representationlearning for human pose estimation[C] / / Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition.2019:5693-5703.

[0068] [2]Wang Jiang.Research on Camera Pose Estimation Method Based on Deep Learning[D].Shanxi University,2020

[0069] [3]Yan S,Xiong Y,Lin D.Spatial temporal graph convolutional networksfor skeleton-based action recognition[C] / / Thirty-second AAAI conference onartificial intelligence.2018。

Claims

1. A multi-camera time synchronization method based on human body postures, characterized in that the steps As follows: Step 1: Given videos captured by two cameras with different perspectives of the same space, denoted as V 1 and V 2 ; Step 2: Using the HRNet human 2D pose estimation method, obtain V 1 and V 2 The 2D pose of the human body in each frame, denoted as P 1 and P 2 , where the pre-trained HRNet is used to extract the human pose of each frame in V 1 and V 2 , respectively denoted as P 1 ∈R M×K×D and P 2 ∈R N×K×D , where M and N are the lengths of V 1 and V 2 , K is the number of detected human joint points, and D is the feature dimension of each joint; Step 3: Obtain P using a pre-trained feature embedding network 1 and P 2 The embedded features of each frame, denoted as F 1 and F 2 ; Step 4: According to the similarity between F 1 and F 2 obtain the matching matrix X. The purpose of calculating the matching matrix is to obtain the correspondence between frames in V 1 and V 2 . The specific steps are as follows: (1) Input the P obtained in the previous step 1 and P 2 into the feature embedding network to obtain the embedding features of each frame, denoted as F 1 ∈R M×C and F 2 ∈R N×C , where C is the dimension of the embedding features of each frame, and the feature embedding network is trained in a self-supervised manner based on F 1 and F 2 ; (2) Calculate F 1 and F 2 to obtain the similarity matrix S ∈ R M×N , and define each similarity score in the matrix as follows: where S m，n represents the similarity score between the m-th frame in F 1 and the n-th frame in F 2 ; (3) Perform a Softmax operation on the similarity matrix S to obtain a matching matrix X ∈ R M×N , where each value in the matching matrix is: where X m，n ∈ [0, 1], the larger the value, the more 1 the m-th frame in F 2 matches the n-th frame in F; Step 5: Obtain V according to the matching matrix X 1 and V 2 In, the matching vector O between frames, the specific steps for calculating the matching vector are as follows: Perform the Argmax operation on each row of the matching matrix obtained in the previous step, and then obtain the matching vector O ∈ R M×1 : O = [o1, o2, o3,..., o M ​ where the index of each element in O represents the corresponding frame in V 1 and the value represents the corresponding matching frame in V 2 ; Determine the shooting time difference: Determine V using the matching vector 1 and V 2 The time difference at the start of shooting. The specific method is to obtain the time difference between videos based on the time differences between all pairs of matching frames: Among them, O f is the determined V 1 and V 2 is the time difference at the start of shooting, and Med( ) is the median operation; Step 6: Based on the matching vectors O, V 1 and V 2 and the time difference during shooting, to achieve the goal of video synchronization.

Images