An optical motion capture method for complex backgrounds

Abstract

The present invention discloses an optical motion capture method for use in complex backgrounds. In order to solve the problem of marker point recognition and tracking by an optical motion capture system under complex backgrounds, a two-check method is adopted to supervise the marker recognition and tracking process. The first check is a short-time check, which uses the nearest neighbor similarity and is implemented in each frame image acquisition process; the second check is a long-time check, which uses the DeepSORT multi-target tracking method based on YOLOv5 to be implemented at regular intervals of frames. By ensuring the correctness of marker recognition and tracking during motion capture under complex backgrounds through these two long and short checks, the dependence of motion capture technology on environmental conditions can be greatly reduced, and the application scenarios of motion capture can be expanded.

Description

Technical Field

[0001] The present invention belongs to the technical field of motion capture, and in particular relates to an optical motion capture method applied in complex backgrounds. Background Art

[0002] Optical motion capture technology uses image analysis and visual 3D reconstruction to recover the spatial coordinates of spherical (or circular) markers placed on a rigid target, thereby capturing target motion information. Due to its non-contact, large-scale, and high-precision capabilities, this technology has attracted significant attention in the fields of motion measurement and film and television production. Currently, numerous motion capture systems are commercially available. While all utilize high-frame-rate motion capture cameras (30-300 fps), ensuring high correlation between moving images for typical applications, their algorithms often employ morphological filtering for target recognition and nearest-neighbor frame difference tracking to improve real-time processing speed. This results in demanding ambient lighting requirements for motion capture applications. Complex backgrounds (such as those with numerous reflective objects) can lead to failures in target recognition and motion tracking. Consequently, target recognition is difficult and tracking stability is poor in complex backgrounds.

[0003] With the continuous development of deep learning technology, convolutional neural networks have significantly improved the recognition and tracking performance of targets in complex backgrounds. However, deep learning methods require high computing power and have a low real-time image processing rate (generally no more than 30 frames per second). Therefore, directly using deep neural networks to identify and track markers in complex backgrounds results in low data rates, which cannot meet the data rate requirements of motion capture. Summary of the Invention

[0004] The present invention aims to solve the above-mentioned technical problems existing in the prior art and provides an optical motion capture method for use in complex backgrounds.

[0005] The technical solution of the present invention is: an optical motion capture method applied to complex backgrounds, which is carried out in the following steps:

[0006] Step 1: Set up n motion capture cameras, each of which is connected to a switch, which is connected to a PC. Set any motion capture camera as the main camera and name it the first camera. Then, according to the neighbor relationship, name the n-1 motion capture cameras in a clockwise direction as the second camera, the third camera, the fourth camera, ..., the nth camera. At the same time, set the motion capture frame rate to Vs ;

[0007] Step 2: PC captures the frame rate of the motion VsCollect the images taken by each camera respectively and determine whether it is the first frame image. If yes, proceed to step 3; otherwise, proceed to step 4;

[0008] Step 3: Based on YOLOv5, perform marker point target recognition on the first frame of the captured image, select the M marker point positions that constitute the rigid body on the image plane and store them, and return to step 2;

[0009] Step 4: Use the template matching method to search and match the position of the marker point after it moves in each camera image plane, with the coordinates of the marker point in the previous frame as the center and the neighboring set range R as the radius, to obtain the motion vector of the marker point on each camera image plane;

[0010] Step 5: Short-time verification process;

[0011] According to the camera serial number, the cameras are combined in pairs in turn, and the correctness of the motion vector of the marker point on the image plane is identified by the similarity of the topological structure between the marker points on the camera image plane. If the similarity between the topological structures of the marker points of the two combined cameras is higher than the threshold, the motion vector of the marker point on the image plane is determined to be correct. Otherwise, the motion vector of the marker point on the image plane of the camera whose topological structure similarity with the marker point of the other camera is higher than the threshold is used as a reference, and the new marker point coordinates are searched on the image plane of the current camera by using the area matching method. The new marker point coordinates are used to replace the original marker point coordinates, and the corresponding marker point motion vector is formed according to step 4;

[0012] Step 6: Determine whether the acquired image is Vp frame, the Vp = Vs / k, K is a positive integer; if the image of the first camera is Vp Frame, proceed to step 7 and 8 in sequence, otherwise proceed to step 10; if the image of the second camera, the third camera, the fourth camera, ..., the nth camera is Vp Frame, go to step 9, otherwise go to step 10;

[0013] Step 7: First Camera Vp The frame image starts the DeepSORT motion tracking thread;

[0014] Step 8: Long-term verification process;

[0015] Compare the DeepSORT tracking result obtained in step 7 with the motion vector of the marker point on the image plane. If they are consistent, the tracking result is determined to be correct and proceed to step 10. Otherwise, use the DeepSORT tracking result as a reference and use the region matching method to search for new marker point coordinates on the image plane of this camera, and replace the original marker point coordinates with the new marker point coordinates, and form the new marker point motion vector according to step 4. At the same time, use the topological structure similarity of the marker point set of neighboring cameras to replace the M marker points of the second camera to the nth camera with the same-name points corresponding to the updated marker point of the first camera in turn as the marker point position; return to step 2;

[0016] Step 9. Determine whether to re-track based on whether the marker position is updated. If yes, return to step 4; otherwise, proceed to step 10.

[0017] Step 10: Based on the principle of multi-view geometry, perform three-dimensional reconstruction of the marker points that make up the rigid body at each frame moment to obtain its spatial coordinates; calculate the motion information of the rigid body by changing the coordinates of the M marker points that make up the rigid body on the image plane between the current moment and the first frame image.

[0018] The present invention addresses the problem of marker point recognition and tracking in optical motion capture systems under complex backgrounds. It uses a two-check method to supervise the marker recognition and tracking process. The first check is a short-time check, which uses the nearest neighbor similarity and is implemented during each frame image acquisition process; the second check is a long-time check, which uses the DeepSORT multi-target tracking method based on YOLOv5 to be implemented at regular intervals. By ensuring the accuracy of marker recognition and tracking during motion capture under complex backgrounds through these two long and short checks, the dependence of motion capture technology on environmental conditions can be greatly reduced, and the application scenarios of motion capture can be expanded. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The present invention is an optical motion capture method applied to complex backgrounds, such as Figure 1 As shown, follow the steps below:

[0021] Step 1: Set up n motion capture cameras. Each motion capture camera is connected to a switch (POE Gigabit network switch) via an Ethernet cable. The switch is connected to a PC via an Ethernet cable. Set any motion capture camera as the main camera and name it the first camera. Then, according to the neighbor relationship, name the n-1 motion capture cameras in a clockwise direction as the second camera, the third camera, the fourth camera, ..., the nth camera. At the same time, set the motion capture frame rate to Vs ;

[0022] Step 2: PC captures the frame rate of the motion Vs Collect the images taken by each camera respectively and determine whether it is the first frame image (original position moment). If yes, proceed to step 3; otherwise, proceed to step 4.

[0023] Step 3: Based on YOLOv5, perform marker point target recognition on the first frame of the acquired image, and select M marker point positions that constitute the rigid body on the image plane, and return to step 2;

[0024] Step 4: Use the template matching method to search and match the position of the marker point after it moves in each camera image plane, with the coordinates of the marker point in the previous frame as the center and the neighboring set range R as the radius, to obtain the motion vector of the marker point on each camera image plane;

[0025] Step 5: Short-time verification process;

[0026] According to the camera sequence, the cameras are combined in pairs, such as the 1st camera and the 2nd camera, the 2nd camera and the 3rd camera, the 3rd camera and the 4th camera, and so on, the n-1th camera and the nth camera. The correctness of the motion vector of the marker point on the image plane is identified by the topological structure similarity between the marker points on the camera image plane (which can be calculated using the normalized covariance correlation coefficient method). If the similarity between the topological structures of the marker points of the two combined cameras is higher than the threshold, the motion vector of the marker point on the image plane is determined to be correct. Otherwise, the motion vector of the marker point on the image plane of the camera whose topological structure similarity with the marker point of the other camera is higher than the threshold is used as a reference. The new marker point coordinates are searched on the image plane of the current camera using the region matching method, and the new marker point coordinates are used to replace the original marker point coordinates. The corresponding marker point motion vector is formed according to step 4.

[0027] For example, if the similarity between the topological structures of Marker point 1 on the image planes of Camera 2 and Camera 3 is higher than the set threshold, the motion vectors of the Marker points on the image planes of Camera 2 and Camera 3 are determined to be correct. If the similarity between the topological structures of Marker point 1 on the image planes of Camera 3 and Camera 4 is lower than the set threshold, it is necessary to use the Marker 1 coordinate information of Camera 3 as a reference, use the region matching method to search for the Marker coordinates on the image plane of Camera 4, and replace the original Marker 1 coordinates of Camera 4 with these new Marker 1 coordinates;

[0028] Step 6: Determine whether the acquired image is Vp frame, the Vp = Vs / k, K is a positive integer; if the image of the first camera is Vp Frame, proceed to step 7 and 8 in sequence, otherwise proceed to step 10; if the image of the second camera, the third camera, the fourth camera, ..., the nth camera is Vp Frame, go to step 9, otherwise go to step 10;

[0029] Step 7: First Camera Vp The frame image starts the DeepSORT motion tracking thread;

[0030] Step 8: Long-term verification process;

[0031] Compare the DeepSORT tracking result obtained in step 7 with the motion vector of the marker point on the image plane. If they are consistent, the tracking result is determined to be correct and proceed to step 10. Otherwise, use the DeepSORT tracking result as a reference and use the region matching method to search for new marker point coordinates on the image plane of this camera, and replace the original marker point coordinates with the new marker point coordinates, and form the new marker point motion vector according to step 4. At the same time, use the topological structure similarity of the marker point set of the neighboring cameras to replace the M marker points of the second camera to the nth camera with the same-name points corresponding to the updated marker point of the first camera (the particle tracking velocimetry (PTV) method in the particle image velocimetry technology can be used to find the same-name points) as the marker point position; return to step 2.

[0032] Step 9. Determine whether to re-track based on whether the marker position is updated. If yes, return to step 4; otherwise, proceed to step 10.

[0033] Step 10: According to the existing technology, based on the principle of multi-view geometry, the marker points that make up the rigid body at each frame moment are reconstructed in three dimensions to obtain their spatial coordinates; the motion information of the rigid body is calculated by the coordinate changes of the positions of the M marker points that make up the rigid body selected on the image plane between the current moment and the first frame image.

Claims

1. An optical motion capture method for complex backgrounds, characterized in that Follow the steps below: Step 1: Set up n motion capture cameras, each of which is connected to a switch, which is connected to a PC. Set any motion capture camera as the main camera and name it the first camera. Then, according to the neighbor relationship, name the n-1 motion capture cameras in a clockwise direction as the second camera, the third camera, the fourth camera, ..., the nth camera. At the same time, set the motion capture frame rate to Vs ; Step 2: PC captures the frame rate of the motion Vs Collect the images taken by each camera respectively and determine whether it is the first frame image. If yes, proceed to step 3; otherwise, proceed to step 4; Step 3: Based on YOLOv5, perform marker point target recognition on the first frame of the captured image, select the M marker point positions that constitute the rigid body on the image plane and store them, and return to step 2; Step 4: Use the template matching method to search and match the position of the marker point after it moves in each camera image plane, with the coordinates of the marker point in the previous frame as the center and the neighboring set range R as the radius, to obtain the motion vector of the marker point on each camera image plane; Step 5: Short-time verification process; According to the camera serial number, the cameras are combined in pairs in turn, and the correctness of the motion vector of the marker point on the image plane is identified by the similarity of the topological structure between the marker points on the camera image plane. If the similarity between the topological structures of the marker points of the two combined cameras is higher than the threshold, the motion vector of the marker point on the image plane is determined to be correct. Otherwise, the motion vector of the marker point on the image plane of the camera whose topological structure similarity with the marker point of the other camera is higher than the threshold is used as a reference, and the new marker point coordinates are searched on the image plane of the current camera by using the area matching method. The new marker point coordinates are used to replace the original marker point coordinates, and the corresponding marker point motion vector is formed according to step 4; Step 6: Determine whether the acquired image is Vp frame, the Vp=Vs / K , K is a positive integer; if the image of the first camera is Vp Frame, proceed to steps 7 and 8 in sequence, otherwise proceed to step 10; if the image of the second camera, the third camera, the fourth camera, ..., the nth camera is Vp Frame, go to step 9, otherwise go to step 10; Step 7: First Camera Vp The frame image starts the DeepSORT motion tracking thread; Step 8: Long-term verification process; Compare the DeepSORT tracking result obtained in step 7 with the motion vector of the marker point on the image plane. If they are consistent, the tracking result is determined to be correct and proceed to step 10. Otherwise, use the DeepSORT tracking result as a reference and use the region matching method to search for new marker point coordinates on the image plane of this camera, and replace the original marker point coordinates with the new marker point coordinates, and form the new marker point motion vector according to step 4. At the same time, use the topological structure similarity of the marker point set of neighboring cameras to replace the M marker points of the second camera to the nth camera with the same-name points corresponding to the updated marker point of the first camera in turn as the marker point position; return to step 2; Step 9. Determine whether to re-track based on whether the marker position is updated. If yes, return to step 4; otherwise, proceed to step 10. Step 10: Based on the principle of multi-view geometry, perform three-dimensional reconstruction of the marker points that make up the rigid body at each frame moment to obtain its spatial coordinates; calculate the motion information of the rigid body by changing the coordinates of the M marker points that make up the rigid body on the image plane between the current moment and the first frame image.

Images