Method for extracting three-dimensional sliding trajectory of bobsleigh and luge based on video and point cloud data

Abstract

The present invention discloses a method for extracting three-dimensional sliding trajectories of bobsleighs and luges based on video and point cloud data. The method comprises: camera calibration based on 3D point cloud and video frame key point matching point pairs; video data segmentation based on scene content; fitting curves on both sides of the track edge to avoid interference with detection by moving objects outside the track; detecting the position of the bobsleigh and luge using a background difference method; tracking using a Kalman filter; smoothing the trajectory after tracking is completed; and converting the 2D coordinates of the sliding trajectory into 3D coordinates. The present invention uses a set of video frame and point cloud key point coordinate pairs to estimate camera parameters, and can obtain accurate camera calibration results without calibration objects, saving manpower and costs. 2D coordinates are converted to 3D coordinates by establishing a 2D-3D mapping table using track surface constraints. Without the need for complex manual labeling, the 2D sliding trajectory of the bobsleigh and luge is accurately extracted and converted into a 3D trajectory.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data. Background Art

[0002] Target tracking technology is widely used in autonomous driving systems, intelligent robots, intelligent security, human motion recognition and other fields. Applying target tracking technology to bobsleigh and skeleton to extract the motion trajectory of bobsleigh and skeleton is of great significance for guiding athlete training and improving performance.

[0003] Before converting 3D world coordinates to 2D image coordinates, camera calibration is required. Conventional camera calibration requires the placement of a calibration object of known size, such as a calibration plate (Z. Zhang, "A flexible new technique for camera calibration," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, Nov. 2000, doi: 10.1109 / 34.888718.). Efficiently and accurately estimating camera parameters based on only video data and 3D point cloud data, without a calibration object in the video image, remains a challenging problem.

[0004] Object detection and tracking methods based on machine learning and deep learning require a large amount of labeled data when training models (S. Ren, K. He, R. Girshick and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017, doi: 10.1109 / TPAMI.2016.2577031.). However, how to accurately and efficiently extract the sliding trajectory of bobsleighs and sleds without the need for labeling needs to be studied.

[0005] Due to the lack of depth information, deriving 3D trajectories from 2D trajectories using camera parameters is an ill-posed problem. Most methods assume that objects move in the same plane, such as the ground (Z. Boukhers, K. Shirahama and M. Grzegorzek, "Example-Based 3D Trajectory Extraction of Objects From 2D Videos," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 9, pp. 2246-2260, Sept. 2018, doi: 10.1109 / TCSVT.2017.2727963.). However, bobsleigh and luge tracks cannot be approximated as a flat surface. Therefore, research is needed to find a way to efficiently and uniquely translate 2D trajectories into 3D trajectories by adding track constraints. Summary of the Invention

[0006] To solve the above problems, the present invention proposes a method for extracting the three-dimensional sliding trajectory of bobsleigh and luge based on video and point cloud data, which avoids a lot of complicated work of camera calibration using calibration plates, efficiently converts 2D coordinates on the track to 3D coordinates, and optimizes according to the characteristics of the task, thereby extracting accurate trajectory coordinates while eliminating a lot of labeling work.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data includes the following steps:

[0009] S1: Acquire 3D point cloud data of the track and use multiple cameras to shoot videos of the bobsleigh and luge sliding on the track. Combine the videos shot by multiple cameras into a video recording the sliding process of the bobsleigh and luge based on synchronized time information.

[0010] S2: Video data preprocessing: dividing the video recording the bobsleigh and sled sliding process into several video segments, each corresponding to a camera, and further dividing the video segment into several images by frame;

[0011] S3: calibrating all cameras to estimate their parameters and generating a 2D-3D coordinate mapping table of the track surface corresponding to each camera;

[0012] S4: Fitting the curve expressions of the edges on both sides of the track for each camera’s lens scene;

[0013] S5: Detect the position of the bobsleigh in the current frame image using a background difference algorithm;

[0014] S6: Use the Kalman filter to predict the position of the bobsleigh in the next frame of the image;

[0015] S7: Select the real target from several candidate targets and update the Kalman filter;

[0016] S8: Perform smoothing post-processing on the 2D trajectory coordinates to obtain the 2D coordinates of the sliding trajectory of the bobsleigh under each camera;

[0017] S9: Convert all 2D sliding trajectory segments into 3D trajectories according to the track surface 2D-3D coordinate mapping table of each camera to obtain the 3D trajectory of the sliding process.

[0018] In S2, the video data preprocessing specifically includes the following steps:

[0019] S2.1: Automatically segment the bobsleigh and sled sliding process video into video segments corresponding to different cameras based on scene detection.

[0020] S2.2: Read the time code to split the entire bobsleigh and skeleton sliding video into video segments corresponding to different cameras;

[0021] S2.3: After obtaining the segmented video segments, further segment the video segments into a plurality of frame images.

[0022] In S3, the process of camera parameter estimation is as follows:

[0023] S3.1: Select several corner points on the image and use Cyclone software to extract the corresponding 3D points in the 3D point cloud data to obtain the set of 2D-3D key point coordinate pairs under the current camera and save them to a file;

[0024] The number of selected corner points is greater than or equal to 6;

[0025] S3.2: Assume that the image pixel coordinate system is The image plane coordinate system is The image space coordinate system is The Cartesian coordinate system (3D world coordinate system) is The transformation relationship between the pixel coordinate system and the Cartesian space coordinate system is shown in formula (1):

[0026]

[0027] From formula (1), we can get:

[0028]

[0029] Among them, l1,l2,…,l 12 is the relationship coefficient between the Cartesian space coordinate system and the pixel coordinate system, let l 12 =1, find the parameter l 11 -l 11 ;

[0030] Transform formula (1) into formula (2):

[0031]

[0032] Among them, the parameter l1-l 11 is an unknown number. Substituting each 2D-3D point pair into equation (2) yields two equations. At least six 2D-3D point pairs are required to solve equation (2).

[0033] make L=[l1l2 l3 l4 l5l6 l7 l8 l9 l 10 l 11 ] T , C=[uv] T , Equation (2) is recorded as BL-C=0, then the analysis of the unknown matrix L is shown in Equation (3)

[0034] L=(B T B) -1 (B T C) (3)

[0035] The value of L is obtained by the least square method;

[0036] S3.3: For each camera, save the parameters of mapping the 3D points to the 2D image through steps S3.1-S3.2.

[0037] In S3, the process of generating the 2D-3D coordinate mapping table includes the following steps:

[0038] SS3.1: Extract a set of 3D points from the track surface from the 3D point cloud data. This serves as a constraint for mapping the 2D trajectory to 3D. The track surface data for each camera lens scene is then exported separately.

[0039] SS3.2: Read the 3D points on the track surface beneath each camera. Each 3D point is represented by its (x, y, z) coordinates. Based on the estimated camera parameters, calculate the corresponding 2D pixel coordinates (u, v). Decimal 2D coordinates are rounded.

[0040] SS3.3: Establish a (W, H, 3) matrix as a mapping table from 2D points to 3D points, where W and H are the width and height of any frame image of the current camera, respectively, and 3 represents the number of channels, which are used to store the coordinates of the 3D point corresponding to the current pixel position. When a 2D image point corresponds to multiple 3D points in the coordinates (u, v) of the 2D pixel point, the average value of the coordinates of the multiple 3D points is calculated and used as the 3D mapping result of the 2D point. If a 2D image point does not have a corresponding 3D point, the 3D coordinates corresponding to the current image point are estimated based on the 3D points corresponding to the nearby points on the image. With the 2D point as the center, a 7×7 range is framed, and the average value of the corresponding 3D coordinates of all 2D points with corresponding 3D coordinates in the range is calculated. The 3D average value is used as the corresponding point of the 2D point. If no point in the 7×7 range has a corresponding 3D coordinate, its 3D point is set to (0, 0, 0), and it is considered that the 2D image point has no corresponding 3D point.

[0041] In S4, for each shot scene, the process of fitting the curve expressions of the edges on both sides of the track is as follows:

[0042] S4.1: In any frame of the scene, evenly sample points along the edges of the two tracks;

[0043] S4.2: The track edge is fitted with multiple polynomials. The polynomial coefficients of the curves at both edges of the track are estimated using the least squares method. The polynomial coefficients are read to construct the curve expression.

[0044] In S5, the process of detecting the position of the bobsleigh in the image using the background difference algorithm is as follows:

[0045] S5.1: Obtain background image frame B through Gaussian mixture model, current image frame is F i , the grayscale values ​​of the corresponding pixels in the background frame and the current frame are recorded as B(x,y) and F i (x, y), subtract the grayscale values ​​of the corresponding pixels of the two frames of images, and take their absolute values ​​to obtain the difference image D n , as shown in formula (4):

[0046] D n (x,y)=|F i (x,y)-B(x,y)| (4)

[0047] S5.1: Set the threshold T and perform binarization on each pixel to obtain the binary image R' n , as shown in formula (5):

[0048]

[0049] Among them, the point with a gray value of 255 is the foreground point, indicating the moving target, and the point with a gray value of 0 is the background point.

[0050] S5.2: For image R' i Perform connected component analysis to obtain the image R containing the complete moving target i , according to R i The bounding box coordinates of the target are obtained, and the targets outside the track are filtered out using the track edge curve fitted in S4.

[0051] In S7, selecting a real target from a number of candidate targets and updating the Kalman filter includes the following steps:

[0052] S7.1: If there are multiple candidate targets near the bobsleigh position predicted by the Kalman filter in the next frame, identify the true target based on its area and proximity to the predicted position. Obtain the bounding box coordinates of all candidate targets and calculate their areas.

[0053] S7.2: Gaussian weighting is performed on the areas of all candidate targets, as shown in formula (6):

[0054]

[0055] Among them, (c x ,c y ) is the center point coordinate of the target, σ c The standard deviation is obtained based on the target bounding box area. The larger the target bounding box area, the larger the Gaussian kernel range of the target, and vice versa. The result of Gaussian weighting is that candidate targets close to the predicted target receive higher weights.

[0056] S7.3: Select the target with the largest weighted area as the true target in the current frame image and update the Kalman filter prediction value.

[0057] In S8, the 2D trajectory coordinate smoothing post-processing includes the following steps:

[0058] S8.1: Use Kalman filter to remove noise points in trajectory coordinate data;

[0059] S8.2: Read the denoised trajectory coordinate data and perform smoothing. Select n1 consecutive frames corresponding to bobsleigh coordinate points, where n1 is ≥ 5. Calculate the average of the horizontal and vertical coordinates of the n1 bobsleigh coordinate points, and use them as the smoothed intermediate frame bobsleigh coordinates.

[0060] S8.3: De-noise and smooth the n1 bobsleigh and sled coordinate points centered on each frame until all points on the trajectory are traversed.

[0061] S8.4: Finally, these local smoothing results are combined to obtain the smoothed 2D trajectory coordinates.

[0062] In S9, the 2D coordinates of the motion trajectory are converted into 3D coordinates, including the following steps:

[0063] S9.1: Read the smoothed 2D coordinates of the trajectory to obtain the center coordinates of the moving target;

[0064] S9.2: According to the coordinates of the center point of the moving target, the corresponding 3D coordinates are searched in the 2D-3D coordinate mapping table, and the corresponding data is saved in a file.

[0065] Beneficial technical effects

[0066] 1. Calibrate the camera more accurately without using a calibration plate. If the camera moves a certain amount, you can recalibrate it.

[0067] 2. Target tracking is performed based on a background difference algorithm, which is optimized for task characteristics, allowing for efficient and accurate extraction of bobsleigh and luge trajectories without requiring extensive manual annotation.

[0068] 3. When the depth is unknown, the 2D sliding trajectory of the bobsleigh and skeleton extracted from the video is mapped to the 3D real track.

[0069] 4. The present invention can be applied to sports such as bobsleigh and skeleton. The resulting 3D trajectory can help athletes understand the problems in their sliding process and the differences with excellent sliding trajectories, thereby improving training. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flowchart of a method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data provided by an embodiment of the present invention;

[0071] Figure 2 An example diagram of 2D image data and 3D point cloud data provided by an embodiment of the present invention;

[0072] in, Figure 2 (a) is a schematic diagram of the 3D point cloud data of the first scene, Figure 2 (b) is a schematic diagram of the point cloud data of the track surface extracted in the first scene. Figure 2 (c) is a schematic diagram of the corresponding 2D image data of the first scene, Figure 2 (d) The first scene is a schematic diagram of the result of mapping the 3D point cloud data of the track surface onto the 2D image;

[0073] Figure 3Schematic diagram of corresponding point sampling on a 2D image and a 3D point cloud provided by an embodiment of the present invention;

[0074] in, Figure 3 (a) Schematic diagram of key points used for camera calibration on 3D point cloud data in the second scene; Figure 3 (b) is a schematic diagram showing the key points used for camera calibration on the 2D image data corresponding to the third scene and the projection result of the camera parameters estimated using 2D-3D point pairs from 3D to 2D; Figure 3 (c) is a schematic diagram of key points used for camera calibration on the 3D point cloud data in the second scene; Figure 3 (d) is a schematic diagram showing the key points used for camera calibration on the 2D image data corresponding to the third scene and the projection result of the camera parameters estimated using 2D-3D point pairs from 3D to 2D;

[0075] Among them, the white points represent key points, and the gray points represent the results of projecting the camera parameters estimated by using 2D-3D point pairs from 3D to 2D;

[0076] Figure 4 A visualization of the track curve fitting results provided by an embodiment of the present invention;

[0077] in, Figure 4 (a) is a visualization diagram of the fitted track curve. Figure 4 (b) is a schematic diagram of the original image;

[0078] Figure 5 An example diagram of the detection results provided by an embodiment of the present invention;

[0079] in Figure 5 (a) is the visualization of the tracking trajectory on the 2D image of the fourth scene; Figure 5 (b) is the trajectory visualization on the 3D point cloud of the fourth scene; Figure 5 (c) is a schematic diagram of the mapping result of the 3D point cloud coordinates on the 2D image of the fourth scene; Figure 5 (d) is the tracking trajectory visualization on the 2D image of the fifth scene; Figure 5 (e) is the trajectory visualization on the 3D point cloud of the fourth scene; Figure 5 (f) is a schematic diagram of the mapping result of the 3D point cloud coordinates on the 2D image of the fourth scene; Figure 5 (g) is a visualization of the tracking trajectory on the 2D image of the sixth scene; Figure 5 (h) is the trajectory visualization on the 3D point cloud of the fourth scene; Figure 5 (i) is a schematic diagram of the mapping results of the 3D point cloud coordinates of the fourth scene on the 2D image. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in combination with the specific embodiments of this application and the corresponding drawings.

[0081] This embodiment provides a method for extracting three-dimensional sliding tracks of a bobsleigh and snowmobile based on video and point cloud data. Figure 1 As shown, the following steps are included:

[0082] S1: Obtain 3D point cloud data of the track and use multiple cameras to shoot the sliding video of the bobsleigh and skeleton on the track, such as Figure 2 As shown, the videos shot by multiple cameras are combined into a synchronized bobsleigh and snowmobile sliding video according to the synchronized time information;

[0083] S2: Video data preprocessing: dividing the synchronized bobsleigh and snowmobile sliding video into several video segments, each corresponding to a camera, and further dividing the video segment into several frame images; including the following steps:

[0084] S2.1: Automatically segment the synchronous bobsleigh and luge video into video segments corresponding to different cameras based on scene detection;

[0085] S2.2: Then, the time code is read to split the entire synchronous bobsleigh and skeleton sliding video into video segments corresponding to different cameras;

[0086] S2.3: After obtaining the segmented video segments, further segment the video segments into a plurality of frame images to facilitate bobsleigh and snowmobile detection and tracking.

[0087] S3: Calibrate all cameras separately, estimate the parameters of all cameras, and generate a 2D-3D coordinate mapping table of the track surface corresponding to each camera;

[0088] The process of camera parameter estimation includes the following steps:

[0089] S3.1: Select several corner points on the image and use Cyclone software to extract the corresponding 3D points in the 3D point cloud data, such as Figure 3 As shown, the 2D-3D key point coordinate pair set under the current camera is obtained and saved to a file;

[0090] The number of selected corner points is greater than or equal to 6;

[0091] S3.2: Assume that the image pixel coordinate system is The image plane coordinate system is The image space coordinate system is The Cartesian coordinate system (3D world coordinate system) is The transformation relationship between the pixel coordinate system and the Cartesian space coordinate system is shown in formula (1):

[0092]

[0093] From formula (1), we can get:

[0094]

[0095] Among them, l1,l2,…,l 12 is the relationship coefficient between the Cartesian space coordinate system and the pixel coordinate system, because l 12 The value of has no effect on the values ​​of other parameters. Let l 12 =1, find the parameter l 11 -l 11 ;

[0096] Transform formula (1) into formula (2):

[0097]

[0098] Among them, the parameter l 11 -l 11 is an unknown number, and each 2D-3D point pair requires two equations. At least 6 2D-3D point pairs are needed to solve equation (2);

[0099] In this embodiment, 10-20 point pairs are collected under each camera to meet the needs of calculating parameters;

[0100] make L=[l1 l2 l3 l4 l5l6 l7 l8 l9 l 10 l 11 ] T , C=[uv] T , Equation (2) is recorded as BL-C=0, then the analysis of the unknown matrix L is shown in Equation (3)

[0101] L=(B T B) -1 (B T C) (3)

[0102] The value of L is obtained by the least square method;

[0103] S3.3: For each camera, save the parameters of mapping the 3D points to the 2D image through steps S3.1-S3.2;

[0104] The generation process of the 2D-3D coordinate mapping table includes the following steps:

[0105] SS3.1: Extract a set of 3D points from the track surface from the 3D point cloud data. This serves as a constraint for mapping the 2D trajectory to 3D. The track is then exported for each camera lens scene.

[0106] SS3.2: Read the 3D points on the track surface beneath each camera. Each row of data contains (x, y, z) coordinates. Based on the estimated camera parameters, calculate the corresponding 2D pixel coordinates (u, v). Decimal 2D coordinates are rounded.

[0107] SS3.3: Establish a (W, H, 3) matrix as a mapping table from 2D points to 3D points, where W and H are the width and height of any frame image of the current camera, respectively, and 3 represents the number of channels, which are used to store the coordinates of the 3D point corresponding to the current pixel position. When a 2D image point corresponds to multiple 3D points in the coordinates (u, v) of the 2D pixel point, the average value of the coordinates of the corresponding multiple 3D points is calculated and used as the 3D mapping result of the 2D point. If a 2D image point does not have a corresponding 3D point, the 3D coordinates corresponding to the current image point are estimated based on the 3D points corresponding to the nearby points on the image. With the 2D point as the center, a 7×7 range is framed, and the average value of the corresponding 3D coordinates of all 2D points with corresponding 3D coordinates in the range is calculated. The 3D average value is used as the corresponding point of the 2D point. If no point has a corresponding 3D coordinate within the 7×7 range, its 3D point is set to (0, 0, 0), and it is considered that the 2D image point has no corresponding 3D point.

[0108] S4: For each camera’s shot scene, e.g. Figure 4 As shown in the figure, fitting the curve expressions of the edges on both sides of the track includes the following steps:

[0109] S4.1: In any frame of the scene, evenly sample points along the edges of the two tracks;

[0110] S4.2: The track boundary curve can be set using the quartic polynomial y = a0x 4 +a1x 3 +a2x 2 +a3x+a4 fitting, using the least squares method to estimate the values ​​of the polynomial coefficients a0,...,a4 of the curves at the two boundaries of the track, and read the polynomial coefficients to construct the curve expression;

[0111] S5: Detecting the position of the bobsleigh and snowmobile in the current frame image using a background difference algorithm; comprising the following steps:

[0112] S5.1: Obtain background image frame B through Gaussian mixture model, current image frame is F i , the grayscale values ​​of the corresponding pixels in the background frame and the current frame are recorded as B(x,y) and Fi (x, y), subtract the grayscale values ​​of the corresponding pixels of the two frames of images, and take their absolute values ​​to obtain the difference image D n , as shown in formula (4):

[0113] D n (x,y)=|F i (x,y)-B(x,y)| (4)

[0114] S5.1: Set the threshold T and perform binarization on each pixel to obtain the binary image R' n , as shown in formula (5):

[0115]

[0116] Among them, the point with a gray value of 255 is the foreground point, indicating the moving target, and the point with a gray value of 0 is the background point.

[0117] S5.2: For image R' i Perform connected component analysis to obtain the image R containing the complete moving target i , according to R i Get the bounding box coordinates of the target;

[0118] S6: Use the Kalman filter to predict the position of the bobsleigh in the next frame of the image;

[0119] S7: Selecting a true target from a number of candidate targets and updating the Kalman filter; including the following steps:

[0120] S7.1: If there are multiple candidate targets near the bobsleigh position predicted by the Kalman filter in the next frame, identify the true target based on its area and proximity to the predicted position. Obtain the bounding box coordinates of all candidate targets and calculate their areas.

[0121] S7.2: Gaussian weighting is performed on the areas of all candidate targets, as shown in formula (6):

[0122]

[0123] Among them, (c x ,c y ) is the center point coordinate of the target, σ c The standard deviation is obtained based on the target border area. The larger the target border area, the larger the Gaussian kernel range of the target, and vice versa. The Gaussian weighted result shows that the candidate target close to the predicted target has a higher weight.

[0124] S7.3: Select the target with the largest weighted area as the true target in the current frame image and update the Kalman filter prediction value;

[0125] S8: Smoothing the trajectory coordinates to obtain the 2D coordinates of the sliding trajectory of the bobsleigh under each camera; comprising the following steps:

[0126] S8.1: Use Kalman filter to remove noise points in trajectory coordinate data;

[0127] S8.2: Read the denoised trajectory coordinate data and perform smoothing. Select the bobsleigh coordinate points corresponding to five consecutive frames. Calculate the average of the horizontal and vertical coordinates of the five bobsleigh coordinate points to obtain the smoothed intermediate frame bobsleigh coordinates.

[0128] S8.3: Select five bobsleigh and sled coordinate points in sequence for denoising and smoothing until all points on the trajectory are traversed;

[0129] S8.4: Finally, the smoothed trajectory coordinate data results are combined to obtain the complete smoothed trajectory coordinates;

[0130] S9: Convert all 2D sliding track segments into 3D tracks according to the track surface 2D-3D coordinate mapping table of each camera, e.g. Figure 5 As shown in FIG, obtaining the 3D trajectory of the entire sliding process includes the following steps:

[0131] S9.1: Read the smoothed 2D coordinates of the trajectory to obtain the center coordinates of the moving target;

[0132] S9.2: According to the coordinates of the center point of the moving target, find its corresponding 3D coordinates in the 2D-3D coordinate mapping table and save the corresponding data into a txt file.

Claims

1. A method for extracting three-dimensional sliding trajectories of bobsleighs and sleds based on video and point cloud data, characterized by: The following steps are involved: S1: Acquire 3D point cloud data of the track and use multiple cameras to shoot videos of the bobsleigh and luge sliding on the track. Combine the videos shot by multiple cameras into a video recording the sliding process of the bobsleigh and luge based on the synchronized time information; S2: Video data preprocessing: dividing the video recording the bobsleigh and sled sliding process into several video segments, each corresponding to a camera, and further dividing the video segment into several images by frame; S3: calibrating all cameras to estimate their parameters and generating a 2D-3D coordinate mapping table of the track surface corresponding to each camera; S4: Fitting the curve expressions of the edges on both sides of the track for each camera’s lens scene; S5: Detect the position of the bobsleigh in the current frame image using a background difference algorithm; S6: Use the Kalman filter to predict the position of the bobsleigh in the next frame of the image; S7: Select the real target from several candidate targets and update the Kalman filter; S8: Perform smoothing post-processing on the 2D trajectory coordinates to obtain the 2D coordinates of the sliding trajectory of the bobsleigh under each monitoring camera; S9: Convert all 2D sliding trajectory segments into 3D trajectories according to the track surface 2D-3D coordinate mapping table of each monitoring camera to obtain the 3D trajectory of the sliding process.

2. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, characterized in that: In S2, the video data preprocessing specifically includes the following steps: S2.1: Automatically segment the bobsleigh and sled sliding process video into video segments corresponding to different cameras based on scene detection. S2.2: Read the time code to split the entire bobsleigh and skeleton sliding video into video segments corresponding to different cameras; S2.3: After obtaining the segmented video segments, further segment the video segments into a plurality of frame images.

3. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, wherein: In S3, the process of camera parameter estimation is as follows: S3.1: Select several corner points on the image and use Cyclone software to extract the corresponding 3D points in the 3D point cloud data to obtain the set of 2D-3D key point coordinate pairs under the current camera and save them to a file; S3.2: Assume that the image pixel coordinate system is The image plane coordinate system is The image space coordinate system is The Cartesian coordinate system is The transformation relationship between the pixel coordinate system and the Cartesian space coordinate system is shown in formula (1): From formula (1), we can get: Among them, l1,l2,…,l 12 is the relationship coefficient between the Cartesian space coordinate system and the pixel coordinate system, let l 12 =1, find the parameter l 11 -l 11 ; Transform formula (1) into formula (2): Among them, the parameter l1-l 11 is an unknown number. Substituting each 2D-3D point pair into equation (2) yields two equations. At least six 2D-3D point pairs are required to solve equation (2). make L=[l1 l2 l3 l4 l5 l6 l7l8 l9 l 10 l 11 ] T , C=[uv] T , Equation (2) is recorded as BL-C=0, then the analysis of the unknown matrix L is shown in Equation (3) L=(B T B) -1 (B T C) (3) The value of L is obtained by the least square method; S3.3: For each camera, save the parameters of mapping the 3D points to the 2D image through steps S3.1-S3.

2.

4. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 3, characterized in that: The number of the selected corner points is greater than or equal to 6.

5. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, characterized in that: In S3, the process of generating the 2D-3D coordinate mapping table includes the following steps: SS3.1: Extract a set of 3D points from the track surface from the 3D point cloud data. This serves as a constraint for mapping the 2D trajectory to 3D. The track surface data for each camera lens scene is then exported separately. SS3.2: Read the 3D points on the track surface beneath each camera. Each 3D point is represented by its (x, y, z) coordinates. Based on the estimated camera parameters, calculate the corresponding 2D pixel coordinates (u, v). Decimal 2D coordinates are rounded. SS3.3: Establish a (W, H, 3) matrix as a mapping table from 2D points to 3D points, where W and H are the width and height of any frame image of the current camera, respectively, and 3 represents the number of channels, which are used to store the coordinates of the 3D point corresponding to the current pixel position. When a 2D image point corresponds to multiple 3D points in the coordinates (u, v) of the 2D pixel point, the average value of the coordinates of the multiple 3D points is calculated and used as the 3D mapping result of the 2D point. If a 2D image point does not have a corresponding 3D point, the 3D coordinates corresponding to the current image point are estimated based on the 3D points corresponding to the nearby points on the image. With the 2D point as the center, a 7×7 range is framed, and the average value of the corresponding 3D coordinates of all 2D points with corresponding 3D coordinates in the range is calculated. The 3D average value is used as the corresponding point of the 2D point. If no point in the 7×7 range has a corresponding 3D coordinate, its 3D point is set to (0, 0, 0), and it is considered that the 2D image point has no corresponding 3D point.

6. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, wherein: In S4, for each shot scene, the process of fitting the curve expressions of the edges on both sides of the track is as follows: S4.1: In any frame of the scene, evenly sample points along the edges of the two tracks; S4.2: The track edge is fitted with multiple polynomials. The polynomial coefficients of the curves at both edges of the track are estimated using the least squares method. The polynomial coefficients are read to construct the curve expression.

7. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, characterized in that: In S5, the process of detecting the position of the bobsleigh in the image using the background difference algorithm is as follows: S5.1: Obtain background image frame B through Gaussian mixture model, current image frame is F i , the grayscale values ​​of the corresponding pixels in the background frame and the current frame are recorded as B(x,y) and F i (x, y), subtract the grayscale values ​​of the corresponding pixels of the two frames of images, and take their absolute values ​​to obtain the difference image D n , as shown in formula (4): D n (x,y)=|F i (x,y)-B(x,y)| (4) S5.1: Set the threshold T and perform binarization on each pixel to obtain the binary image R′ n , as shown in formula (5): Among them, the point with a gray value of 255 is the foreground point, indicating the moving target, and the point with a gray value of 0 is the background point. S5.2: For image R′ i Perform connected domain analysis to obtain the image R containing the complete moving target i , according to R i The bounding box coordinates of the target are obtained, and the targets outside the track are filtered out using the track edge curve fitted in S4.

8. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, wherein: In S7, selecting a real target from a number of candidate targets and updating the Kalman filter includes the following steps: S7.1: If there are multiple candidate targets near the bobsleigh position predicted by the Kalman filter in the next frame, identify the true target based on its area and proximity to the predicted position. Obtain the bounding box coordinates of all candidate targets and calculate their areas. S7.2: Gaussian weighting is performed on the areas of all candidate targets, as shown in formula (6): Among them, (c x ,c y ) is the center point coordinate of the target, σ c The standard deviation is obtained based on the target bounding box area. The larger the target bounding box area, the larger the Gaussian kernel range of the target, and vice versa. The result of Gaussian weighting is that candidate targets close to the predicted target receive higher weights. S7.3: Select the target with the largest weighted area as the true target in the current frame image and update the Kalman filter prediction value.

9. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, wherein: In S8, the 2D trajectory coordinate smoothing post-processing includes the following steps: S8.1: Use Kalman filter to remove noise points in trajectory coordinate data; S8.2: Read the denoised trajectory coordinate data and perform smoothing. Select n1 consecutive frames corresponding to bobsleigh coordinate points, where n1 is ≥ 5. Calculate the average of the horizontal and vertical coordinates of the n1 bobsleigh coordinate points, and use them as the smoothed intermediate frame bobsleigh coordinates. S8.3: De-noise and smooth the n1 bobsleigh and sled coordinate points centered on each frame until all points on the trajectory are traversed. S8.4: Finally, these local smoothing results are combined to obtain the smoothed 2D trajectory coordinates.

10. The method for extracting three-dimensional sliding trajectories of a bobsleigh and snowmobile based on video and point cloud data according to claim 1, wherein: In S9, the 2D coordinates of the motion trajectory are converted into 3D coordinates, including the following steps: S9.1: Read the smoothed 2D coordinates of the trajectory to obtain the center coordinates of the moving target; S9.2: According to the coordinates of the center point of the moving target, the corresponding 3D coordinates are searched in the 2D-3D coordinate mapping table, and the corresponding data is saved in a file.

Images