A method for recovering venue data based on competition videos

Abstract

The present invention provides a method for recovering field data based on match videos. Step 1: Based on the actual dimensions of the field for the sport to be calibrated, a standard field template is established. Step 2: Using a twin neural network training method, the standard field template established in Step 1 and random camera parameters are used to generate labeled training set images. Step 3: Using the trained neural network, field contour features are extracted and a camera-feature database is established. Step 4: After inputting the image to be calibrated, a scene data recovery algorithm is used to obtain the target image, thereby completing scene data recovery. This invention addresses the problem of multiple interference factors in visual methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and particularly relates to a method for recovering field data based on game videos. Background Art

[0002] Capturing athletes' training processes through video capture is a crucial tool for coaches to control training quality. Analyzing televised match videos is also a primary method for studying opponents. However, video images lacking scale information provide limited motion data, making large-scale quantitative analysis difficult. Recovering the field data captured during video capture can provide a basis for obtaining motion trajectory data, enabling video digitization and analysis. This is highly beneficial for motion analysis, especially for large amounts of third-party televised video. Furthermore, field data with scale information is fundamental for 3D reconstruction of sports scenes. Field data is typically obtained through camera calibration. For example, digitizing an ice rink relies on capturing images from multiple cameras with known parameters to perform 3D reconstruction of the rink. However, if one wishes to analyze an opponent's technical skills through match videos, this problem becomes a single camera calibration problem with unknown parameters. This requires recovering the planar coordinates of the ice rink from the video. This is challenging due to the limited features of the ice surface and the inherent self-calibration nature of the rink, which lacks specific landmarks (such as calibration plates). Summary of the Invention

[0003] The present invention provides a field data recovery method based on game videos, which is used to solve the problem that methods using vision are subject to many interference factors.

[0004] The present invention is achieved through the following technical solutions:

[0005] A method for recovering field data based on a game video, the method comprising the following steps:

[0006] Step 1: Create a standard venue template based on the actual dimensions of the venue according to the sports event that needs to be calibrated;

[0007] Step 2: Generate labeled training set images using the twin neural network training method based on the site standard template established in step 1 and random camera parameters;

[0008] Step 3: Use the trained neural network to extract the site edge contour feature image and establish a camera-feature database;

[0009] Step 4: After inputting the image to be calibrated into the camera-feature database established in step 3, the scene data recovery algorithm is used to obtain the target image to complete the scene data recovery.

[0010] Furthermore, the training of the twin neural network in step 2 is used to extract features of the edge map, and the twin neural network is composed of two identical convolutional neural networks (CNN).

[0011] Furthermore, the step 2 is specifically as follows:

[0012] Training a twin neural network based on the standard template of the venue and random camera parameters,

[0013] The twin neural network loss function:

[0014] L(w,x1,x2,y)=yDw(x1,x2)+(1-y)max(0,m-Dw(x1,x2))

[0015] in

[0016] CNN input: edge image 320×180; CNN output: 16-dimensional features.

[0017] Furthermore, the site edge contour feature image in step 3 is specifically:

[0018] The extraction of the site edge image is mainly achieved through two GAN networks;

[0019] The first GAN is used to distinguish the foreground and background of the scene and retain the foreground area of the image;

[0020] The second GAN is used to extract the required site contours from the foreground area for the next step.

[0021] Furthermore, the establishment of the camera-feature database in step 3 specifically includes the following steps:

[0022] Step 3.1: Input the standard template image of the site;

[0023] Step 3.2: Input the camera-feature database size N;

[0024] Step 3.3: The camera-feature database is empty;

[0025] Step 3.4: Determine whether i is less than N. If it is less than N, go to step 3.10. If it is not less than N, go to step 3.5, where i is the loop count.

[0026] Step 3.5: Randomly generate camera parameters within a certain range;

[0027] Step 3.6: Calculate the homography matrix M based on the camera parameters camera in step 3.5;

[0028] Step 3.7: Convolve the homography matrix M from step 3.6 with the standard template image of the venue to obtain the edge map of the venue from the camera's perspective;

[0029] Step 3.8: Use Siamese-Network to extract the features of the site edge map

[0030] Step 3.9: Add the camera parameter camera from step 3.5 and the feature featur from step 3.8 to the camera-feature database, and determine whether i is less than N and return to step 3.4.

[0031] Step 3.10: The camera-feature is persisted to the database.

[0032] Furthermore, the certain range in step 3.5 refers to the focal length fl range [250, 450];

[0033] Pitch angle range [-75°, -50°];

[0034] Yaw angle range [-35°, 35°];

[0035] The world coordinates and roll angle are fixed values, the world coordinates are (300, 450, 150), and the roll angle is 0°.

[0036] Furthermore, the scene data recovery algorithm in step 4 specifically includes the following steps:

[0037] Step 4.1: Input the source image and use the trained GAN network to distinguish the foreground and background and extract the edge-image of the corresponding site;

[0038] Step 4.2: Use the trained twin neural network to extract the features of the edge-image of the site in step 4.1;

[0039] Step 4.3: Perform KNN nearest neighbor matching on the feature in step 4.2 and the training set; where the sample in the training set is the feature and the label is the camera parameter;

[0040] Step 4.4: Obtain the camera parameters θ corresponding to the site edge map based on the nearest neighbor matching in step 4.3;

[0041] Step 4.5: Use the camera parameters θ from step 4.4 to find the projection matrix M from the site plane to the edge map;

[0042] Step 4.6: Apply the inverse matrix of the projection matrix to the source image in step 4.1 to obtain the projection of the source image on the venue plane, thus completing the restoration of the video venue data.

[0043] Furthermore, the step 4.3 of creating the training set for KNN specifically includes first inputting the site template image and the required database size N, and then generating a set of features and camera parameters each time as the sample and label of the training set respectively;

[0044] A pinhole camera model is used to convert the world coordinate system to the image coordinate system. The parameters include the camera's three-dimensional position cc in the world coordinate system, the pitch angle of the camera coordinate system, the roll angle of the camera coordinate system, the yaw angle of the camera coordinate system, the focal length fl of the camera, and the translation u and v from the camera coordinate system to the image coordinate system.

[0045] The parameters to be calibrated are recorded as 9-dimensional vectors u, v, fl, cc[0], cc[1], cc[2], roll, pitch, yaw;

[0046] When randomly generating camera parameters, the values of the 9-dimensional vector will change for different competition events. The 9-dimensional vector is used to extract the features of the venue template image generated in the previous step using a twin neural network, and a one-to-one correspondence is established between the 9-dimensional vector and the camera parameters. The data is then stored in the database to complete the creation of the KNN training set.

[0047] The beneficial effects of the present invention are:

[0048] The present invention does not require any prerequisites such as calibration plates in the video frames, but is based on a game video shot by a PTZ camera with no fixed source, and directly restores scene data information from the video.

[0049] The venues for various sports in the present invention have specific venue features, which are usually reflected in the form of points and lines. By making full use of these venue feature information, the data recovery effect can be significantly improved.

[0050] The present invention can avoid interference factors in existing visual methods, such as venue background (audience seats), athletes, motion blur, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Attachment Figure 1 It is a schematic diagram of input on a football field video of the present invention.

[0052] Attachment Figure 2 It is a schematic diagram of output on a football field video of the present invention.

[0053] Attachment Figure 3 Schematic diagram of the input of the GAN network of the present invention.

[0054] Attachment Figure 4 It is a schematic diagram of the foreground and background mask results of the present invention.

[0055] Attachment Figure 5 Schematic diagram of feature point extraction of the present invention.

[0056] Attachment Figure 6 It is a flow chart of the method of the present invention.

[0057] Attachment Figure 7 It is a flow chart of creating a camera parameter-feature database of the present invention.

[0058] Attachment Figure 8 It is a flow chart of the scene data recovery algorithm of the present invention.

[0059] Attachment Figure 9 It is a schematic diagram of the short track speed skating venue template of the present invention.

[0060] Attachment Figure 10 It is the template image I in the camera coordinate system of the present invention.

[0061] Attachment Figure 11 It is the template image II in the camera coordinate system of the present invention. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] The goal of sports field data recovery is to establish a mapping relationship between video images and real-world field coordinates. The challenge lies in matching the characteristics of the field in the video with those of the real-world field. Field data recovery is achieved by calibrating camera parameters from video captured by a single PTZ camera and restoring field data.

[0064] A method for recovering field data based on a game video, the method comprising the following steps:

[0065] Step 1: Create a standard venue template based on the actual dimensions of the venue according to the sports event that needs to be calibrated;

[0066] Step 2: Generate labeled training set images using the twin neural network training method based on the site standard template established in step 1 and random camera parameters;

[0067] Step 3: Use the trained neural network to extract the site edge contour feature image and establish a camera-feature database;

[0068] Step 4: After inputting the image to be calibrated into the camera-feature database established in step 3, the scene data recovery algorithm is used to obtain the target image to complete the scene data recovery.

[0069] The overall implementation plan is as follows Figure 6 First, we need to establish a standard template based on the actual size of the corresponding venue for the sports event that needs to be calibrated, such as Figure 9 The figure shows the short track speed skating venue template. Then, the twin neural network training method is used to generate training set images with short track speed skating venue labels based on the venue template and random camera parameters. Some images are shown in the figure. Figure 10 、 Figure 11 As shown in the figure, a neural network capable of extracting site contour features is trained; then the neural network is used to extract the site contour features. Figure 7 The database establishment method shown in the figure is to establish a camera-feature database; after inputting the image to be calibrated, Figure 8 By using the method shown, the target image is obtained and the scene data restoration can be completed.

[0070] Furthermore, the training of the twin neural network in step 2 is used to extract features of the edge map, and the twin neural network is composed of two identical convolutional neural networks (CNN).

[0071] Furthermore, the step 2 is specifically as follows:

[0072] Training a twin neural network based on the standard template of the venue and random camera parameters,

[0073] The twin neural network loss function:

[0074] L(w,x1,x2,y)=yDw(x1,x2)+(1-y)max(0,m-Dw(x1,x2))

[0075] in

[0076] CNN input: edge image 320×180; CNN output: 16-dimensional features.

[0077] Furthermore, the site edge contour feature image in step 3 is specifically:

[0078] The extraction of the site edge image is mainly achieved through two GAN networks;

[0079] The first GAN is used to distinguish the foreground and background of the scene and retain the foreground area of the image;

[0080] The second GAN is used to extract the required site contours from the foreground area for the next step.

[0081] The input and output of the GAN network are as follows Figure 3 Figure 4 Figure 5 As shown, the foreground (ice surface) and background (audience seats) are automatically distinguished from the video frame, and the feature point information of the venue is extracted after masking, such as the curve markings of the ice rink.

[0082] Furthermore, the establishment of the camera-feature database in step 3 specifically includes the following steps:

[0083] Step 3.1: Input the standard template image of the site;

[0084] Step 3.2: Input the camera-feature database size N;

[0085] Step 3.3: The camera-feature database is empty;

[0086] Step 3.4: Determine whether i is less than N. If it is less than N, go to step 3.10. If it is not less than N, go to step 3.5, where i is the loop count.

[0087] Step 3.5: Randomly generate camera parameters within a certain range;

[0088] Step 3.6: Calculate the homography matrix M based on the camera parameters camera in step 3.5;

[0089] Step 3.7: Convolve the homography matrix M from step 3.6 with the standard template image of the venue to obtain the edge map of the venue from the camera's perspective;

[0090] Step 3.8: Use Siamese-Network to extract the features of the site edge map

[0091] Step 3.9: Add the camera parameter camera from step 3.5 and the feature featur from step 3.8 to the camera-feature database, and determine whether i is less than N and return to step 3.4.

[0092] Step 3.10: Persist the camera-feature to the database. The camera-feature results are saved as database files on disk, rather than just being cleared when the memory is freed after the runtime calculation.

[0093] Furthermore, the scene data recovery algorithm in step 4 specifically includes the following steps:

[0094] Step 4.1: Input the source image and use the trained GAN network to distinguish the foreground and background and extract the edge-image of the corresponding site;

[0095] Step 4.2: Use the trained twin neural network to extract the features of the edge-image of the site in step 4.1;

[0096] Step 4.3: Perform KNN nearest neighbor matching on the feature in step 4.2 and the training set; where the sample in the training set is the feature and the label is the camera parameter;

[0097] Step 4.4: Obtain the camera parameters θ corresponding to the site edge map based on the nearest neighbor matching in step 4.3;

[0098] Step 4.5: Use the camera parameters θ from step 4.4 to find the projection matrix M from the site plane to the edge map;

[0099] Step 4.6: Apply the inverse matrix of the projection matrix to the source image in step 4.1 to obtain the projection of the source image on the venue plane, thus completing the restoration of the video venue data.

[0100] Furthermore, the creation of the training set for KNN in step 4.3 is as follows: the effect of KNN training depends on the degree of fit between the training set and reality, so it is necessary to generate a training sample set that is as close as possible to the real game scene.

[0101] The process of establishing the Feature-camera database is as follows Figure 7 As shown, first input the site template image and the required database size N, and then generate a set of features and camera parameters each time as the sample and label of the training set respectively;

[0102] A pinhole camera model is used to convert the world coordinate system to the image coordinate system. The parameters include the camera's three-dimensional position cc in the world coordinate system, the pitch angle of the camera coordinate system, the roll angle of the camera coordinate system, the yaw angle of the camera coordinate system, the focal length fl of the camera, and the translation u and v from the camera coordinate system to the image coordinate system.

[0103] The parameters to be calibrated are recorded as 9-dimensional vectors u, v, fl, cc[0], cc[1], cc[2], roll, pitch, yaw;

[0104] When randomly generating camera parameters, the values of the 9-dimensional vector will change for different competition events. The 9-dimensional vector is used to extract the features of the venue template image generated in the previous step using a twin neural network, and a one-to-one correspondence is established between the 9-dimensional vector and the camera parameters. The data is then stored in the database to complete the creation of the KNN training set.

[0105] Different strategies are used for different events. For example, a standard short-track speed skating venue is scaled to 600 pixels long and 300 pixels wide. Based on the correspondence between the camera coordinate system and the image coordinate system, u = 300 and v = 150. The camera is typically suspended high to one side of the center of the venue, so cc(xyz) = (300, 450, 150). The focal length varies with the athlete's position, ranging from (250, 450). In actual competitions, camera roll rarely occurs, with only pitch and yaw variations. Therefore, roll = 0, pitch = (50, 75), and yaw = (-35, 35) (unit: degrees) are set. Randomly selecting values within this range can generate a template image of the venue with angles similar to those in the actual competition scene.

[0106] Figure 1 、 Figure 2 This is an example of the system's input and output on a video of a soccer field. It can accurately restore the real-world coordinates of the field and the players in the video.

Claims

1. A method for recovering field data based on game videos, characterized in that: The site data recovery method comprises the following steps: Step 1: Create a standard venue template based on the actual dimensions of the venue according to the sports event that needs to be calibrated; Step 2: Generate training set images based on the site standard template and random camera parameters established in step 1 and train the twin neural network; Step 3: Use the trained twin neural network to extract the edge contour feature image of the site and establish a camera-feature database; Step 4: After inputting the image to be calibrated into the camera-feature database established in step 3, the scene data recovery algorithm is used to obtain the target image to complete the scene data recovery; The scene data recovery algorithm in step 4 specifically includes the following steps: Step 4.1: Input the source image and use the trained GAN network to distinguish the foreground and background and extract the edge-image of the corresponding site; Step 4.2: Use the trained twin neural network to extract the features of the edge-image of the site in step 4.1; Step 4.3: Perform KNN nearest neighbor matching on the features from step 4.2 and the training set. Input the site template image and the required camera-feature database size N. Each time, a set of features and camera parameters are generated as the sample and label of the training set, respectively. The sample is the feature, and the label is the camera parameter. Step 4.4: Obtain the camera parameters θ corresponding to the site edge map based on the nearest neighbor matching in step 4.3; Step 4.5: Use the camera parameters θ from step 4.4 to obtain the projection matrix M from the site plane to the edge map; Step 4.6: Apply the inverse matrix of the projection matrix to the source image in step 4.1 to obtain the projection of the source image on the venue plane, thus completing the restoration of the video venue data.

2. The method for recovering field data based on a match video according to claim 1, characterized in that: The training of the twin neural network in step 2 is used to extract features of the edge map, and the twin neural network is composed of two identical convolutional neural networks (CNN).

3. The method for recovering field data based on a match video according to claim 1, characterized in that: The site edge contour feature image in step 3 is specifically: The extraction of the site edge image is mainly achieved through two GAN networks; The first GAN is used to distinguish the foreground and background of the scene and retain the foreground area of the image; The second GAN is used to extract the required site contours from the foreground area for the next step.

4. The method for recovering field data based on a match video according to claim 1, characterized in that: The establishment of the camera-feature database in step 3 specifically includes the following steps: Step 3.1: Input the standard template image of the site; Step 3.2: Input the camera-feature database size N; Step 3.3: The camera-feature database is empty; Step 3.4: Determine whether i is less than N. If it is less than N, go to step 3.

10. If it is not less than N, go to step 3.5, where i is the loop count. Step 3.5: Randomly generate camera parameters camera within a certain range; where the certain range refers to the focal length fl range [250,450]; Pitch angle range [-75°, -50°]; Yaw angle range [-35°, 35°]; The world coordinates and roll angle are fixed values, the world coordinates are (300, 450, 150), and the roll angle is 0°; Step 3.6: Calculate the homography matrix M based on the camera parameters camera in step 3.5; Step 3.7: Convolve the homography matrix M from step 3.6 with the standard template image of the venue to obtain the edge map of the venue from the camera's perspective; Step 3.8: Use Siamese-Network to extract the features of the site edge map; Step 3.9: Add the camera parameter camera from step 3.5 and the feature featur from step 3.8 to the camera-feature database, and determine whether i is less than N and return to step 3.

4. Step 3.10: The camera-feature is persisted to the database.

5. The method for recovering field data based on a match video according to claim 1, characterized in that: Specifically, the KNN nearest neighbor matching of the training set in step 4.3 is performed by using a pinhole camera model, from the world coordinate system to the image coordinate system, and the parameters include the three-dimensional position cc of the camera in the world coordinate system, the pitch angle of the camera coordinate system, the roll angle of the camera coordinate system, the yaw angle of the camera coordinate system, the focal length fl of the camera, and the translation amounts u and v from the camera coordinate system to the image coordinate system; The parameters to be calibrated are recorded as 9-dimensional vectors u, v, fl, cc[0], cc[1], cc[2], roll, pitch, yaw; When randomly generating camera parameters, the values of the 9-dimensional vector will change for different competition events. The 9-dimensional vector is used to extract the features of the venue template image generated in the previous step using a twin neural network, and a one-to-one correspondence is established between the 9-dimensional vector and the camera parameters. The data is then stored in the database to complete the creation of the KNN training set.

Images