Curling ball motion state detection method, system, computer device and storage medium for broadcast video

Through semantic segmentation and object detection technology, the broadcast video of curling ball game is converted from a slash view to a top view, and combined with Hough transformed straight line fitting, the problem of difficulty in extracting curling ball information is solved and curling ball is realized Accurate detection of status.

CN115512262BActive Publication Date: 2024-11-19HARBIN INST OF TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211150368.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-21
Publication Date
2024-11-19
Estimated Expiration
2042-09-21

AI Technical Summary

Technical Problem

The prior art is difficult to extract the movement information of the curling ball from the curling ball game broadcast video. It is mainly because the parameters of the game broadcast camera are unknown and the viewing angle is frequently switched, which makes it difficult to extract information.

Method used

Through semantic segmentation technology, the curling track coordinate system in the broadcast video is converted into a top view angle, and the curling ball target detection network and handle angle detection model are established. The curling ball is obtained by combining YOLOX and DeepLabV3+ networks with Hough transform linear fitting to obtain the curling ball position, speed and rotation angular velocity.

Benefits of technology

It accurately extracts the movement status information of the curling ball from the broadcast video, improves the accuracy and standardization of the recovery process of the curling ball, and is suitable for the movement status detection of the curling game.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115512262B_ABST
    Figure CN115512262B_ABST
Patent Text Reader

Abstract

The invention relates to a curling ball motion state detection method, system, computer equipment and storage medium for broadcast video, which belongs to the field of artificial intelligence detection technology and solves the problem that it is difficult to extract motion information of curling balls from broadcast videos. The method of the invention comprises: semantically segmenting the curling track in the broadcast video, calculating the coordinate transformation matrix between the broadcast video plane and the curling track plane based on the size prior information in the curling track, and transforming the curling game broadcast video from an oblique perspective to a top-down perspective; then training the curling ball target detection network, designing the data set annotation principle, automatically amplifying the data set through the data enhancement method, obtaining the detection frame and position information of the curling ball, and then obtaining the handle angle information describing the rotation of the curling ball based on the image segmentation algorithm and the Hough transform straight line fitting method, thereby realizing the motion state detection of the curling ball in the broadcast video. The invention is suitable for the motion state detection of the curling ball in the broadcast video of the curling game.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence detection technology, and in particular to the detection of the motion state of a curling ball in a curling game broadcast video. Background Art

[0002] Curling game broadcast videos contain a large amount of professional information. Extracting the movement information of the curling ball from the broadcast videos can provide expert samples for training strategies, and quickly and accurately extract battle information from important broadcast events. It has important reference value for analyzing the throwing strategies of curling games, extracting expert experience, and developing curling robots.

[0003] However, due to the unknown parameters of the camera used to broadcast the game and the need to switch perspectives between multiple cameras for better broadcast effects, it is difficult to extract the motion information of the curling ball from the broadcast video. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that it is difficult to extract motion information of a curling ball from a broadcast video of a curling game, and to provide a curling ball motion state detection method, system, computer equipment and storage medium for broadcast video.

[0005] The present invention is implemented by the following technical solutions. On one hand, the present invention provides a curling ball motion state detection method for broadcast video, the method comprising:

[0006] Step 1: semantically segment the curling track in the broadcast video, calculate the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system based on the prior information of the size of the curling track, and transform the broadcast video from an oblique perspective to a bird's-eye view;

[0007] Step 2: Establish a curling ball target detection network, including:

[0008] Collect curling ball broadcast videos, convert the curling ball broadcast videos using the method in step 1, and establish a curling ball target detection training data set through a data enhancement method; based on the curling ball target detection training data set, establish a curling ball target detection network using a YOLOX network, and the output of the curling ball target detection network is a curling ball detection frame;

[0009] Step 3: Establish a curling ball handle angle detection model, including:

[0010] Based on the curling ball detection frame output by the curling ball target detection network in step 2, a curling ball detection frame dataset is constructed; based on the curling ball detection frame dataset, a curling ball handle angle detection model is established by using the DeepLabV3+ semantic segmentation network and the Hough transform straight line fitting method, wherein the output of the curling ball handle angle detection model is the curling ball handle angle;

[0011] Step 4, obtaining a curling match broadcast video to be detected, and converting the curling match broadcast video to be detected using the method of step 1;

[0012] Using the converted curling game broadcast video to be detected and the curling ball target detection network to obtain a curling ball detection frame to be detected, and obtaining the position and speed of the curling ball to be detected according to the curling ball detection frame;

[0013] Step 5: According to the curling ball detection frame to be detected and using the curling ball handle angle detection model, the handle angle of the curling ball to be detected is obtained, and the rotational angular velocity of the curling ball to be detected is obtained according to the handle angle of the curling ball to be detected.

[0014] Furthermore, step 1 specifically includes:

[0015] Step 1.1, semantically segment the curling track in the broadcast video, and the semantically segmented image annotation includes 8 categories, specifically including: blank track, vanguard line, blue ring base camp, white ring base camp, red ring base camp, white circle base camp, scoreboard and logo;

[0016] Step 1.2, segment the curling track based on the DeepLabV3+ semantic segmentation network, specifically: perform dilated convolution and dilated spatial pyramid pooling on the broadcast video, set the initial learning rate, use the exponential transformation attenuation strategy to dynamically adjust the learning rate during training, and set the objective function to the intersection loss function. Based on the 8-category image annotation in step 1.1, the semantic segmentation result of the curling track image in the broadcast video is obtained;

[0017] Step 1.3: Based on the semantic segmentation results obtained in step 1.2, a blank track, a blue ring base camp, and a white ring base camp are segmented from the broadcast video, and the analytical expressions of the left boundary line and the right boundary line of the blank track, as well as the analytical expressions of the ellipse corresponding to the inner and outer contours of the blue ring are obtained;

[0018] Step 1.4, using the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipse corresponding to the inner and outer contours of the blue ring, and based on the prior information of the curling track size, calculate the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system, and transform the broadcast video from a squint perspective to a top-down perspective.

[0019] Furthermore, the loss function F is specifically:

[0020]

[0021] Among them, y is the real sample, y′ is the predicted category probability given by the model, γ is the loss factor, and α is the balance factor.

[0022] Furthermore, step 1.3 specifically includes:

[0023] Step 1.3.1, set the pixel positions corresponding to the blank track, blue ring base camp, and white ring base camp to 1, and the other positions to 0, and obtain mask images respectively;

[0024] Step 1.3.2, if there is a 0 pixel in the 8-connected domain or 4-connected domain of a pixel in the mask image, then this pixel is a boundary point, and multiple contour point sets of each connected area of ​​the mask image are obtained. Then the area corresponding to each contour point set is calculated, and the contour with the largest area is selected as the edge contour;

[0025] Step 1.3.3, the coordinates of each point on the contour line in the curling track coordinate system are (xi, yi), xi is used as the independent variable and yi is used as the output for low-pass filtering, the absolute value of the difference between the original value and the filtered value of yi is used as the difference, and the points whose difference exceeds a preset number of pixels are deleted;

[0026] Step 1.3.4: Based on the contour point set after filtering, the least square method is used to fit the contour point set to obtain the analytical expressions of the left boundary line and the right boundary line of the blank track, as well as the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring.

[0027] Furthermore, step 1.4 specifically includes:

[0028] Step 1.4.1, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the intersection point A of the outer contour tangent line of the blue ring and the left boundary line, the intersection point B of the outer contour tangent line of the blue ring and the right boundary line, the intersection point C of the inner contour tangent line of the blue ring and the left boundary line, and the intersection point D of the inner contour tangent line of the blue ring and the right boundary line in the broadcast video coordinate system, and the outer contour tangent line of the blue ring and the inner contour tangent line of the blue ring are both parallel to the T line;

[0029] Then, according to the prior information of the curling track size, the coordinates of the intersection points A, B, C and D in the curling track coordinate system are obtained to obtain a first homography matrix;

[0030] Step 1.4.2, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the tangent point E of the outer contour tangent of the blue ring and the tangent point F of the inner contour tangent of the blue ring in the broadcast video coordinate system;

[0031] Then, the coordinates of the tangent point E and the tangent point F in the curling track coordinate system are obtained according to the prior information of the curling track size, and a second homography matrix is ​​obtained;

[0032] Step 1.4.3, multiplying the first homography matrix and the second homography matrix to obtain a coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system;

[0033] Step 1.4.4: multiply the broadcast video by the coordinate transformation matrix to obtain a bird's-eye view image, and transform the broadcast video from a squint view to a bird's-eye view.

[0034] Furthermore, step 2 specifically includes:

[0035] Step 2.1, collect a number of curling ball images from broadcast videos with curling balls at different angles, positions, occlusions and different backgrounds, convert the curling ball images using the method in step 1, randomly scale each preset number of converted images, and then splice and amplify them according to random distribution, and mark the curling balls in the converted images according to color categories to obtain a curling ball target detection training data set;

[0036] Step 2.2, use the YOLOX network to establish a curling ball target detection network, the input of the curling ball target detection network is the curling ball target detection training data set, and the output is the curling ball detection frame; set the initial learning rate of the curling ball target detection network, and adjust the learning rate at each gradient descent by the exponential transformation attenuation strategy, set the training cycle, and train the network using small batch gradient descent.

[0037] Furthermore, step 3 specifically includes:

[0038] Step 3.1: Based on the curling ball detection frame output by the curling ball target detection network in step 2, obtain several curling ball detection frame images, use the bilinear interpolation method to unify the several curling ball detection frame images to the same size, and mark the handle on the curling ball to construct a curling ball detection frame dataset.

[0039] Step 3.2: Based on the curling ball detection frame dataset, a curling ball handle angle detection model is established using the DeepLabV3+ semantic segmentation network and the Hough transform straight line fitting method. The output of the curling ball handle angle detection model is the curling ball handle angle, which specifically includes:

[0040] Based on the curling ball detection frame dataset, a curling ball handle image is obtained using a DeepLabV3+ semantic segmentation network;

[0041] The curling ball handle image is subjected to Hough transform by using the Hough transform straight line fitting method to obtain a set of straight lines, which are discretized at equal intervals according to the straight line angles with an accuracy of 1 degree. Several discrete points are overlapped and fitted into a straight line, and the slope of the fitted straight line is converted into an angle, and the angle mean is taken as the curling ball handle angle.

[0042] In a second aspect, the present invention provides a curling ball motion state detection system for broadcast video, the system comprising:

[0043] A coordinate conversion module is used to perform semantic segmentation on the curling track in the broadcast video. Based on the prior information of the size of the curling track, a coordinate conversion matrix between the broadcast video coordinate system and the curling track coordinate system is calculated to convert the broadcast video from an oblique perspective to a bird's-eye view.

[0044] The curling ball target detection network establishment module is used to establish the curling ball target detection network, specifically including:

[0045] An image acquisition and conversion module is used to acquire a curling ball broadcast video, convert the curling ball broadcast video using a coordinate conversion module, and establish a curling ball target detection training data set through a data enhancement method;

[0046] A network building module, used for building a curling ball target detection network based on the curling ball target detection training data set and using a YOLOX network, wherein the output of the curling ball target detection network is a curling ball detection frame;

[0047] A curling ball handle angle acquisition module is used to construct a curling ball detection frame dataset based on the curling ball detection frame output by the curling ball target detection network in the curling ball target detection network establishment module; based on the curling ball detection frame dataset, and using the DeepLabV3+ semantic segmentation network and Hough transform straight line fitting method, the curling ball handle angle is acquired;

[0048] A curling ball position and speed acquisition module, used to acquire a curling game broadcast video to be detected, and transform the curling game broadcast video to be detected using a coordinate conversion module;

[0049] Using the converted curling game broadcast video to be detected and the curling ball target detection network to obtain a curling ball detection frame to be detected, and obtaining the position and speed of the curling ball to be detected according to the curling ball detection frame;

[0050] The curling ball rotation angular velocity acquisition module is used to acquire the handle angle of the curling ball to be detected according to the curling ball detection frame to be detected and the curling ball handle angle detection model, and acquire the rotation angular velocity of the curling ball to be detected according to the curling ball handle angle to be detected.

[0051] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the steps of a method for detecting the motion state of a curling ball for broadcast video as described above are executed.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein a plurality of computer instructions are stored in the computer-readable storage medium, and the plurality of computer instructions are used to enable a computer to execute a method for detecting a curling ball motion state for a broadcast video as described above.

[0053] Beneficial effects of the present invention:

[0054] Firstly, the present invention utilizes artificial intelligence image segmentation and target detection technology, combined with coordinate system transformation, to obtain the position and rotation information of the curling ball in the broadcast video, and restore the motion process of the curling ball from the broadcast video;

[0055] Secondly, the present invention is aimed at the curling game broadcast video, and proposes a method for calculating the projection transformation matrix of the curling game broadcast video from an oblique perspective to a top-down perspective, so as to transform the pixel coordinates of the curling ball into a curling track coordinate system with practical significance;

[0056] Finally, based on the target detection algorithm, the position information of the curling ball in the curling track coordinate system was obtained. Based on the image segmentation algorithm and the Hough transform straight line fitting method, the handle angle information describing the rotation of the curling ball was obtained, realizing the motion state detection of the curling ball in the broadcast video.

[0057] The present invention is suitable for detecting the motion state of a curling ball in a curling game broadcast video. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] Figure 1 It is a flow chart of the method for obtaining the analytical formula according to the present invention;

[0060] Figure 2 A schematic diagram of straight lines AC and BD in a curling game broadcast video according to the method of the present invention;

[0061] Figure 3 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0062] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0063] Embodiment 1: A curling ball motion state detection method for broadcast video, the method comprising:

[0064] Step 1: semantically segment the curling track in the broadcast video, calculate the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system based on the prior information of the size of the curling track, and transform the broadcast video from an oblique perspective to a bird's-eye view;

[0065] Step 2: Establish a curling ball target detection network, including:

[0066] Collect curling ball broadcast videos, convert the curling ball broadcast videos using the method in step 1, and establish a curling ball target detection training data set through a data enhancement method; based on the curling ball target detection training data set, establish a curling ball target detection network using a YOLOX network, and the output of the curling ball target detection network is a curling ball detection frame;

[0067] Step 3: Establish a curling ball handle angle detection model, including:

[0068] Based on the curling ball detection frame output by the curling ball target detection network in step 2, a curling ball detection frame dataset is constructed; based on the curling ball detection frame dataset, a curling ball handle angle detection model is established by using the DeepLabV3+ semantic segmentation network and the Hough transform straight line fitting method, wherein the output of the curling ball handle angle detection model is the curling ball handle angle;

[0069] Step 4, obtaining a curling match broadcast video to be detected, and converting the curling match broadcast video to be detected using the method of step 1;

[0070] Using the converted curling game broadcast video to be detected and the curling ball target detection network to obtain a curling ball detection frame to be detected, and obtaining the position and speed of the curling ball to be detected according to the curling ball detection frame;

[0071] Step 5: According to the curling ball detection frame to be detected and using the curling ball handle angle detection model, the handle angle of the curling ball to be detected is obtained, and the rotational angular velocity of the curling ball to be detected is obtained according to the handle angle of the curling ball to be detected.

[0072] In this embodiment, firstly, the position and rotation information of the curling ball in the broadcast video can be obtained by using artificial intelligence image segmentation and target detection technology in combination with coordinate system transformation, and the motion process of the curling ball can be restored from the broadcast video;

[0073] Secondly, for the curling game broadcast video, a method for calculating the projection transformation matrix of the curling game broadcast video from the oblique perspective to the top-down perspective is proposed to transform the pixel coordinates of the curling ball into the curling track coordinate system with practical significance.

[0074] Finally, based on the target detection algorithm, the position information of the curling ball in the curling track coordinate system was obtained. Based on the image segmentation algorithm and the Hough transform straight line fitting method, the handle angle information describing the rotation of the curling ball was obtained, realizing the motion state detection of the curling ball in the broadcast video.

[0075] Implementation method 2: This implementation method further limits the curling ball motion state detection method for broadcast video described in implementation method 1. In this implementation method, step 1 is further limited, specifically including:

[0076] Step 1.1, semantically segment the curling track in the broadcast video, and the semantically segmented image annotation includes 8 categories, specifically including: blank track, vanguard line, blue ring base camp, white ring base camp, red ring base camp, white circle base camp, scoreboard and logo;

[0077] Step 1.2, segment the curling track based on the DeepLabV3+ semantic segmentation network, specifically: perform dilated convolution and dilated spatial pyramid pooling on the broadcast video, set the initial learning rate, use the exponential transformation attenuation strategy to dynamically adjust the learning rate during training, and set the objective function to the intersection loss function. Based on the 8-category image annotation in step 1.1, the semantic segmentation result of the curling track image in the broadcast video is obtained;

[0078] Step 1.3: Based on the semantic segmentation results obtained in step 1.2, the blank track, the blue ring base camp, and the white ring base camp are segmented from the broadcast video, and the left boundary line of the blank track and the right boundary line of the blank track (i.e. Figure 2 The analytical expressions of the blank track left boundary line AC and the blank track right boundary line BD), as well as the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring;

[0079] Step 1.4, using the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipse corresponding to the inner and outer contours of the blue ring, and based on the prior information of the curling track size, calculate the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system, and transform the broadcast video from a squint perspective to a top-down perspective.

[0080] This embodiment provides a specific method for transforming the broadcast video from an oblique perspective to a top-down perspective. This method can accurately convert the broadcast video into a uniform curling track coordinate system. The subsequent establishment of the curling ball target detection network and the curling ball handle angle detection model are both calculated based on the curling track coordinate system, thereby achieving improved accuracy and standardization in the extraction of curling ball motion information.

[0081] Implementation method 3: This implementation method further limits the curling ball motion state detection method for broadcast video described in implementation method 2. In this implementation method, the loss function is further limited, specifically including:

[0082] The loss function F is specifically:

[0083]

[0084] Among them, y is the real sample, y′ is the predicted category probability given by the model, γ is the loss factor, and α is the balance factor.

[0085] The loss function in this implementation can set a balance factor α=0.25 to solve the uneven ratio of the number of positive and negative samples between the blank track and other categories of data.

[0086] Implementation method 4: This implementation method further defines the curling ball motion state detection method for broadcast video described in any one of implementation methods 2 or 3. In this implementation method, step 1.3 is further defined, specifically including:

[0087] Step 1.3.1, set the pixel positions corresponding to the blank track, blue ring base camp, and white ring base camp to 1, and the other positions to 0, and obtain mask images respectively;

[0088] Step 1.3.2, if there is a 0 pixel in the 8-connected domain or 4-connected domain of a pixel in the mask image, then this pixel is a boundary point, and multiple contour point sets of each connected area of ​​the mask image are obtained. Then the area corresponding to each contour point set is calculated, and the contour with the largest area is selected as the edge contour;

[0089] Step 1.3.3, the coordinates of each point on the contour line in the curling track coordinate system are (xi, yi), xi is used as the independent variable and yi is used as the output for low-pass filtering, the absolute value of the difference between the original value and the filtered value of yi is used as the judgment basis, and the points whose difference (i.e., the absolute value of the difference between the original value and the filtered value of yi) exceeds a preset number of pixels are deleted;

[0090] Step 1.3.4: Based on the contour point set after filtering, the least square method is used to fit the contour point set to obtain the analytical expressions of the left boundary line and the right boundary line of the blank track, as well as the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring.

[0091] This embodiment provides an analytical expression for the left boundary line of the blank track and the right boundary line of the blank track, as well as a specific method for obtaining the analytical expression of the ellipse corresponding to the inner and outer contours of the blue ring. This method eliminates the need to place a calibration plate on the curling track, thereby improving the accuracy of extracting curling ball motion information.

[0092] Implementation mode 5: This implementation mode further limits the curling ball motion state detection method for broadcast video described in Implementation mode 2. In this implementation mode, step 1.4 is further limited, specifically including:

[0093] Step 1.4.1, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the intersection point A of the outer contour tangent line of the blue ring and the left boundary line, the intersection point B of the outer contour tangent line of the blue ring and the right boundary line, the intersection point C of the inner contour tangent line of the blue ring and the left boundary line, and the intersection point D of the inner contour tangent line of the blue ring and the right boundary line in the broadcast video coordinate system, and the outer contour tangent line of the blue ring and the inner contour tangent line of the blue ring are both parallel to the T line;

[0094] Then, according to the prior information of the curling track size, the coordinates of the intersection points A, B, C and D in the curling track coordinate system are obtained to obtain a first homography matrix;

[0095] Step 1.4.2, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the tangent point E of the outer contour tangent of the blue ring and the tangent point F of the inner contour tangent of the blue ring in the broadcast video coordinate system;

[0096] Then, the coordinates of the tangent point E and the tangent point F in the curling track coordinate system are obtained according to the prior information of the curling track size, and a second homography matrix is ​​obtained;

[0097] Step 1.4.3, multiplying the first homography matrix and the second homography matrix to obtain a coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system;

[0098] Step 1.4.4: multiply the broadcast video by the coordinate transformation matrix to obtain a bird's-eye view image, and transform the broadcast video from a squint view to a bird's-eye view.

[0099] It should be noted that the T line is the same width as the field, parallel to the front throw line and the back throw line, and passes through the center of the base camp, i.e. Figure 2 The bottom dotted line in the figure; the T line can also be replaced by a back line parallel to it.

[0100] This embodiment provides a specific method for transforming the broadcast video from an oblique perspective to a bird's-eye view. The advantage of this method is that it improves the accuracy of feature point extraction through curling field contour segmentation and size prior information, thereby improving the accuracy of curling ball motion information extraction.

[0101] Implementation method 6: This implementation method further limits the curling ball motion state detection method for broadcast video described in implementation method 1. In this implementation method, step 2 is further limited, specifically including:

[0102] Step 2.1, collect a number of curling ball images from broadcast videos with curling balls at different angles, positions, occlusions and different backgrounds, convert the curling ball images using the method in step 1, randomly scale each preset number of converted images, and then splice and amplify them according to random distribution, and mark the curling balls in the converted images according to color categories to obtain a curling ball target detection training data set;

[0103] Step 2.2, use the YOLOX network to establish a curling ball target detection network, the input of the curling ball target detection network is the curling ball target detection training data set, and the output is the curling ball detection frame; set the initial learning rate of the curling ball target detection network, and adjust the learning rate at each gradient descent by the exponential transformation attenuation strategy, set the training cycle, and train the network using small batch gradient descent.

[0104] The curling ball target detection network established using the method of this embodiment can accurately identify the motion state information of the curling ball (such as position and speed), and based on this information can also accurately identify the rotational angular velocity of the curling ball, so as to improve the accuracy of extracting the curling ball motion information.

[0105] Embodiment 7: This embodiment further limits the curling ball motion state detection method for broadcast video described in Embodiment 1. In this embodiment, step 3 is further limited, specifically including:

[0106] Step 3.1: Based on the curling ball detection frame output by the curling ball target detection network in step 2, obtain several curling ball detection frame images, use the bilinear interpolation method to unify the several curling ball detection frame images to the same size, and mark the handle on the curling ball to construct a curling ball detection frame dataset.

[0107] Step 3.2: Based on the curling ball detection frame dataset, a curling ball handle angle detection model is established using the DeepLabV3+ semantic segmentation network and the Hough transform straight line fitting method. The output of the curling ball handle angle detection model is the curling ball handle angle, which specifically includes:

[0108] Based on the curling ball detection frame dataset, a curling ball handle image is obtained using a DeepLabV3+ semantic segmentation network;

[0109] The curling ball handle image is subjected to Hough transform by using the Hough transform straight line fitting method to obtain a set of straight lines, which are discretized at equal intervals according to the straight line angles with an accuracy of 1 degree. Several discrete points are overlapped and fitted into a straight line, and the slope of the fitted straight line is converted into an angle, and the angle mean is taken as the curling ball handle angle.

[0110] It should be noted that in the above steps, several straight lines will be fitted, thereby obtaining several straight line slopes, and then the angles corresponding to the conversions of the several straight line slopes are averaged.

[0111] This embodiment provides a method for obtaining the angle of the handle of a curling ball. The advantage of this method is that the handle is separated from the curling ball to eliminate the influence of the background of the curling competition venue, thereby achieving the accuracy of obtaining the rotational angular velocity of the curling ball.

[0112] Embodiment 8: This embodiment is an example of the method of the present invention. Figure 3 As shown, specifically including:

[0113] Step 1: Perform semantic segmentation on the curling track in the broadcast video. Based on the prior information of the size of the curling track, calculate the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system, and transform the curling game broadcast video from an oblique perspective to a bird's-eye view.

[0114] The first step is to semantically segment and annotate the curling track in the broadcast video. The curling track in the broadcast video is semantically segmented and annotated into 8 categories: blank track, vanguard line, blue ring base camp, white ring base camp, red ring base camp, white circle base camp, scoreboard, and logo.

[0115] The second step is to segment the curling track based on the DeepLabV3+ semantic segmentation network. The curling match broadcast video is subjected to dilated convolution and dilated spatial pyramid pooling, the initial learning rate is set to 0.01, the exponential transformation decay strategy is used to dynamically adjust the learning rate during training, and the objective function is set to the intersection loss function F. Based on the 8-category image annotation in the first step, the semantic segmentation result of the curling track image of the curling match broadcast video is obtained.

[0116]

[0117] Where y is the real sample, and y′ is the predicted category probability given by the model. Set the loss factor γ = 2 to reduce the loss of easy-to-classify samples. Set the balance factor α = 0.25 to solve the uneven ratio of positive and negative samples between blank tracks and other categories of data.

[0118] The third step is to extract the contour information after semantic segmentation of the curling track. Based on the semantic segmentation results obtained by the DeepLabV3+ network in the second step, the blank track, the blue ring base camp, and the white ring base camp are segmented from the broadcast video, and the contour information is extracted according to the semantic segmentation results obtained by the DeepLabV3+ network in the second step. Figure 1 The process gets Figure 2 The analytical expressions of the straight lines AC and BD in the figure, and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring.

[0119] (1) Extract the binary mask of the image. Set the pixel positions corresponding to the blank track, blue ring base camp, and white ring base camp to 1, and the other positions to 0, and obtain the mask images respectively.

[0120] (2) Calculate the edge contour. If there is a zero pixel in the 8-connected domain or 4-connected domain of a pixel in the mask image, then this pixel is a boundary point. Multiple contour point sets can be obtained for each connected area of ​​the mask image. The area corresponding to each contour point set is calculated, and the contour with the largest area is selected as the edge contour.

[0121] (3) Contour line filtering. The horizontal coordinate of each point on the contour line is used as the independent variable and the vertical coordinate is used as the output (the coordinates of each point on the contour line in the curling track coordinate system are (xi, yi), xi is used as the independent variable, and yi is used as the output). Low-pass filtering is performed, and the absolute value of the difference between the original value and the filtered value of the vertical coordinate of each point is used as the judgment basis. Points with a difference of more than 10 pixels are deleted.

[0122] (4) Contour line analytical fitting. Based on the contour point set after filtering, the least square method is used to fit the contour point set to obtain the analytical expression of the ellipse corresponding to the inner and outer contours of the straight lines AC and BD at the edge of the curling track and the blue ring in the broadcast video.

[0123] The fourth step is to calculate the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system based on the prior information of the curling track size, and transform the curling game broadcast video from an oblique perspective to a bird's-eye view.

[0124] The coordinates of points A, B, C, and D in the broadcast video coordinate system are calculated based on the analytical expression of the ellipse corresponding to the inner and outer contours of the curling track edge lines AC and BD and the blue ring in the broadcast video, and then the coordinates of points A, B, C, and D in the curling track coordinate system are obtained based on the prior information of the curling track size, and the first homography matrix of the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system can be calculated; the coordinates of points E and F in the broadcast video coordinate system are calculated based on the analytical expression of the ellipse corresponding to the inner and outer contours of the curling track edge lines AC and BD and the blue ring in the broadcast video, and then the coordinates of points E and F in the curling track coordinate system are obtained based on the prior information of the curling track size, and the second homography matrix of the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system can be calculated. The two homography matrices are multiplied to obtain the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system, and then the oblique perspective image of the curling game broadcast is multiplied by the coordinate transformation matrix to obtain the overhead perspective image.

[0125] Step 2: For the broadcast video containing curling balls from a bird's-eye view, design the curling ball target detection dataset annotation principles, automatically expand the dataset through data enhancement methods, and output the curling ball detection frame based on the YOLOX network; then based on the DeepLabV3+ semantic segmentation network and Hough transform straight line fitting method, realize the detection of the curling ball handle angle.

[0126] Step 1. Extract 1,000 images from broadcast videos with curling balls at different angles, positions, occlusions, and backgrounds. Based on step 1, obtain the overhead view images. Randomly scale every 4 images, splice and amplify them according to random distribution, and label the curling balls in the overhead view images into two categories, "red" and "yellow", to obtain the curling ball target detection training dataset.

[0127] Step 2: Build a curling ball target detection network based on YOLOX, input the curling ball training set image, set the initial learning rate to 0.01, and adjust the learning rate slightly each time the gradient descent is adjusted by the exponential transformation attenuation strategy. The training cycle is set to 300, and the network is trained using small batch gradient descent to output the curling ball detection box.

[0128] Step 3: Segment the handle from the curling ball based on the DeepLabV3+ semantic segmentation network. Based on the curling ball detection frame in step 2, 1,000 curling ball images of different sizes can be obtained. These images are unified to the same size using the bilinear interpolation method, and the handle on the curling ball is marked as a training data set. The image of the curling ball handle is obtained based on the DeepLabV3+ network segmentation.

[0129] Step 4: Get the angle of the curling ball handle by Hough transform straight line fitting. Performing Hough transform on the curling ball handle image can get a set of straight lines, which are discretized at equal intervals according to the straight line angle with an accuracy of 1 degree. Fit 200 discrete points into a straight line, convert the slope of the fitted straight line into an angle, and take the average as the angle of the curling ball handle.

[0130] Step 3: The position and speed of the curling ball can be calculated based on the curling ball detection frame output in step 2, and the rotational angular velocity of the curling ball can be calculated based on the curling ball handle angle output in step 4, that is, the motion state of the curling ball in the broadcast video is obtained.

[0131] The method of the present invention performs semantic segmentation on a curling track in a broadcast video, calculates a coordinate transformation matrix between a broadcast video plane and a curling track plane based on size prior information in the curling track, and transforms the broadcast video of the curling game from an oblique perspective to a bird's-eye perspective; then, a curling ball target detection network is trained, a data set annotation principle is designed, and a data set is automatically amplified by a data enhancement method to obtain a detection frame and position information of the curling ball; then, based on an image segmentation algorithm and a Hough transform straight line fitting method, handle angle information describing the rotation of the curling ball is obtained, thereby realizing the motion state detection of the curling ball in the broadcast video.

Claims

1. A curling ball motion state detection method for broadcast video, It is characterized in that The method comprises: Step 1: semantically segment the curling track in the broadcast video, calculate the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system based on the prior information of the size of the curling track, and transform the broadcast video from an oblique perspective to a bird's-eye view; Step 2: Establish a curling ball target detection network, including: Collect curling ball broadcast videos, convert the curling ball broadcast videos using the method in step 1, and establish a curling ball target detection training data set through a data enhancement method; based on the curling ball target detection training data set, establish a curling ball target detection network using a YOLOX network, and the output of the curling ball target detection network is a curling ball detection frame; Step 3: Establish a curling ball handle angle detection model, including: Based on the curling ball detection frame output by the curling ball target detection network in step 2, a curling ball detection frame dataset is constructed; based on the curling ball detection frame dataset, a curling ball handle angle detection model is established by using the DeepLabV3+ semantic segmentation network and the Hough transform straight line fitting method, wherein the output of the curling ball handle angle detection model is the curling ball handle angle; Step 4, obtaining a curling match broadcast video to be detected, and converting the curling match broadcast video to be detected using the method of step 1; Using the converted curling game broadcast video to be detected and the curling ball target detection network to obtain a curling ball detection frame to be detected, and obtaining the position and speed of the curling ball to be detected according to the curling ball detection frame; Step 5, according to the curling ball detection frame to be detected and using the curling ball handle angle detection model, obtain the curling ball handle angle to be detected, and obtain the rotational angular velocity of the curling ball to be detected according to the curling ball handle angle to be detected; Step 1 specifically includes: Step 1.1, semantically segment the curling track in the broadcast video, and the semantically segmented image annotation includes 8 categories, specifically including: blank track, vanguard line, blue ring base camp, white ring base camp, red ring base camp, white circle base camp, scoreboard and logo; Step 1.2, segment the curling track based on the DeepLabV3+ semantic segmentation network, specifically: perform dilated convolution and dilated spatial pyramid pooling on the broadcast video, set the initial learning rate, use the exponential transformation attenuation strategy to dynamically adjust the learning rate during training, and set the objective function to the intersection loss function. Based on the 8-category image annotation in step 1.1, the semantic segmentation result of the curling track image in the broadcast video is obtained; Step 1.3: Based on the semantic segmentation results obtained in step 1.2, a blank track, a blue ring base camp, and a white ring base camp are segmented from the broadcast video, and the analytical expressions of the left boundary line and the right boundary line of the blank track, as well as the analytical expressions of the ellipse corresponding to the inner and outer contours of the blue ring are obtained; Step 1.4, using the analytical expressions of the left boundary line and the right boundary line of the blank track and the analytical expressions of the ellipse corresponding to the inner and outer contours of the blue ring, and based on the prior information of the curling track size, the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system is calculated to transform the broadcast video from an oblique perspective to a top-down perspective; Step 1.4 specifically includes: Step 1.4.1, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the intersection point A of the outer contour tangent line of the blue ring and the left boundary line, the intersection point B of the outer contour tangent line of the blue ring and the right boundary line, the intersection point C of the inner contour tangent line of the blue ring and the left boundary line, and the intersection point D of the inner contour tangent line of the blue ring and the right boundary line in the broadcast video coordinate system, and the outer contour tangent line of the blue ring and the inner contour tangent line of the blue ring are both parallel to the T line; Then, according to the prior information of the curling track size, the coordinates of the intersection points A, B, C and D in the curling track coordinate system are obtained to obtain a first homography matrix; Step 1.4.2, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the tangent point E of the outer contour tangent of the blue ring and the tangent point F of the inner contour tangent of the blue ring in the broadcast video coordinate system; Then, the coordinates of the tangent point E and the tangent point F in the curling track coordinate system are obtained according to the prior information of the curling track size, and a second homography matrix is ​​obtained; Step 1.4.3, multiplying the first homography matrix and the second homography matrix to obtain a coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system; Step 1.4.4: multiply the broadcast video by the coordinate transformation matrix to obtain a bird's-eye view image, and transform the broadcast video from a squint view to a bird's-eye view.

2. A curling ball motion state detection method for broadcast video according to claim 1, It is characterized in that The loss function F is specifically: Among them, y is the real sample, y′ is the predicted category probability given by the model, γ is the loss factor, and α is the balance factor.

3. A curling ball motion state detection method for broadcast video according to any one of claims 1 or 2, It is characterized in that Step 1.3 specifically includes: Step 1.3.1, set the pixel positions corresponding to the blank track, blue ring base camp, and white ring base camp to 1, and the other positions to 0, and obtain mask images respectively; Step 1.3.2, if there is a 0 pixel in the 8-connected domain or 4-connected domain of a pixel in the mask image, then this pixel is a boundary point, and multiple contour point sets of each connected area of ​​the mask image are obtained. Then the area corresponding to each contour point set is calculated, and the contour with the largest area is selected as the edge contour; Step 1.3.3, the coordinates of each point on the contour line in the curling track coordinate system are (x i ,y i ), and x i As the independent variable, y i As the output, low-pass filtering is performed to obtain y i The absolute value of the difference between the original value and the filtered value is taken as the difference, and the points of pixels whose difference exceeds a preset number are deleted; Step 1.3.4: Based on the contour point set after filtering, the least square method is used to fit the contour point set to obtain the analytical expressions of the left boundary line and the right boundary line of the blank track, as well as the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring.

4. The curling ball motion state detection method for broadcast video according to claim 1, It is characterized in that Step 2 specifically includes: Step 2.1, collect a number of curling ball images from broadcast videos with curling balls at different angles, positions, occlusions and different backgrounds, convert the curling ball images using the method in step 1, randomly scale each preset number of converted images, and then splice and amplify them according to random distribution, and mark the curling balls in the converted images according to color categories to obtain a curling ball target detection training data set; Step 2.2, use the YOLOX network to establish a curling ball target detection network, the input of the curling ball target detection network is the curling ball target detection training data set, and the output is the curling ball detection frame; set the initial learning rate of the curling ball target detection network, and adjust the learning rate at each gradient descent by the exponential transformation attenuation strategy, set the training cycle, and train the network using small batch gradient descent.

5. The curling ball motion state detection method for broadcast video according to claim 1, It is characterized in that Step 3 specifically includes: Step 3.1: Based on the curling ball detection frame output by the curling ball target detection network in step 2, obtain several curling ball detection frame images, use the bilinear interpolation method to unify the several curling ball detection frame images to the same size, and mark the handle on the curling ball to construct a curling ball detection frame dataset. Step 3.2: Based on the curling ball detection frame dataset, a curling ball handle angle detection model is established using the DeepLabV3+ semantic segmentation network and the Hough transform straight line fitting method. The output of the curling ball handle angle detection model is the curling ball handle angle, which specifically includes: Based on the curling ball detection frame dataset, a curling ball handle image is obtained using a DeepLabV3+ semantic segmentation network; The curling ball handle image is subjected to Hough transform by using the Hough transform straight line fitting method to obtain a set of straight lines, which are discretized at equal intervals according to the straight line angles with an accuracy of 1 degree. Several discrete points are overlapped and fitted into a straight line, and the slope of the fitted straight line is converted into an angle, and the angle mean is taken as the curling ball handle angle.

6. A curling ball motion state detection system for broadcast video, It is characterized in that The system comprises: A coordinate conversion module is used to perform semantic segmentation on the curling track in the broadcast video. Based on the prior information of the size of the curling track, a coordinate conversion matrix between the broadcast video coordinate system and the curling track coordinate system is calculated to convert the broadcast video from an oblique perspective to a bird's-eye view. The curling ball target detection network establishment module is used to establish the curling ball target detection network, specifically including: An image acquisition and conversion module is used to acquire a curling ball broadcast video, convert the curling ball broadcast video using a coordinate conversion module, and establish a curling ball target detection training data set through a data enhancement method; A network building module, used for building a curling ball target detection network based on the curling ball target detection training data set and using a YOLOX network, wherein the output of the curling ball target detection network is a curling ball detection frame; A curling ball handle angle acquisition module is used to construct a curling ball detection frame dataset based on the curling ball detection frame output by the curling ball target detection network in the curling ball target detection network establishment module; based on the curling ball detection frame dataset, and using the DeepLabV3+ semantic segmentation network and Hough transform straight line fitting method, the curling ball handle angle is acquired; A curling ball position and speed acquisition module, used to acquire a curling game broadcast video to be detected, and to transform the curling game broadcast video to be detected using a coordinate conversion module; Using the converted curling game broadcast video to be detected and the curling ball target detection network to obtain a curling ball detection frame to be detected, and obtaining the position and speed of the curling ball to be detected according to the curling ball detection frame; a curling ball rotation angular velocity acquisition module, configured to acquire the handle angle of the curling ball to be detected according to the curling ball detection frame to be detected and using the curling ball handle angle detection model, and to acquire the rotation angular velocity of the curling ball to be detected according to the handle angle of the curling ball to be detected; The coordinate conversion module specifically includes: Step 1.1, semantically segment the curling track in the broadcast video, and the semantically segmented image annotation includes 8 categories, specifically including: blank track, vanguard line, blue ring base camp, white ring base camp, red ring base camp, white circle base camp, scoreboard and logo; Step 1.2, segment the curling track based on the DeepLabV3+ semantic segmentation network, specifically: perform dilated convolution and dilated spatial pyramid pooling on the broadcast video, set the initial learning rate, use the exponential transformation attenuation strategy to dynamically adjust the learning rate during training, and set the objective function to the intersection loss function. Based on the 8-category image annotation in step 1.1, the semantic segmentation result of the curling track image in the broadcast video is obtained; Step 1.3: Based on the semantic segmentation results obtained in step 1.2, a blank track, a blue ring base camp, and a white ring base camp are segmented from the broadcast video, and the analytical expressions of the left boundary line and the right boundary line of the blank track, as well as the analytical expressions of the ellipse corresponding to the inner and outer contours of the blue ring are obtained; Step 1.4, using the analytical expressions of the left boundary line and the right boundary line of the blank track and the analytical expressions of the ellipse corresponding to the inner and outer contours of the blue ring, and based on the prior information of the curling track size, the coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system is calculated to transform the broadcast video from an oblique perspective to a top-down perspective; Step 1.4 specifically includes: Step 1.4.1, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the intersection point A of the outer contour tangent line of the blue ring and the left boundary line, the intersection point B of the outer contour tangent line of the blue ring and the right boundary line, the intersection point C of the inner contour tangent line of the blue ring and the left boundary line, and the intersection point D of the inner contour tangent line of the blue ring and the right boundary line in the broadcast video coordinate system, and the outer contour tangent line of the blue ring and the inner contour tangent line of the blue ring are both parallel to the T line; Then, according to the prior information of the curling track size, the coordinates of the intersection points A, B, C and D in the curling track coordinate system are obtained to obtain a first homography matrix; Step 1.4.2, according to the analytical expressions of the left boundary line of the blank track and the right boundary line of the blank track and the analytical expressions of the ellipses corresponding to the inner and outer contours of the blue ring, obtain the coordinates of the tangent point E of the outer contour tangent of the blue ring and the tangent point F of the inner contour tangent of the blue ring in the broadcast video coordinate system; Then, the coordinates of the tangent point E and the tangent point F in the curling track coordinate system are obtained according to the prior information of the curling track size, and a second homography matrix is ​​obtained; Step 1.4.3, multiplying the first homography matrix and the second homography matrix to obtain a coordinate transformation matrix between the broadcast video coordinate system and the curling track coordinate system; Step 1.4.4: multiply the broadcast video by the coordinate transformation matrix to obtain a bird's-eye view image, and transform the broadcast video from a squint view to a bird's-eye view.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, It is characterized in that When the processor runs the computer program stored in the memory, the steps of the method according to any one of claims 1 to 5 are performed.

8. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a plurality of computer instructions, and the plurality of computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 5.