An artificial intelligence-based auxiliary penalty method and a real scene hawk eye system

Abstract

The application provides an artificial intelligence-based auxiliary penalty method and a real scene hawk eye system, which comprises the following steps: acquiring continuous image frames collected by a camera in real time, wherein the image frames comprise a tennis court and a tennis ball located in the tennis court; detecting the position of the tennis ball and the landing position of the tennis ball in a tennis bounce frame from the continuous image frames; taking N pixel points to the left and right and M pixel points to the top and bottom of the landing position of the tennis ball in the tennis bounce frame as a center to form a region of interest; identifying the tennis ball contour and the court boundary line in the region of interest; and generating a penalty result according to the positional relationship between the tennis ball contour and the court boundary line. The application can reduce the system cost, ensure the accuracy, improve the real-time performance, and has important significance for popularizing the hawk eye system in the field of national fitness.

Description

An AI-based assisted judgment method and a real-scene eagle-eye system Technical Field

[0001] This invention relates to the field of sports equipment technology, and in particular to an artificial intelligence-based assisted judgment method and a real-view eagle-eye system. Background Technology

[0002] "Hawk-Eye," also known as a real-time replay system, is used to assist referees in making accurate calls. Professional Hawk-Eye systems, such as those used in the ATP, WTA, and Grand Slams, typically employ 8-10 high-speed cameras to determine and display results within 10 seconds. While offering extremely high accuracy, the high cost of these cameras (often tens to hundreds of thousands of dollars per system) and the high latency resulting from simultaneously processing 8-10 channels of high-frame-rate, high-resolution data make them both expensive and lacking in real-time performance. Lightweight video analytics solutions on the market use ordinary cameras and open-source target detection algorithms to detect tennis balls. While this reduces costs, accuracy in detecting tennis trajectories at high resolutions is difficult to guarantee, making their Hawk-Eye results unconvincing to referees, players, and spectators.

[0003] Therefore, how to reduce system costs while ensuring detection accuracy and improving real-time performance is of great significance for the popularization of the Eagle Eye system in the field of national fitness. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide an artificial intelligence-based assisted judgment method and a real-scene eagle eye system that solves or at least partially solves the above technical problems.

[0005] One aspect of the present invention provides an artificial intelligence-based assisted judgment method, the method comprising:

[0006] Acquire continuous image frames captured in real time by a camera device, the image frames including a tennis court and tennis balls located in the tennis court;

[0007] Detect the position of the tennis ball in consecutive image frames and the landing point of the tennis ball in the tennis ball bounce frame;

[0008] In the tennis ball bounce frame, take N pixels to the left and right of the landing point of the tennis ball as the center, and M pixels to the up and down to form a region of interest;

[0009] Identify the tennis ball outline and court boundary lines in the region of interest;

[0010] The ruling is generated based on the positional relationship between the tennis ball outline and the court boundary lines.

[0011] Furthermore, detecting the position of the tennis ball from consecutive image frames and the landing point of the tennis ball in the bounce frame includes:

[0012] A pre-designed lightweight multi-task joint learning network model is used to detect the position of the tennis ball in consecutive image frames and the landing point of the tennis ball in the tennis ball bounce frame.

[0013] The multi-task joint learning network model includes an encoder, a feature enhancement network, and two task heads: a tennis ball detection task head network and a landing point recognition task head network. The encoder extracts tennis ball features at different scales from the input image frame. The feature enhancement module fuses the tennis ball features at different scales to obtain a tennis ball fusion feature that combines semantic and edge texture information. The tennis ball detection task head network identifies the position of the tennis ball in the image frame based on the fusion feature, and the landing point recognition task head network identifies the landing point of the tennis ball in the image frame based on the fusion feature and determines the tennis ball bounce frame.

[0014] Furthermore, the encoder includes a first downsampling network layer and a second downsampling network layer connected in sequence;

[0015] The first downsampling network layer consists of a 3×3 standard convolutional layer and a 3×3 max pooling layer, which is used to extract primary feature maps from the image to achieve preliminary dimensionality reduction;

[0016] The second downsampling network layer includes a downsampling module and at least three feature transformation modules. The downsampling module is used to extract tennis features of different scales from the primary feature map. It includes a first branch consisting of 3×3 depthwise separable convolutions in parallel and a second branch consisting of 1×1 standard convolutions, 3×3 depthwise separable convolutions, and 1×1 standard convolutions connected in sequence. The feature output channels of the first and second branches are channel-mixed. Each feature transformation module is used to transform the tennis features of different scales obtained by the downsampling module. It includes a third branch with identity mapping function in parallel and a fourth branch consisting of standard 1×1 convolutions, 3×3 depthwise separable convolutions, and 1×1 standard convolutions connected in sequence. The feature output channels of the downsampling module are mixed and divided into two equal parts, which are used as the inputs of the third and fourth branches, respectively. The feature output channels of the third and fourth branches are then channel-mixed.

[0017] Furthermore, the feature enhancement network includes a first upsampling network layer and a second upsampling network layer connected in sequence;

[0018] The first upsampling network layer includes a 3×3 RepVGG convolutional layer and a first upsampling module. The 3×3 RepVGG convolutional layer is used to enhance local feature extraction of the output features of the second downsampling network layer. The first upsampling module is used to upsample the output features of the RepVGG convolutional layer using bilinear interpolation. The upsampled feature output channels are concatenated with the feature input channels of the second downsampling network layer to obtain 72 feature output channels.

[0019] The second upsampling network layer includes a 3×3 standard convolutional layer and a second upsampling module. The 3×3 standard convolutional layer is used to extract features from the output features of the 72 feature output channels of the first upsampling network layer and convert them into 48 feature output channels for output. The first upsampling module is used to upsample the current input features using bilinear interpolation. The upsampled feature output channels are concatenated with the feature output channels of the first downsampling network layer to obtain 72 feature output channels.

[0020] Furthermore, the tennis ball detection task head network includes a third upsampling module, a 3×3 RepVGG convolutional layer, and a 3×3 standard convolutional layer connected in sequence. The third upsampling module is used to upsample the output features of the feature enhancement network using bilinear interpolation. The 3×3 RepVGG convolutional layer is used to enhance local feature extraction from the output features of the third upsampling module. The 3×3 standard convolutional layer is used to transform the 72-channel input feature channels into 3-channel feature output channels.

[0021] Furthermore, the landing point recognition task head network includes three sets of 3×3 RepVGG convolutional layers, a max pooling layer, three sets of 3×3 RepVGG convolutional layers, a channel attention layer, and a fully connected layer, which are connected in sequence. By establishing a channel attention mechanism in the landing point recognition task head network, the channel weights of each feature input channel of the landing point recognition task head network are dynamically adjusted according to the channel attention mechanism to enhance the attention of the landing point recognition task head network to the motion features of the tennis ball in the image and output the landing point position of the tennis ball in the image frame.

[0022] Furthermore, after acquiring continuous image frames captured in real time by the camera device, the method further includes:

[0023] Pre-fusion of image frames is achieved by merging a preset number of consecutive image frames along the channel dimension.

[0024] Furthermore, the method also includes:

[0025] The position of the tennis ball detected from multiple consecutive image frames and the landing point of the tennis ball in the tennis ball bounce frame are plotted on a preset real-world map to form a dynamic video or static image corresponding to the tennis ball landing event, which includes the tennis ball's trajectory before bounce, landing point, and trajectory after bounce.

[0026] Furthermore, identifying the tennis ball outline and court boundary lines in the region of interest includes:

[0027] Acquire at least the third image frame whose acquisition time is earlier than the tennis ball bounce frame as the background frame, and select the region of interest in the background frame in the same way with the coordinates of the tennis ball's landing point in the tennis ball bounce frame as the center.

[0028] The regions of interest in the tennis ball bounce frame and the background frame are converted into grayscale images using a color conversion operation.

[0029] The absolute difference between the regions of interest in the tennis ball bounce frame and the background frame is calculated using the frame difference method to obtain a local difference image, and the local difference image after absolute difference is binarized.

[0030] The binarized image is eroded, and the contour of the eroded image is detected to obtain the tennis ball contour.

[0031] In the tennis ball bounce frame, the width and height of the region of interest in the tennis ball bounce frame are reduced by a preset ratio to obtain a rectangular edge region.

[0032] Convert the border area to a grayscale image;

[0033] The grayscale image is binarized and dilated. Contour detection is then performed on the dilated image to obtain the contour of the site boundary line.

[0034] Another aspect of the present invention provides an artificial intelligence-based real-time eagle-eye system, the system comprising:

[0035] At least one camera device is used to film the tennis court and the tennis balls and people located in the tennis court to capture real-time footage;

[0036] A server includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the method as claimed in any one of claims 1-9.

[0037] The AI-based assisted judgment method and real-time eagle-eye system provided in this invention can achieve eagle-eye system assisted judgment using only a single camera combined with computer vision algorithms. Compared to solutions using multiple high-speed cameras, this significantly reduces the installation and deployment cost of the eagle-eye system. Compared to other computer vision algorithm solutions, this invention simultaneously learns tennis ball detection and landing point recognition. By combining tennis ball detection and landing point recognition, it effectively improves the time and location accuracy of landing point recognition, making the tennis ball trajectory more reliable and effectively ensuring the accuracy of eagle-eye recognition. This invention can reduce system costs while ensuring accuracy and improving real-time performance, which is of great significance for the popularization of eagle-eye systems in the field of national fitness.

[0038] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. In the drawings:

[0040] Figure 1 is a flowchart of an artificial intelligence-based assisted judgment method according to an embodiment of the present invention;

[0041] Figure 2 is a schematic diagram of the output of the tennis ball detection task head in an embodiment of the present invention;

[0042] Figure 3 is a schematic diagram of the output of the Eagle Eye system algorithm in an embodiment of the present invention;

[0043] Figure 4 is a schematic diagram of 3D virtual scene annotation in an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram showing the penalty result in a three-dimensional virtual scene in an embodiment of the present invention. Detailed Implementation

[0045] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0046] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the meaning consistent with their meaning in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined.

[0047] This invention provides an artificial intelligence-based assisted judgment method, as shown in Figure 1. The artificial intelligence-based assisted judgment method proposed in this invention includes the following steps:

[0048] S1. Acquire continuous image frames captured in real time by the camera device. The image frames include a tennis court and tennis balls located within the tennis court. Specifically, the continuous image frames can be selected as three consecutive frames. A single camera device deployed on the tennis court can capture images of the tennis court, tennis balls, and people located within the tennis court to obtain multiple sets of continuous image frames in real time. In this embodiment, the system input is three consecutive image frames, which can effectively utilize temporal information, better capture the displacement of objects between consecutive frames, maintain the continuity of event development (such as object appearance → movement → disappearance), and reduce single-frame misjudgment.

[0049] S2. Detect the position of the tennis ball in consecutive image frames and the landing point of the tennis ball in the bounce frame. Specifically, detect the position of the tennis ball and whether the tennis ball bounces and lands in consecutive image frames. The bounce frame can be selected as the middle image frame in the consecutive images. In a specific example, if the input is 3 consecutive images, the middle frame of the three consecutive images is the bounce frame.

[0050] S3. In the tennis ball bounce frame, take N pixels to the left and right of the landing point of the tennis ball as the center, and M pixels to the up and down to form a region of interest.

[0051] S4. Identify the tennis ball outline and court boundary lines in the region of interest.

[0052] S5. Generate the ruling based on the positional relationship between the tennis ball outline and the court boundary lines.

[0053] Specifically, after determining the positional relationship between the tennis ball's outline and the court's boundary lines—that is, whether the landing point is on or off the sideline—if it's on the sideline, it's considered "in" according to tennis rules. If it's off the sideline, the shot needs to be linked. By linking the landing point to the shot (the shot event data can be obtained from the existing system), the valid landing area is determined based on the shot type (serve, return) and the match mode (singles / doubles), thus calculating the true judgment result (IN / OUT) for that landing point.

[0054] The AI-based assisted judgment method provided in this invention utilizes only a single camera combined with computer vision algorithms to implement eagle-eye system-assisted judgment. Compared to solutions using multiple high-speed cameras, this significantly reduces the installation and deployment costs of the eagle-eye system. Compared to other computer vision algorithm solutions, this invention simultaneously learns tennis ball detection and landing point recognition. By combining tennis ball detection and landing point recognition, it effectively improves the time and location accuracy of landing point recognition, making the tennis ball trajectory more reliable and effectively ensuring the accuracy of eagle-eye recognition. This invention can reduce system costs while maintaining accuracy and improving real-time performance, which is of great significance for the popularization of eagle-eye systems in the field of national fitness.

[0055] In this embodiment of the invention, after acquiring continuous image frames captured in real time by the camera device, the method further includes: merging a preset number of continuous image frames in the channel dimension to achieve pre-fusion of image frames.

[0056] Specifically, the utilization of temporal information can be achieved in different ways. Pre-fusion involves merging multiple frames of data during the input stage, such as stacking three images along the channel dimension. Intermediate fusion can also be used, occurring in the middle layers of the network, such as fusing temporal information after feature extraction at different stages. Post-fusion can be used after the network output, such as integrating the segmentation results of multiple frames. While pre-fusion increases the number of input channels, its processing method is simple and its computational efficiency is relatively higher than the other two methods, especially after optimization with the TensorRT inference engine. Intermediate fusion can capture features at different levels, but may increase network complexity and computational load. Post-fusion can optimize the final result, but requires processing multiple frames of output, affecting real-time performance. Different fusion methods can be designed according to actual needs in the specific implementation and process.

[0057] To achieve a balance between accuracy and speed, and given that it's real-time detection, inference speed is crucial, so the fusion method cannot be overly complex. Simultaneously, tennis balls are fast-moving small targets, requiring effective capture of their motion trajectories using temporal information to avoid missed or false detections. In a specific example, this embodiment of the invention employs a pre-fusion approach, which is simple to implement; multiple frames are directly merged from the input, and the network can automatically learn spatiotemporal features.

[0058] The core algorithm of the real-scene Eagle Eye system lies in accurately determining the tennis ball's bounce frame and bounce position, which requires high accuracy in both timing and location of the landing point. To improve both time and location accuracy, the Eagle Eye system employs a multi-task joint learning scheme, simultaneously learning the landing point recognition task and the tennis ball detection task. The tennis ball detection task locates the tennis ball's position in the image. By introducing supervised learning for tennis ball detection, the model can focus more intently on the tennis ball's movement across the entire image, thereby improving the accuracy of landing point recognition.

[0059] In this embodiment of the invention, the specific implementation of detecting the tennis ball position from consecutive image frames and the landing point of the tennis ball in the tennis ball bounce frame is as follows: a preset lightweight multi-task joint learning network model is used to detect the tennis ball position from consecutive image frames and the landing point of the tennis ball in the tennis ball bounce frame. The multi-task joint learning network model includes an encoder, a feature enhancement network, and two task heads, namely a tennis ball detection task head network and a landing point recognition task head network. The encoder is used to extract tennis ball features of different scales from the input image frames. The feature enhancement module is used to fuse tennis ball features of different scales to obtain a tennis ball fusion feature that combines semantic information and edge texture information. The tennis ball detection task head network is used to identify the position of the tennis ball in the image frame based on the fusion feature. The landing point recognition task head network is used to identify the landing point of the tennis ball in the image frame based on the fusion feature and to determine the tennis ball bounce frame. In this embodiment, the multi-task network model is used to predict the bounce frame. The input of this model is three consecutive image frames, and the output has two paths: one is the prediction of the tennis ball position, and the other is whether the three consecutive image frames have bounced. If they have bounced, the middle frame of the three image frames is the bounce frame. Specifically, the landing point refers to the point where the tennis ball bounces off the ground. The landing point identification task head is used to find the tennis ball bounce frames. This task head will output whether a bounce occurred in these three frames. If a bounce occurred, then the middle frame is the bounce frame. With the assistance of the tennis ball's positional features, it is possible to more accurately identify whether a bounce has occurred.

[0060] In this embodiment, the Eagle Eye system algorithm is designed using a convolutional neural network. To enable accurate and efficient deployment and application of the Eagle Eye system, a lightweight multi-task joint learning network model is designed. The multi-task joint learning network model consists of an encoder, a feature enhancement module, and two task heads. The encoder extracts tennis ball features, the feature enhancement module fuses tennis ball features at different scales, and the task heads further process the features to obtain task-oriented outputs, outputting the results of tennis ball position and landing point recognition, respectively.

[0061] Specifically, the encoder includes a first downsampling network layer and a second downsampling network layer connected in sequence;

[0062] The first downsampling network layer consists of a 3×3 standard convolutional layer and a 3×3 max pooling layer, which is used to extract primary feature maps from the image to achieve preliminary dimensionality reduction;

[0063] The second downsampling network layer includes a downsampling module and at least three feature transformation modules. The downsampling module is used to extract tennis features of different scales from the primary feature map. It includes a first branch consisting of 3×3 depthwise separable convolutions in parallel and a second branch consisting of 1×1 standard convolutions, 3×3 depthwise separable convolutions, and 1×1 standard convolutions connected in sequence. The feature output channels of the first and second branches are channel-mixed. Each feature transformation module is used to transform the tennis features of different scales obtained by the downsampling module. It includes a third branch with identity mapping function in parallel and a fourth branch consisting of standard 1×1 convolutions, 3×3 depthwise separable convolutions, and 1×1 standard convolutions connected in sequence. The feature output channels of the downsampling module are mixed and divided into two equal parts, which are used as the inputs of the third and fourth branches, respectively. The feature output channels of the third and fourth branches are then channel-mixed.

[0064] In a specific example, the encoder consists of two stages. Considering that the tennis ball occupies a small portion of the image, if we adopt the 16x or 32x downsampling common in many existing object detection models, too much object information will be lost. Therefore, the model is designed to downsample by only 8x.

[0065] 1. The initial layer, also known as the first downsampling network layer (Stage 1)

[0066] Input resolution: 288×512

[0067] • Layer type: Standard convolutional layer + max pooling layer

[0068] Design details:

[0069] o Standard convolutional layer: 3×3 convolution, stride = 2, output channels = 24.

[0070] o Max pooling layer: 3×3 pooling, stride = 2, used for fast downsampling to 72×128 resolution.

[0071] • Function: Quickly extract low-level features such as edges and textures, and perform preliminary dimensionality reduction.

[0072] 2. Stage 2, also known as the second downsampling network layer

[0073] Input resolution: 72×128 → Output resolution: 36×64 (downsampled via the first downsampling block)

[0074] • Structure: 1 downsampling block (i.e., downsampling module) + 3 regular blocks (i.e., feature transformation modules). To increase network depth and enhance feature extraction capabilities, at least 3 regular blocks are required.

[0075] • Downsampling Block Design:

[0076] Branch 1: 3×3 depthwise separable convolution (DWConv), stride = 2, output channels = 24.

[0077] Branch 2: 1×1 convolution + 3×3 depthwise separable convolution (stride = 2) + 1×1 convolution, output channels = 24.

[0078] o Channel Merging: Merges the outputs of branch 1 and branch 2 (a total of 48 channels), and then mixes the channels. Assuming the output of branch 1 is 1234 and the output of branch 2 is abcd, mixing means recombining these eight results into 1a2b3c4d.

[0079] • Standard Block Design:

[0080] o Channel splitting: Input channel = 48 → split into 24 + 24.

[0081] Branch 1: Identity Mapping (24 channels). Features obtained from depthwise separable convolution correspond to a low-dimensional space with fewer features. Subsequent identity mapping can preserve most of the features.

[0082] Branch 2: 1×1 convolution + 3×3 depthwise separable convolution + 1×1 convolution.

[0083] o Channel merging: The merged output has 48 channels → channel mixing.

[0084] Specifically, the feature enhancement network includes a first upsampling network layer and a second upsampling network layer connected in sequence;

[0085] The first upsampling network layer includes a 3×3 RepVGG convolutional layer and a first upsampling module. The 3×3 RepVGG convolutional layer is used to enhance local feature extraction of the output features of the second downsampling network layer. The first upsampling module is used to upsample the output features of the RepVGG convolutional layer using bilinear interpolation. The upsampled feature output channels are concatenated with the feature input channels of the second downsampling network layer to obtain 72 feature output channels.

[0086] The second upsampling network layer includes a 3×3 standard convolutional layer and a second upsampling module. The 3×3 standard convolutional layer is used to extract features from the output features of the 72 feature output channels of the first upsampling network layer and convert them into 48 feature output channels for output. The first upsampling module is used to upsample the current input features using bilinear interpolation. The upsampled feature output channels are concatenated with the feature output channels of the first downsampling network layer to obtain 72 feature output channels.

[0087] In a specific example, the feature enhancement module consists of two upsampling structures: a first upsampling network layer and a second upsampling network layer. Features downsampled by 8 times possess good semantic information, but suffer significant loss of detail such as texture and color. Therefore, it is necessary to combine the downsampled features by 8 times with the network's underlying high-resolution features to further improve recognition performance. Considering deployment speed, convolutional layers are typically used to reduce the number of channels and computational complexity before upsampling. To avoid a large number of parameters during upsampling, this invention uses the upsample module for upsampling.

[0088] First upsampling network layer: Use features downsampled by 8 times. Since the number of channels is relatively small (48 channels), there is no need to reduce the dimensionality of the channels. Use a RepVGG convolution to enhance the features. During deployment, the parameters can be reparameterized to improve speed.

[0089] • Upsampling Block Design:

[0090] o RepVGG convolution: 3×3 convolution, stride = 2, output channels = 48.

[0091] o Upsampling operation: Bilinear interpolation + 3*3 standard convolution.

[0092] The input of stage o is concatenated with the input of stage 2, i.e., the second downsampling network layer, using channel concatenation (48+24=72 channels). Instead of elementwise addition, channel concatenation is used to better preserve feature details.

[0093] Second upsampling network layer:

[0094] • Upsampling Block Design:

[0095] o 3*3 standard convolution: output channels 72→48.

[0096] o Upsampling operation: Bilinear interpolation + 3*3 standard convolution.

[0097] o performs a channel concatenation operation with the output of stage1, i.e., the first downsampling network layer (48+24=72 channels).

[0098] In this embodiment, by concatenating the upsampled features of the first upsampled network layer with the input features of the second downsampled network layer, and concatenating the upsampled features of the second upsampled network layer with the output features of the first downsampled network layer, the features downsampled by 8 times can be fused with the edge texture and other detailed features of the bottom layer of the network, thereby improving the tennis ball detection effect.

[0099] Specifically, the tennis ball detection task head network includes a third upsampling module, a 3×3 RepVGG convolutional layer, and a 3×3 standard convolutional layer connected in sequence. The third upsampling module is used to upsample the output features of the feature enhancement network using bilinear interpolation. The 3×3 RepVGG convolutional layer is used to enhance local feature extraction from the output features of the third upsampling module. The 3×3 standard convolutional layer is used to transform the 72-channel input feature channels into 3-channel feature output channels.

[0100] This application enhances the nonlinearity and expressive power of the model by stacking multiple 3×3 convolutional layers without increasing the number of parameters, thus significantly improving the model's performance and efficiency.

[0101] In this embodiment, the tennis ball detection task head utilizes the features output by the feature enhancement module to further process and generate a prediction of the tennis ball's position. Tennis ball detection employs semantic segmentation, which, compared to object detection that only detects a center point and predicts distances to the four borders (top, bottom, left, and right), uses a tennis ball mask for supervision, thus providing more supervisory information. Therefore, this task head continues upsampling and ultimately outputs the tennis ball's position.

[0102] • Tennis testing head design:

[0103] o Upsampling operation: Bilinear interpolation + 3*3 convolution.

[0104] o RepVGG convolution: 3×3 convolution, stride = 2, output channels = 72.

[0105] o 3×3 convolutional layer: 72 input channels, 3 output channels, and the final output is batch*3*288*512.

[0106] Figure 2 shows an example of the output from the tennis ball detection task head, with pink dots indicating the trajectory of the tennis ball.

[0107] Specifically, the landing point recognition task head network includes three sets of 3×3 RepVGG convolutional layers, a max pooling layer, three sets of 3×3 RepVGG convolutional layers, a channel attention layer, and a fully connected layer, which are connected in sequence. By establishing a channel attention mechanism in the landing point recognition task head network, the channel weights of each feature input channel of the landing point recognition task head network are dynamically adjusted according to the channel attention mechanism to enhance the landing point recognition task head network's attention to the motion features of the tennis ball in the image and output the landing point position of the tennis ball in the image frame.

[0108] In this embodiment, the landing point recognition task head integrates spatial and temporal information to identify whether the tennis ball bounces off the ground. An attention mechanism is added to make the model more focused on the movement of the tennis ball in the image.

[0109] • Landing point recognition task head design:

[0110] o 3 sets of RepVGG convolutions + max pooling layer + 3 sets of RepVGG convolutions.

[0111] o Se channel attention mechanism.

[0112] The fully connected layer outputs the result of a tennis ball bouncing off the ground.

[0113] The AI-based assisted judgment method provided in this embodiment of the invention further includes: drawing the tennis ball position detected from multiple sets of consecutive image frames and the landing point of the tennis ball in the tennis ball bounce frame on a preset real-world map to form a dynamic video or static image corresponding to the tennis ball landing event, which is marked with the tennis ball's trajectory before bounce, landing point, and trajectory after bounce.

[0114] Specifically, when a player triggers the Hawk-Eye challenge, the Hawk-Eye system service host locates a 2-3 second video corresponding to the landing frame and uses video editing technology to add tennis ball trajectory, camera movement effects, etc., to this video segment. This step generates a visual image from the Hawk-Eye's judgment, which is then displayed to the referee, player, and audience. The entire rendering process runs on WebGL, enabling real-time rendering and playback on mobile devices, large screens, and live streams, greatly improving timeliness. From triggering the Hawk-Eye challenge to making it visually viewable, it only takes a few seconds.

[0115] Furthermore, after the special effects live-action video is produced, it needs to be provided for external playback. The traditional solution is to upload the video to the cloud, and then the playback device retrieves the video content from the cloud. However, this process requires network transmission, which affects the real-time performance of Eagle Eye. To address this, this invention effectively avoids the impact on real-time performance by adding a reverse proxy layer. Specifically, a cloud server is deployed in the cloud, acting as a proxy for the Eagle Eye host on-site. The playback device can access the address on the cloud server to find the video on the Eagle Eye host and play it, thus eliminating the network transmission process.

[0116] Figure 3 shows an example of the output of the Hawk-Eye system algorithm. The red dots indicate the trajectory of the tennis ball before it bounces off the ground, the green dots indicate the trajectory of the tennis ball after it bounces off the ground, and the blue dots indicate the point of impact. It can be seen that the Hawk-Eye system accurately identified the time and location of the tennis ball's bounce.

[0117] In this embodiment of the invention, the steps for identifying the tennis ball outline and court boundary lines in the region of interest specifically include:

[0118] (1) Tennis ball precise contour detection steps: Obtain at least the third image frame whose acquisition time is earlier than the tennis ball bounce frame as the background frame, and select the region of interest (ROI) of the background frame in the same way with the coordinates of the tennis ball's landing point in the tennis ball bounce frame as the center; convert the ROI of the tennis ball bounce frame and the background frame into grayscale images through color conversion operation; calculate the absolute difference between the ROI of the tennis ball bounce frame and the background frame using the frame difference method to obtain a local difference image, and binarize the local difference image after absolute difference; perform erosion on the binarized image, and perform contour detection on the eroded image to obtain the tennis ball contour. In this embodiment, the background frame must be at least the previous third frame, because the background frame cannot contain moving targets. The purpose of this step is to find the ROI through the frame difference method. Since the current frame contains the target, the background frame should be as clean as possible, with only pure background and no tennis ball. Specifically, the ROI of the current bounce frame is cropped as the foreground, and the ROI of the third frame before is cropped as the background. The two are first converted into grayscale images, and then the difference between the pixels is directly calculated to obtain the difference image. After obtaining the difference image, pixel values ​​greater than 5 are directly set to 255, and those less than or equal to 5 are set to 0, thus obtaining the binarized image.

[0119] Specifically, on the detected tennis ball bounce frame, a region of interest (ROI) is formed by taking N pixels to the left and right, and M pixels up and down, centered on the bounce location. The value of N can be the image width divided by a first preset statistical value and rounded down, and the value of M can be the image height divided by a second preset statistical value and rounded down. In this embodiment, the first preset statistical value can be 10, and the second preset statistical value can be 4. The specific values ​​of the preset statistical values ​​can be calculated based on the ratio of the original image to the target image; this invention does not impose specific limitations on this. The purpose of selecting the ROI is to focus only on the motion area of ​​the tennis ball during frame differencing, as other parts may cause interference.

[0120] • Take three frames forward from the tennis ball bounce frame as the background frame. The background frame is also cropped using the same region of interest cropping method as described above. Since the specific bounce position (x, y) in the bounce frame is known, the same position (x, y) in the background frame can be cropped.

[0121] • The regions of interest in the pop-up and background frames are converted from RGB 3-channel color images to single-channel grayscale images through a color conversion.

[0122] • Add Gaussian blur to smooth out noise in the image;

[0123] • The frame difference method calculates the absolute difference between two regions of interest and binarizes the image after the difference.

[0124] • Erosion is applied to the binarized image because only the bottom part of the tennis ball contacts the ground;

[0125] • Contour detection is performed on the eroded image to obtain a refined tennis ball contour.

[0126] (2) Steps for detecting the boundary line of a tennis court: In the tennis ball bounce frame, the width and height of the region of interest in the tennis ball bounce frame are reduced by a preset ratio to obtain a rectangular boundary line region; the boundary line region is converted into a grayscale image; the grayscale image is binarized and dilated; the dilated image is subjected to contour detection to obtain the boundary line contour of the court.

[0127] Specifically, taking the tennis ball bounce frame as the center, the width and height of the region of interest obtained in the previous step are divided by a third preset statistical value and rounded down. The third preset statistical value can be 10, so as to obtain a square edge region.

[0128] • Convert the RGB three-color channels of the edge area into a single-channel grayscale image;

[0129] Binarize the grayscale image and dilate it;

[0130] • Perform contour detection on the dilated image to obtain refined edge contours.

[0131] The AI-based assisted judging method provided in this invention also includes the implementation of virtual scene construction. Specifically, the camera pose is recovered from corresponding 2D-3D points based on real-scene video images, and the camera is correctly placed in the pre-modeled 3D tennis court scene to achieve a seamless connection between the 3D virtual scene and the real-world image. The specific steps are as follows:

[0132] (1) Camera intrinsic parameter matrix (K):

[0133] ·f x f y The focal length (in pixels) in the x and y directions is calculated from the camera's physical focal length and sensor size.

[0134] ·c x c y The coordinates (in pixels) of the optical center in the image are calculated from the image resolution.

[0135]

[0136] (2) Distortion parameters (distCoeffs):

[0137] The distortion parameters include radial distortion (k1, k2, k3) and tangential distortion (p1, p2), which are obtained by the checkerboard calibration method and are in the following form:

[0138] distCoeffs = [k1, k2, p1, p2, k3]

[0139] (3) Prepare 19 sets of 2D-3D corresponding coordinate points, which are obtained through image annotation and 3D scene annotation. The schematic diagram of 3D virtual scene annotation is shown in Figure 4.

[0140] (4) Use the solvePnP open-source algorithm to solve for the camera extrinsic rotation matrix (R) and translation vector (T).

[0141] (5) Camera cone vertical field of view (fov):

[0142]

[0143] Where h is the height of the camera sensor (in pixels), derived from the supplier's specifications.

[0144] Finally, the camera rotation matrix R, translation vector T, and vertical field of view angle fov are input into the camera of the 3D engine to restore the real camera pose in the virtual scene.

[0145] This invention also enables the prediction of the landing angle. Since the landing point is an ellipse, it needs to be directional. Based on the bound hitting position and landing point position, a trajectory is predicted, and the angle between the trajectory and the Y-axis is calculated, thus obtaining the slope of the landing ellipse. Specifically, the hitting point and the landing point are connected by a line, and the angle with the Y-axis is calculated. This angle represents the landing orientation of the ellipse.

[0146] Furthermore, during the penalty assessment, the obtained 2D coordinates of the landing point obtained from the Eagle Eye can be converted into 3D coordinates, and the landing point can be drawn in a 3D virtual scene. Simultaneously, the camera is moved to create a top-down view of the landing point, clearly showing the relationship between the landing point and the edge line, and marking IN / OUT. The penalty result displayed in the 3D virtual scene is shown in Figure 5. Specifically, when drawing the landing point ellipse, the orientation of the ellipse can be determined based on the included angle obtained during the landing point angle prediction process.

[0147] The AI-based assisted judgment method provided in this invention can significantly reduce the installation and deployment cost of the Hawk-Eye system by utilizing only a single camera combined with computer vision algorithms. Compared to solutions using multiple high-speed cameras, it is more cost-effective. Compared to other computer vision algorithm solutions, it simultaneously learns tennis ball detection and landing point recognition, resulting in more reliable tennis ball trajectories and effectively ensuring the accuracy of Hawk-Eye recognition. Furthermore, by utilizing WebGL real-time rendering technology, it produces a display effect that combines reality and virtuality, allowing Hawk-Eye results to be played within 5 seconds on mobile devices, large screens, and live streams, effectively improving the timeliness of the event.

[0148] This invention proposes an artificial intelligence-based Hawk-Eye recognition scheme. The core of the Hawk-Eye system lies in the proposed lightweight landing point recognition model. By combining tennis ball detection with landing point recognition, the time and location accuracy of landing point recognition are effectively improved. Based on this high-precision landing point, the Hawk-Eye judgment result screen combining virtual and real data is output and rendered and played on various terminals in real time, which also effectively improves the timeliness of the game.

[0149] For the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0150] Another embodiment of the present invention also provides an artificial intelligence-based real-scene eagle-eye system, the system comprising:

[0151] At least one camera device is used to film the tennis court and the tennis balls and people located in the tennis court to capture real-time footage;

[0152] The server includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the AI-based assisted judgment method as described in any of the above embodiments.

[0153] For the system implementation, since its implementation of the AI-based assisted judgment process is basically similar to that of the method implementation, the description is relatively simple. For relevant details, please refer to the description of the method implementation. It also has the corresponding technical effects.

[0154] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, any of the claimed embodiments can be used in any combination.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An artificial intelligence-based assisted judgment method, characterized in that, The method includes: acquiring continuous image frames captured in real time by a camera device, wherein the image frames include a tennis court and a tennis ball located within the tennis court; specifically, acquiring continuous image frames captured in real time by a camera device deployed on the tennis court to capture images of the tennis court, the tennis ball, and a person located within the tennis court, thereby acquiring real-time footage and obtaining multiple sets of continuous image frames; detecting the position of the tennis ball and its landing point in the tennis ball bounce frame from the continuous image frames, including: using a preset lightweight multi-task joint learning network model to detect the position of the tennis ball and its landing point in the tennis ball bounce frame from the continuous image frames; the multi-task joint learning network model includes an encoder, a feature enhancement network, and two task heads, the two task heads being for tennis ball detection. The system comprises a task head network and a landing point recognition task head network. The encoder extracts tennis ball features at different scales from the input image frames. The feature enhancement module fuses these features to obtain a tennis ball fusion feature that combines semantic and edge texture information. The tennis ball detection task head network identifies the position of the tennis ball in the image frame based on the fusion feature, while the landing point recognition task head network identifies the landing point of the tennis ball in the image frame based on the fusion feature and determines the tennis ball bounce frame. Within the bounce frame, a region of interest (ROI) is formed by taking N pixels to the left and right, and M pixels upwards and downwards, centered on the landing point. The tennis ball outline and court boundary lines are identified within the ROI. A penalty decision is generated based on the positional relationship between the tennis ball outline and the court boundary lines.

2. The method according to claim 1, characterized in that, The encoder includes a first downsampling network layer and a second downsampling network layer connected in sequence. The first downsampling network layer consists of a 3×3 standard convolutional layer and a 3×3 max pooling layer, used to extract primary feature maps from the image to achieve preliminary dimensionality reduction. The second downsampling network layer includes a downsampling module and at least three feature transformation modules. The downsampling module is used to extract tennis ball features of different scales from the primary feature map. It includes a first branch consisting of a 3×3 depthwise separable convolution and a second branch consisting of a 1×1 standard convolution, a 3×3 depthwise separable convolution, and a 1×1 standard convolution connected in sequence. The feature output channels of the first branch and the second branch are channel-mixed. Each feature transformation module is used to perform feature transformation on tennis features of different scales obtained by the downsampling module. This includes a third branch with identity mapping function set in parallel and a fourth branch consisting of a standard 1×1 convolution, a 3×3 depthwise separable convolution, and a 1×1 standard convolution connected in sequence. The feature output channels of the downsampling module are mixed and divided into two equal parts, which are used as the inputs of the third and fourth branches, respectively. The feature output channels of the third and fourth branches are then mixed.

3. The method according to claim 2, characterized in that, The feature enhancement network includes a first upsampling network layer and a second upsampling network layer connected in sequence. The first upsampling network layer includes a 3×3 RepVGG convolutional layer and a first upsampling module. The 3×3 RepVGG convolutional layer is used to enhance local feature extraction of the output features of the second downsampling network layer. The first upsampling module is used to upsample the output features of the RepVGG convolutional layer using bilinear interpolation. The upsampled feature output channels are concatenated with the feature input channels of the second downsampling network layer to obtain 72 feature output channels. The second upsampling network layer includes a 3×3 standard convolutional layer and a second upsampling module. The 3×3 standard convolutional layer is used to extract features from the output features of the 72 feature output channels of the first upsampling network layer and convert them into 48 feature output channels for output. The first upsampling module is used to upsample the current input features using bilinear interpolation. The upsampled feature output channels are concatenated with the feature output channels of the first downsampling network layer to obtain 72 feature output channels.

4. The method according to claim 1, characterized in that, The tennis ball detection task head network includes a third upsampling module, a 3×3 RepVGG convolutional layer, and a 3×3 standard convolutional layer connected in sequence. The third upsampling module is used to upsample the output features of the feature enhancement network using bilinear interpolation. The 3×3 RepVGG convolutional layer is used to enhance local feature extraction from the output features of the third upsampling module. The 3×3 standard convolutional layer is used to transform the 72-channel input feature channels into 3-channel feature output channels.

5. The method according to claim 1, characterized in that, The landing point recognition task head network includes three sets of 3×3 RepVGG convolutional layers, a max pooling layer, three sets of 3×3 RepVGG convolutional layers, a channel attention layer, and a fully connected layer, which are connected in sequence. By establishing a channel attention mechanism in the landing point recognition task head network, the channel weights of each feature input channel of the landing point recognition task head network are dynamically adjusted according to the channel attention mechanism to enhance the attention of the landing point recognition task head network to the motion features of the tennis ball in the image and output the landing point position of the tennis ball in the image frame.

6. The method according to any one of claims 1-5, characterized in that, After acquiring continuous image frames captured in real time by the camera device, the method further includes: merging a preset number of continuous image frames in the channel dimension to achieve pre-fusion of image frames.

7. The method according to any one of claims 1-5, characterized in that, The method further includes: plotting the tennis ball positions detected from multiple sets of consecutive image frames and the landing points of the tennis ball in the tennis ball bounce frames onto a preset real-world map to form a dynamic video or static image corresponding to the tennis ball landing event, which is marked with the tennis ball's trajectory before bounce, landing point, and trajectory after bounce.

8. The method according to any one of claims 1-5, characterized in that, The process of identifying the tennis ball outline and court boundary line in the region of interest includes: acquiring at least a third image frame whose acquisition time is earlier than the tennis ball bounce frame as a background frame, and selecting a region of interest in the background frame in the same manner, centered on the coordinates of the tennis ball's landing point in the tennis ball bounce frame; converting the regions of interest in the tennis ball bounce frame and the background frame into grayscale images through a color conversion operation; calculating the absolute difference between the regions of interest in the tennis ball bounce frame and the background frame using the frame difference method to obtain a local difference image, and binarizing the local difference image after absolute difference; eroding the binarized image, and performing contour detection on the eroded image to obtain the tennis ball outline; reducing the width and height of the region of interest in the tennis ball bounce frame by a preset ratio to obtain a rectangular edge region; converting the edge region into a grayscale image; binarizing the grayscale image and dilating it, and performing contour detection on the dilated image to obtain the court boundary line outline.

9. A real-scene eagle-eye system based on artificial intelligence, characterized in that, The system includes: at least one camera device for capturing images of the tennis court and tennis balls and people located within the tennis court in real time; and a server including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any one of claims 1-8.

Images