Real-time detection method and device for table tennis image target, computer equipment, storage medium and program product

Abstract

The invention relates to a table tennis image target real-time detection method and device, equipment, a storage medium and a program product, and relates to the technical field of machine vision and artificial intelligence. According to the invention, the detection precision and speed can be improved. The method comprises the following steps: obtaining an image of a standard calibration plate at each angle in a target scene shot by an optical binocular camera, obtaining a target calibration material according to the image of the standard calibration plate, calibrating the optical binocular camera through the target calibration material, and obtaining internal and external parameters and distortion parameters of the optical binocular camera; identifying and tracking a table tennis ball to be detected in the target scene to obtain an identified and tracked mark image; according to the internal and external parameters and the distortion parameters, calculating the three-dimensional coordinates of the table tennis ball in each frame of marked image shot by the optical binocular camera; according to the three-dimensional coordinates of the table tennis ball in every two frames of marked images shot by the optical binocular camera, the movement speed of the table tennis ball and the target distance between the table tennis ball and a left camera in the optical binocular camera are calculated.

Description

Technical Field

[0001] This application relates to the fields of machine vision and artificial intelligence, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for real-time detection of ping-pong ball image targets. Background Technology

[0002] With the rapid development of artificial intelligence and computer technology, techniques have emerged for recognizing and tracking ping-pong balls using single image processing algorithms or traditional machine learning algorithms. For example, current methods may rely on simple color segmentation and morphological operations to identify ping-pong balls, or use traditional target tracking algorithms such as mean-shift algorithms for tracking.

[0003] However, traditional methods often fail to perform well in handling fast-moving objects like ping-pong balls. For example, traditional mean-shift algorithms use color or edge information as features, lacking spatial information and necessary template updates, which can lead to deformation or target tracking loss when identifying and tracking fast-moving targets. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for real-time detection of ping-pong ball image targets to address the aforementioned technical problems.

[0005] Firstly, this application provides a real-time detection method for targets in ping-pong ball images, comprising:

[0006] Images of a standard calibration board at various angles in a target scene captured by an optical binocular camera are acquired. Target calibration material is obtained based on the images of the standard calibration board. The optical binocular camera is calibrated using the target calibration material to obtain the intrinsic, extrinsic, and distortion parameters of the optical binocular camera.

[0007] In response to the ping-pong ball detection command, the system identifies and tracks the ping-pong ball to be detected in the target scene, and obtains a marked image of the identification and tracking.

[0008] Based on the intrinsic and extrinsic parameters and the distortion parameters, calculate the three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera;

[0009] Based on the three-dimensional coordinates of the ping-pong ball in every two frames of the marked images captured by the optical binocular camera, the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera of the optical binocular camera are calculated.

[0010] In one embodiment, before identifying and tracking the ping-pong ball to be detected in the target scene, the method further includes: acquiring a self-built ping-pong dataset, a weight file, and a configuration file; the ping-pong dataset includes a training set, a test set, and a validation set, and the weight file is generated by training the training set;

[0011] The step of identifying and tracking the ping-pong ball to be detected in the target scene includes: identifying and tracking the ping-pong ball to be detected in the target scene using a YOLOv7-PG model that includes PConv convolution and GIoU loss function, based on the ping-pong ball dataset, the weight file, and the configuration file.

[0012] In one embodiment, the method further includes:

[0013] The initial YOLOv7 model is trained and updated using the training set and the validation set to obtain a pre-trained YOLOv7 model. Convolutional layers are replaced and the loss function is optimized in the pre-trained YOLOv7 model, and an attention mechanism is introduced to obtain an initial YOLOv7-PG model. The initial YOLOv7-PG model is tested and optimized using the test set until its accuracy on the test set meets a threshold condition, thus obtaining the final YOLOv7-PG model.

[0014] In one embodiment, obtaining the target calibration material based on the image of the standard calibration board includes:

[0015] The image of the standard calibration board is cut into left and right camera images according to the position of the optical binocular camera. The left and right camera images are denoised and grayscale processed to obtain preprocessed left and right camera images as candidate calibration materials. Corner detection is performed on the candidate calibration materials to exclude abnormal materials in the candidate calibration materials based on the corner detection results, so as to obtain the target calibration materials.

[0016] In one embodiment, calibrating the optical binocular camera using the target calibration material to obtain the intrinsic and extrinsic parameters and distortion parameters of the optical binocular camera includes:

[0017] The optical binocular camera is calibrated using the target calibration material to obtain the basic matrix and essential matrix of the optical binocular camera; based on the basic matrix and the essential matrix, the intrinsic and extrinsic parameters and the distortion parameters are determined.

[0018] In one embodiment, the method further includes:

[0019] Based on the marked image, the speed of the ping-pong ball, and the target distance, a velocity frame count line graph and a distance frame count line graph are generated; based on the identification information of the optical binocular camera, the corresponding target information entry in the information database is matched; the intrinsic and extrinsic parameters, the distortion parameters, the velocity frame count line graph, and the distance frame count line graph are entered into the target information entry in the information database for storage.

[0020] Secondly, this application also provides a real-time detection device for a ping-pong ball image target, comprising:

[0021] The image processing module is used to acquire images of a standard calibration board at various angles in the target scene captured by the optical binocular camera, obtain target calibration material based on the images of the standard calibration board, calibrate the optical binocular camera using the target calibration material, and obtain the intrinsic and extrinsic parameters and distortion parameters of the optical binocular camera.

[0022] The identification and tracking module is used to identify and track the ping-pong ball to be detected in the target scene in response to the ping-pong ball detection command, and obtain the identified and tracked marked image;

[0023] The coordinate calculation module is used to calculate the three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera based on the intrinsic and extrinsic parameters and the distortion parameters.

[0024] The distance calculation module is used to calculate the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera of the optical binocular camera based on the three-dimensional coordinates of the ping-pong ball in every two frames of the marked images captured by the optical binocular camera.

[0025] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0026] Images of a standard calibration board at various angles in a target scene captured by an optical binocular camera are acquired. Target calibration material is obtained based on the images of the standard calibration board. The optical binocular camera is calibrated using the target calibration material to obtain its intrinsic and extrinsic parameters and distortion parameters. In response to a ping-pong ball detection command, a ping-pong ball to be detected in the target scene is identified and tracked, resulting in identified and tracked marked images. Based on the intrinsic and extrinsic parameters and the distortion parameters, the three-dimensional coordinates of the ping-pong ball in each frame of the marked images captured by the optical binocular camera are calculated. Based on the three-dimensional coordinates of the ping-pong ball in every two frames of the marked images captured by the optical binocular camera, the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical binocular camera are calculated.

[0027] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0028] Images of a standard calibration board at various angles in a target scene captured by an optical binocular camera are acquired. Target calibration material is obtained based on the images of the standard calibration board. The optical binocular camera is calibrated using the target calibration material to obtain its intrinsic and extrinsic parameters and distortion parameters. In response to a ping-pong ball detection command, a ping-pong ball to be detected in the target scene is identified and tracked, resulting in identified and tracked marked images. Based on the intrinsic and extrinsic parameters and the distortion parameters, the three-dimensional coordinates of the ping-pong ball in each frame of the marked images captured by the optical binocular camera are calculated. Based on the three-dimensional coordinates of the ping-pong ball in every two frames of the marked images captured by the optical binocular camera, the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical binocular camera are calculated.

[0029] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0030] Images of a standard calibration board at various angles in a target scene captured by an optical binocular camera are acquired. Target calibration material is obtained based on the images of the standard calibration board. The optical binocular camera is calibrated using the target calibration material to obtain its intrinsic and extrinsic parameters and distortion parameters. In response to a ping-pong ball detection command, a ping-pong ball to be detected in the target scene is identified and tracked, resulting in identified and tracked marked images. Based on the intrinsic and extrinsic parameters and the distortion parameters, the three-dimensional coordinates of the ping-pong ball in each frame of the marked images captured by the optical binocular camera are calculated. Based on the three-dimensional coordinates of the ping-pong ball in every two frames of the marked images captured by the optical binocular camera, the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical binocular camera are calculated.

[0031] The aforementioned real-time detection method, apparatus, computer equipment, computer-readable storage medium, and computer program product for ping-pong ball images first calibrates an optical binocular camera using target calibration materials to obtain the camera's intrinsic and extrinsic parameters and distortion parameters. Then, it identifies and tracks the ping-pong ball to be detected in the target scene, obtaining marked images for recognition and tracking. Next, based on the endo- and extrinsic parameters and distortion parameters, it calculates the 3D coordinates of the ping-pong ball in each frame of marked images captured by the optical binocular camera in real time. Finally, based on the 3D coordinates of the ping-pong ball in every two frames of marked images captured by the optical binocular camera, it calculates the ping-pong ball's speed and the target distance between the ping-pong ball and the left camera in the optical binocular camera in real time. This optimizes small target recognition, i.e., ping-pong ball recognition, and balances the trade-off between accuracy and speed in object recognition. This application uses a YOLOv7-PG model containing PConv convolution and GIoU loss function for real-time recognition and tracking of ping-pong ball motion, thereby improving detection accuracy and speed, addressing the problem of poor detection performance for fast-moving ping-pong balls, and effectively solving the problems of deformation or target tracking loss. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is an application environment diagram of a real-time target detection method for ping-pong ball images in one embodiment;

[0034] Figure 2 This is a physical schematic diagram of an optical binocular camera in one embodiment;

[0035] Figure 3 This is a flowchart illustrating a real-time target detection method for a ping-pong ball image in one embodiment.

[0036] Figure 4 This is a schematic diagram of the PConv convolution structure in one embodiment;

[0037] Figure 5 This is a partial structural diagram of an improved version of the Pconv convolution in a YOLOv7 model, as shown in one embodiment.

[0038] Figure 6 This is a schematic diagram of the ECA attention module in one embodiment;

[0039] Figure 7 This is a schematic diagram of the CBAM attention module in one embodiment;

[0040] Figure 8 Here is a feature channel structure diagram of the CBAM attention module in one embodiment;

[0041] Figure 9 Here is a feature space structure diagram of the CBAM attention module in one embodiment;

[0042] Figure 10 This is a structural diagram of an improved part of the YOLOv7 model with an added attention module in one embodiment;

[0043] Figure 11 This is a schematic diagram of the structure of a YOLOv7 model in one embodiment;

[0044] Figure 12 This is a schematic diagram of a standard calibration plate in one embodiment;

[0045] Figure 13 This is a flowchart illustrating a real-time target detection method for a ping-pong ball image in a specific embodiment.

[0046] Figure 14 This is a screenshot of the interface of a YOLOv7 model in one embodiment;

[0047] Figure 15 This is a structural block diagram of a real-time detection device for a ping-pong ball image target in one embodiment;

[0048] Figure 16 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] The real-time target detection method for ping-pong ball images provided in this application embodiment can be applied to, for example... Figure 1 The application environment shown is as follows. The optical binocular camera transmits images to the operating module via a network. The optical binocular camera consists of a left camera, a right camera, and a synchronization chip. Each of the left and right cameras has one camera to capture images of the same scene from two different perspectives. The operating module includes a processor and a memory, which are coupled together. The memory stores the data that the processor needs to process.

[0051] Specifically, the real-time detection method for ping-pong ball image targets provided in this application embodiment can be executed by a processor.

[0052] For example, the processor acquires images of a standard calibration board at various angles in the target scene captured by the optical binocular camera, obtains target calibration material based on the images of the standard calibration board, calibrates the optical binocular camera using the target calibration material, and obtains the intrinsic and extrinsic parameters and distortion parameters of the optical binocular camera; in response to a ping-pong ball detection command, it identifies and tracks the ping-pong ball to be detected in the target scene, and obtains the identified and tracked marked images; then, based on the intrinsic and extrinsic parameters and distortion parameters, it calculates the three-dimensional coordinates of the ping-pong ball in each frame of marked images captured by the optical binocular camera; finally, based on the three-dimensional coordinates of the ping-pong ball in every two frames of marked images captured by the optical binocular camera, it calculates the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical binocular camera.

[0053] In such Figure 1 In the application environment shown, such as Figure 2 As shown, an optical binocular camera can be a device that uses two cameras (cameras) to simulate the human eye's perspective. The processor can be a separate processor or a processor cluster consisting of multiple processors. It should be noted that in this application, if the same group of devices performs multiple identification and tracking of ping-pong balls at the same location, the binocular camera calibration step can be omitted in subsequent workflows, except for the first one; if the same binocular camera is used for camera calibration at the same location, the step of capturing and collecting corner detection source images can be omitted.

[0054] In one embodiment, such as Figure 3 As shown, a real-time target detection method for ping-pong ball images is provided, which can be applied to... Figure 1 The processor in the process may include the following steps:

[0055] Step S301: Acquire images of a standard calibration board at various angles in the target scene captured by the optical binocular camera; obtain target calibration material based on the images of the standard calibration board; calibrate the optical binocular camera using the target calibration material; and obtain the intrinsic and extrinsic parameters and distortion parameters of the optical binocular camera.

[0056] The standard calibration board can be a board made up of black and white squares, such as... Figure 4 As shown.

[0057] The intrinsic and extrinsic parameters consist of intrinsic parameters and extrinsic parameters. The intrinsic parameters include focal length, optical center, and pixel size, while the extrinsic parameters include rotation matrix and translation vector. The distortion parameters include radial distortion parameters and tangential distortion parameters.

[0058] Calibration is the process of determining camera parameters (such as intrinsic and extrinsic parameters and distortion parameters), which is crucial for accurate image processing and analysis.

[0059] Specifically, the processor acquires images of the standard calibration board from various angles in the target scene captured by the optical binocular camera. Based on the shooting angle of the images and the positions of the left and right cameras, the images of the standard calibration board are segmented into left and right camera images. Then, the left and right camera images are preprocessed, and the preprocessed images are used as candidate calibration materials. The candidate calibration materials are then filtered to exclude abnormal materials that cannot detect normal corner points, thus obtaining target calibration materials. The optical binocular camera is calibrated using the target calibration materials to obtain the intrinsic, extrinsic, and distortion parameters of the optical binocular camera.

[0060] In step S302, in response to the ping-pong ball detection command, the ping-pong ball to be detected in the target scene is identified and tracked to obtain a marked image of identification and tracking.

[0061] Labeled images typically involve using machine vision technology to detect specific objects (such as ping-pong balls) in a target scene and track the movement of those objects in space. Labeled images can be pictures or videos.

[0062] Specifically, the processor uses a YOLOv7-PG model that includes PConv convolution and GIoU loss function to identify and track ping-pong balls in the target scene, resulting in a labeled image of the identified and tracked ping-pong balls.

[0063] Step S303: Calculate the three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera based on the intrinsic and extrinsic parameters and distortion parameters.

[0064] Three-dimensional coordinates are a coordinate system used to describe the position of a point in three-dimensional space, and are usually represented by three coordinate values: x, y, and z.

[0065] Specifically, the processor calculates the three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera based on the intrinsic and extrinsic parameters and distortion parameters obtained by calibrating the optical binocular camera.

[0066] Step S304: Calculate the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical binocular camera based on the three-dimensional coordinates of the ping-pong ball in every two frames of marked images captured by the optical binocular camera.

[0067] Specifically, the processor acquires each pair of adjacent marked images captured by the optical binocular camera, and calculates the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical binocular camera based on the three-dimensional coordinates of the ping-pong ball in each pair of adjacent marked images.

[0068] In this embodiment, the optical stereo camera is first calibrated using target calibration materials to obtain its intrinsic and extrinsic parameters and distortion parameters. Then, a ping-pong ball to be detected in the target scene is identified and tracked, resulting in marked images for recognition and tracking. Based on the extrinsic and extrinsic parameters and distortion parameters, the 3D coordinates of the ping-pong ball in each frame of marked images captured by the optical stereo camera are calculated in real time. Finally, based on the 3D coordinates of the ping-pong ball in every two frames of marked images captured by the optical stereo camera, the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical stereo camera are calculated in real time. This optimizes small target recognition, i.e., ping-pong ball recognition, and balances the trade-off between accuracy and speed in object recognition. This application uses a YOLOv7-PG model containing PConv convolution and GIoU loss function for real-time recognition and tracking of ping-pong ball motion, thereby improving detection accuracy and speed, addressing the problem of poor detection performance for fast-moving ping-pong balls, and effectively solving the problems of deformation or target tracking loss.

[0069] In one embodiment, before identifying and tracking the ping-pong ball to be detected in the target scene, the method of this application further includes the following steps:

[0070] Obtain the self-built table tennis dataset, weight file, and configuration file; the table tennis dataset includes a training set, a test set, and a validation set, and the weight file is generated by training on the training set;

[0071] In step S302 above, identifying and tracking the ping-pong ball to be detected in the target scene may include the following steps:

[0072] Based on the table tennis dataset, weight file, and configuration file, the YOLOv7-PG model, which includes PConv convolution and GIoU loss function, is used to identify and track table tennis balls to be detected in the target scene.

[0073] The self-built table tennis dataset can be composed of image data taken in real-world scenes, combined with some open-source datasets from the internet. This dataset also contains data corresponding to table tennis balls of various colors.

[0074] Among them, the YOLOv7-PG model is a deep learning-based object detection model, which is an improvement on the YOLOv7 model and has better real-time performance and accuracy.

[0075] The weight file data is generated by running the YOLOv7-PG model and trained on the training set in the self-built dataset; the configuration file data is the data in the configuration file corresponding to the YOLOv7-PG model.

[0076] Specifically, the processor acquires its self-built ping-pong dataset, weight file, and configuration file. Then, based on the ping-pong dataset, weight file, and configuration file, it identifies and tracks the ping-pong balls to be detected in the target scene using a YOLOv7-PG model that includes PConv convolution and GIoU loss function.

[0077] The YOLOv7-PG model in this embodiment improves detection accuracy and speed by replacing the Conv3x3 convolution in the original YOLOv7 model with PConv convolution and replacing the original loss function with the GIoU loss function. This model performs better on the self-built table tennis dataset than the existing large dataset MS COCO or the original YOLOv7 model.

[0078] In one embodiment, the method of this application further includes the following steps:

[0079] The initial YOLOv7 model is trained and updated using the training and validation sets to obtain a pre-trained YOLOv7 model. Convolutional layers are replaced and the loss function is optimized in the pre-trained YOLOv7 model. An attention mechanism is also introduced into the pre-trained YOLOv7 model to obtain the initial YOLOv7-PG model. The initial YOLOv7-PG model is then tested and optimized using the test set until its accuracy on the test set meets a threshold condition, thus obtaining the YOLOv7-PG model.

[0080] Specifically, the core improvements to the initially trained YOLOv7 model in this embodiment are as follows:

[0081] (1) PConv convolutions are used to replace some Conv3x3 convolutions in the head network of the YOLOv7 model to optimize the ELAN (Efficient Layer Aggregation Networks) structure, thereby improving the algorithm's feature extraction capability, reducing the number of parameters, and increasing the detection speed. The structure of the PConv convolution is as follows: Figure 5 As shown, the FLOPs (Floating Point Operations Per Second) and memory accesses of Conv3x3 convolution and PConv convolution are as follows:

[0082]

[0083]

[0084]

[0085]

[0086] exist Figure 5 In this context, Input represents the input feature map, which is a three-dimensional tensor with dimensions h×w×c. p Where h is the height, w is the width, and c is the height. p It represents the number of channels. p Filters (convolution kernels) are also three-dimensional tensors with a size of k×k×c. p (Here k=1, so it's a 1×1 convolution kernel), "c p "c convolutional kernels" indicates that the final output channel count is also c. p (To change the output channels, adjust the number of convolution kernels.) Each convolution kernel performs a convolution operation with all channels of the input feature map. * indicates a convolution operation. A 1×1 convolution iterates through the input feature map point by point, performing a weighted sum with the input data at the corresponding position using the convolution kernel. Since the convolution kernel's spatial size is 1×1, it only changes the information exchange across the channel dimensions, without altering the h and w dimensions of the feature map. Output represents the output feature map, which still has dimensions h×w×c. p The number of channels is the same as the input (because the number of convolution kernels is c). p Information fusion between channels is achieved through 1×1 convolution (such as weighted combination of features from different channels).

[0087] Generally, c (number of channels) in the above formula is c p If PConv's FLOPs are four times that of Conv3x3, then PConv's FLOPs are only four times that of Conv3x3. PConv's memory access is Conv3x3. Therefore, it is feasible to improve the detection speed of the YOLOv7 model by using PConv convolutions of FasterNet (an efficient neural network model).

[0088] In this embodiment, on the neck network, the two consecutive Conv convolutions with unchanged input-output dimensions on the four ELAN-H structures are replaced with PConv convolutions. Figure 6 and Figure 12 The comparison shows that the two middle ones of the four Conv convolutional groups with a kernel of 3 and a stride of 1 are replaced with PConv.

[0089] (2) Introducing attention mechanisms, this embodiment selects two widely used attention mechanisms: ECA attention module and CBAM attention module. The structure of the ECA attention module is as follows: Figure 7 As shown, the structure of the CBAM attention module is as follows: Figure 8 As shown. By Figure 8As can be seen, the CBAM attention mechanism combines attention mechanisms in two dimensions: feature channels and feature space. The structure diagram of the two dimensions is as follows: Figure 9 and Figure 10 As shown.

[0090] To incorporate the aforementioned attention mechanism into the model, this approach requires the following modifications to the initially trained YOLOv7 model:

[0091] 1. Add the corresponding attention mechanism description to the models / common.py file of the model;

[0092] 2. Add the corresponding attention mechanism module name to the models / yolo.py file of the model;

[0093] 3. Modify the yolov7.yaml configuration file.

[0094] In this embodiment, attention mechanisms are added after the four ELAN structures in the backbone network. Specifically, attention mechanisms are added after the critical convolutional layers 11, 24, 37, and 50. Figure 12 The model structure after inserting the attention mechanism is as follows: Figure 11 As shown.

[0095] (3) Optimization of the loss function. In the original YOLOv7 model, the confidence loss and class loss are performed using binary cross-entropy, while the localization loss is performed using CIoU. In this embodiment, the WIoU loss function is used. The WIoU loss function is a new dynamic non-monotonic focusing mechanism. This mechanism uses "outlier" instead of traditional IoU to evaluate the quality of anchor boxes and introduces a gradient gain allocation strategy, which allows WIoU to more effectively process anchor boxes of average quality, thereby improving the overall performance of the detector. In addition, two versions of the WIoU loss function with the parameter monotonous set to False and True were selected.

[0096] To achieve this improvement, this embodiment requires the following modifications to the initially trained YOLOv7 model:

[0097] 1. First, you need to add the corresponding bbiox expression for the loss function to the utils / general.py file and select the corresponding bbiox;

[0098] 2. Modify the `ComputeLoss` and `ComputeLossOTA` functions in the `utils / loss.py` file to adapt them to the new loss function;

[0099] 3. In the utils / general.py file, select the corresponding version for the expression of WIoU, that is, modify the parameter monotonous.

[0100] In one embodiment, step S301 above, obtaining the target calibration material based on the image of the standard calibration board, specifically includes the following steps:

[0101] The image of the standard calibration board is cut into left and right camera images according to the position of the optical binocular camera. The left and right camera images are denoised and grayscale processed to obtain preprocessed left and right camera images as candidate calibration materials. Corner detection is performed on the candidate calibration materials to eliminate abnormal materials in the candidate calibration materials based on the corner detection results, so as to obtain the target calibration materials.

[0102] The noise reduction process includes removing pepper noise, salt noise, and salt-and-pepper noise from the left and right camera images, while the grayscale processing can convert the color left and right camera images into grayscale images.

[0103] Corner detection can be the process of identifying the location of corners in an image. A corner is a point in an image where two edges meet, and these edges can be straight lines, curves, or other structures in the image.

[0104] Specifically, the processor first segments the image of the standard calibration board into left and right camera images according to the position of the optical binocular camera. Then, it performs noise reduction and grayscale processing on the left and right camera images to obtain pre-processed left and right camera images as candidate calibration materials. Then, it performs corner detection on the candidate calibration materials to exclude abnormal materials in the candidate calibration materials based on the corner detection results, thereby quickly and accurately selecting the target calibration materials.

[0105] In one embodiment, step S301 above involves calibrating the optical binocular camera using target calibration materials to obtain the intrinsic and extrinsic parameters and distortion parameters of the optical binocular camera. This specifically includes the following steps:

[0106] The optical stereo camera is calibrated using target calibration materials to obtain its fundamental and essential matrices; based on these matrices, the intrinsic and extrinsic parameters and distortion parameters are determined.

[0107] Specifically, in the calibration of a stereo camera, the processor first calibrates the optical stereo camera using target calibration materials to obtain the basic matrix and essential matrix of the optical stereo camera; then, based on the basic matrix and essential matrix, it determines the intrinsic and extrinsic parameters and distortion parameters.

[0108] In one embodiment, the method of this application further includes the following steps:

[0109] Based on the marked image, the speed of the ping-pong ball, and the target distance, generate velocity frame count line graphs and distance frame count line graphs; match the corresponding target information entries in the information database based on the identification information of the optical binocular camera; and save the intrinsic and extrinsic parameters, distortion parameters, velocity frame count line graphs, and distance frame count line graphs into the target information entries in the information database.

[0110] Specifically, the processor generates velocity frame count line graphs and distance frame count line graphs based on the marked image, motion speed, and target distance, respectively; then, based on the identification information of the optical binocular camera, it matches the corresponding target information entries in the information database; and finally, it records the intrinsic and extrinsic parameters, distortion parameters, velocity frame count line graphs, and distance frame count line graphs into the target information entries in the information database for storage.

[0111] In this embodiment, by using the identification information of the optical binocular camera as the division standard, the result files containing intrinsic and extrinsic parameters, distortion parameters, velocity frame rate line graphs, and distance frame rate line graphs are entered into the information database for storage in a regular and orderly manner, which is conducive to the efficient storage and subsequent use of the result files.

[0112] In one embodiment, such as Figure 13 As shown, a real-time target detection method for a ping-pong ball image is provided in a specific embodiment, which includes the following steps:

[0113] Step S1301: Acquire images of the standard calibration board at various angles in the target scene captured by the optical binocular camera. Cut the images of the standard calibration board into left and right camera images according to the position of the optical binocular camera. Perform noise reduction and grayscale processing on the left and right camera images to obtain preprocessed left and right camera images as candidate calibration materials. Perform corner detection on the candidate calibration materials to exclude abnormal materials in the candidate calibration materials based on the corner detection results, and obtain the target calibration materials.

[0114] Step S1302: The optical binocular camera is calibrated using target calibration materials to obtain the basic matrix and essential matrix of the optical binocular camera; based on the basic matrix and essential matrix, the intrinsic and extrinsic parameters and distortion parameters are determined.

[0115] Step S1303: In response to the ping-pong ball detection command, obtain the self-built ping-pong ball dataset, weight file, and configuration file; based on the ping-pong ball dataset, weight file, and configuration file, identify and track the ping-pong balls to be detected in the target scene using the YOLOv7-PG model containing PConv convolution and GIoU loss function.

[0116] Step S1304: Calculate the three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera based on the intrinsic and extrinsic parameters and distortion parameters; calculate the ping-pong ball's speed and the target distance between the ping-pong ball and the left camera in the optical binocular camera based on the three-dimensional coordinates of the ping-pong ball in every two frames of the marked image captured by the optical binocular camera.

[0117] Step S1305: Generate a velocity frame count line graph and a distance frame count line graph based on the marked image, the speed of the ping-pong ball, and the target distance; match the corresponding target information entries in the information database based on the identification information of the optical binocular camera; and save the intrinsic and extrinsic parameters, distortion parameters, velocity frame count line graph, and distance frame count line graph into the target information entries in the information database.

[0118] The beneficial effects of the above embodiments are as follows:

[0119] 1. Self-built table tennis dataset: In order to improve the accuracy of the algorithm, this embodiment uses a self-built table tennis dataset, which includes data from real-world shooting and some open-source datasets from the Internet. Such a dataset is closer to the actual application scenario, improving the practicality and accuracy of the model.

[0120] 2. YOLOv7-PG Algorithm: This embodiment proposes an improved YOLOv7 algorithm, namely the YOLOv7-PG algorithm. By replacing the original Conv3x3 convolution with PConv convolution and the original loss function with the GIoU loss function, the detection accuracy and speed are improved. This algorithm outperforms the original YOLOv7 algorithm on both the self-built table tennis dataset and the existing large dataset MSCOCO.

[0121] 3. Model Structure Optimization and Innovation: This embodiment innovatively replaces some traditional convolutional layers in the YOLOv7 model with PConv convolutional layers. This replacement not only reduces the number of model parameters and computational complexity but also improves the ability to extract features from small targets like ping-pong balls. This allows the model to maintain high detection accuracy while achieving faster detection speed, better adapting to the detection challenges posed by the rapid movement, small size, and reflectivity of ping-pong balls. Furthermore, replacing some traditional convolutional layers with PConv convolutions enhances the model's ability to extract features from small targets like ping-pong balls, enabling more accurate capture of detailed features and changes during movement. This effectively solves the detection difficulties caused by the small size and reflective surface of ping-pong balls, reducing the probability of missed and false detections.

[0122] 4. Improved Loss Function: By introducing the GIoU loss function to replace the original loss function, this embodiment can more accurately measure the difference between the predicted bounding box and the ground truth bounding box during model training, thereby effectively improving the model's accuracy in ping-pong ball positioning. This improvement is particularly helpful in solving the problem of blurred and difficult-to-define boundaries when the ping-pong ball is moving at high speed, making the model's detection of ping-pong balls more accurate and reliable in complex scenes. Furthermore, the use of the GIoU loss function enables the model to more accurately locate the ping-pong ball in complex backgrounds and lighting conditions, improving the model's performance in recognizing and tracking moving ping-pong balls. This helps to better handle various complex situations that may occur in real-world scenarios, ensuring the stability and reliability of ping-pong ball recognition. The interface of the YOLOv7 model is shown in the figure below. Figure 14 As shown.

[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0124] Based on the same inventive concept, this application also provides a real-time detection device for ping-pong image targets, used to implement the real-time detection method for ping-pong image targets described above. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the real-time detection device for ping-pong image targets provided below can be found in the limitations of the real-time detection method for ping-pong image targets described above, and will not be repeated here.

[0125] In one exemplary embodiment, such as Figure 15 As shown, a real-time target detection device for ping-pong ball images is provided, which may include:

[0126] Image processing module 1501 is used to acquire images of a standard calibration board at various angles in the target scene captured by an optical binocular camera, obtain target calibration material based on the images of the standard calibration board, calibrate the optical binocular camera using the target calibration material, and obtain the intrinsic and extrinsic parameters and distortion parameters of the optical binocular camera.

[0127] The recognition and tracking module 1502 is used to respond to the ping-pong ball detection command, recognize and track the ping-pong ball to be detected in the target scene, and obtain the recognition and tracking marked image;

[0128] The coordinate calculation module 1503 is used to calculate the three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera based on the intrinsic and extrinsic parameters and distortion parameters.

[0129] The distance calculation module 1504 is used to calculate the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera in the optical binocular camera based on the three-dimensional coordinates of the ping-pong ball in every two frames of marked images captured by the optical binocular camera.

[0130] In one embodiment, the device may further include: a data acquisition module for acquiring a self-built ping-pong dataset, a weight file, and a configuration file; the ping-pong dataset includes a training set, a test set, and a validation set, and the weight file is generated by training the training set; and a recognition and tracking module 1502 for recognizing and tracking ping-pong balls to be detected in a target scene using a YOLOv7-PG model that includes PConv convolution and GIoU loss functions, based on the ping-pong dataset, the weight file, and the configuration file.

[0131] In one embodiment, the apparatus may further include: a model training module, configured to train and update an initial YOLOv7 model using a training set and a validation set to obtain a pre-trained YOLOv7 model; to replace convolutional layers and optimize the loss function in the pre-trained YOLOv7 model, and to introduce an attention mechanism into the pre-trained YOLOv7 model to obtain an initial YOLOv7-PG model; and to test and optimize the initial YOLOv7-PG model using a test set until the accuracy metric of the initial YOLOv7-PG model on the test set meets a threshold condition to obtain the YOLOv7-PG model.

[0132] In one embodiment, the image processing module 1501 is further configured to cut the image of the standard calibration board into left and right camera images according to the position of the optical binocular camera, perform noise reduction and grayscale processing on the left and right camera images to obtain pre-processed left and right camera images as candidate calibration materials; perform corner detection on the candidate calibration materials to exclude abnormal materials in the candidate calibration materials according to the corner detection results, and obtain target calibration materials.

[0133] In one embodiment, the image processing module 1501 is further configured to calibrate the optical binocular camera using target calibration materials to obtain the fundamental matrix and essential matrix of the optical binocular camera; and determine the intrinsic and extrinsic parameters and distortion parameters based on the fundamental matrix and essential matrix.

[0134] In one embodiment, the device may further include: a data storage module, configured to generate a velocity frame count line graph and a distance frame count line graph based on the marked image, the speed of the ping-pong ball, and the target distance; match the corresponding target information entry in the information database based on the identification information of the optical binocular camera; and input the intrinsic and extrinsic parameters, distortion parameters, velocity frame count line graph, and distance frame count line graph into the target information entry in the information database for storage.

[0135] Each module in the aforementioned real-time target detection device for ping-pong images can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0136] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 15 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a real-time detection method for ping-pong ball image targets. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0137] Those skilled in the art will understand that Figure 16The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0138] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0139] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0140] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0143] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0144] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for real-time detection of table tennis image target, characterized in that, The method comprises: acquiring images of standard calibration boards at various angles in a target scene captured by an optical binocular camera, obtaining target calibration materials from the images of the standard calibration boards, calibrating the optical binocular camera through the target calibration materials, and obtaining internal and external parameters and distortion parameters of the optical binocular camera; in response to a ping-pong ball detection instruction, recognizing and tracking a ping-pong ball to be detected in the target scene to obtain a marked image of the recognition and tracking; calculating three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera according to the internal and external parameters and the distortion parameters; calculating a motion speed of the ping-pong ball and a target distance between the ping-pong ball and a left camera in the optical binocular camera according to the three-dimensional coordinates of the ping-pong ball in every two frames of the marked image captured by the optical binocular camera.

2. The method of claim 1, wherein, Before the recognition and tracking of the ping-pong ball to be detected in the target scene, the method further comprises: acquiring a self-built ping-pong ball dataset, a weight file, and a configuration file; the ping-pong ball dataset comprises a training set, a test set, and a validation set, and the weight file is generated by training the training set; the recognition and tracking of the ping-pong ball to be detected in the target scene comprises: recognizing and tracking the ping-pong ball to be detected in the target scene through a YOLOv7-PG model containing PConv convolution and GIoU loss function according to the ping-pong ball dataset, the weight file, and the configuration file.

3. The method of claim 2, wherein, The method further comprises: training and updating an initial YOLOv7 model using the training set and the validation set to obtain a preliminarily trained YOLOv7 model; replacing a convolution layer and optimizing a loss function of the preliminarily trained YOLOv7 model, and introducing an attention mechanism into the preliminarily trained YOLOv7 model to obtain an initial YOLOv7-PG model; testing and optimizing the initial YOLOv7-PG model using the test set until an accuracy index of the initial YOLOv7-PG model on the test set meets a threshold condition to obtain the YOLOv7-PG model.

4. The method of claim 1, wherein, The obtaining of the target calibration materials from the images of the standard calibration boards comprises: cutting the images of the standard calibration boards into left and right camera images according to positions of the optical binocular camera, performing denoising processing and grayscale processing on the left and right camera images to obtain preprocessed left and right camera images as candidate calibration materials; performing corner point detection on the candidate calibration materials to exclude abnormal materials in the candidate calibration materials according to a corner point detection result to obtain the target calibration materials.

5. The method of claim 1, wherein, The calibration of the optical binocular camera through the target calibration materials to obtain the internal and external parameters and the distortion parameters of the optical binocular camera comprises: calibrating the optical binocular camera through the target calibration materials to obtain an essential matrix and a fundamental matrix of the optical binocular camera; determining the internal and external parameters and the distortion parameters according to the essential matrix and the fundamental matrix.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: Based on the marked image, the speed of the ping-pong ball, and the target distance, generate a velocity frame count line graph and a distance frame count line graph; Based on the identification information of the optical binocular camera, match the corresponding target information entry in the information database; The intrinsic and extrinsic parameters, the distortion parameters, the velocity frame rate graph, and the distance frame rate graph are entered into the target information entry in the information database and saved.

7. A real-time detection device of table tennis image target, characterized in that, The device includes: The image processing module is used to acquire images of a standard calibration board at various angles in the target scene captured by the optical binocular camera, obtain target calibration material based on the images of the standard calibration board, calibrate the optical binocular camera using the target calibration material, and obtain the intrinsic and extrinsic parameters and distortion parameters of the optical binocular camera. The identification and tracking module is used to identify and track the ping-pong ball to be detected in the target scene in response to the ping-pong ball detection command, and obtain the identified and tracked marked image; The coordinate calculation module is used to calculate the three-dimensional coordinates of the ping-pong ball in each frame of the marked image captured by the optical binocular camera based on the intrinsic and extrinsic parameters and the distortion parameters. The distance calculation module is used to calculate the speed of the ping-pong ball and the target distance between the ping-pong ball and the left camera of the optical binocular camera based on the three-dimensional coordinates of the ping-pong ball in every two frames of the marked images captured by the optical binocular camera.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images