Drop position determination method, device, storage medium and computer program product

Abstract

The application provides a landing point position determination method, device, storage medium and computer program product. The method comprises: performing image processing on multi-view images to obtain first two-dimensional coordinate information corresponding to a ball table in each view image and second two-dimensional coordinate information corresponding to a moving ball in each view image; the multi-view images are multi-view images of the same region; obtaining first three-dimensional coordinate information corresponding to the ball table; determining a first projection matrix corresponding to the ball table in each view based on the first two-dimensional coordinate information and the first three-dimensional coordinate information; determining second three-dimensional coordinate information corresponding to the moving ball according to the first projection matrix and the second two-dimensional coordinate information; and determining a three-dimensional motion trajectory and a landing point position corresponding to the moving ball based on the second three-dimensional coordinate information. The accuracy of the three-dimensional coordinates of the ball can be improved, and the accuracy of the three-dimensional trajectory and the landing point position of the ball can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, device, storage medium and computer program product for determining the landing point. Background Technology

[0002] Calculating the 3D trajectory of a ping-pong ball requires multi-view stereo vision matching technology, typically using a binocular vision system that mimics human vision. This involves calculating the 3D spatial coordinates of the ping-pong ball by identifying corresponding points from different viewpoints. The calculation process requires obtaining the camera's intrinsic and extrinsic parameters. A common method for acquiring these parameters is calibration using a calibration board, which is a laborious and technically demanding process. Furthermore, a large amount of multi-view data is needed to train the ping-pong ball detection model to recognize the ball from different perspectives. Because of the ball's high speed, ordinary cameras often detect significant blurring of the ball's motion, resulting in low accuracy in calculating the ball's 3D coordinates, which in turn leads to low accuracy in obtaining the ball's 3D trajectory and landing point. Summary of the Invention

[0003] This application provides a method, device, storage medium, and computer program product for determining the landing point position, which can improve the accuracy of the three-dimensional coordinates of a sphere, thereby improving the accuracy of the sphere's three-dimensional trajectory and landing point position.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides a method for determining the landing point location, the method including: Image processing is performed on the multi-view images to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop contained in each view image and the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image; the multi-view images are multi-view images of the same area. Obtain the first three-dimensional coordinate information corresponding to the spherical tabletop; Based on the first two-dimensional coordinate information and the first three-dimensional coordinate information, determine the first projection matrix of the spherical tabletop at each viewpoint; The second three-dimensional coordinate information corresponding to the moving sphere is determined based on the first projection matrix and the second two-dimensional coordinate information; and the three-dimensional motion trajectory and landing position of the moving sphere are determined based on the second three-dimensional coordinate information.

[0005] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the landing point determination method provided in the embodiments of this application.

[0006] This application provides a storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the landing point determination method provided in this application.

[0007] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the landing point determination method provided in this application.

[0008] The embodiments of this application have the following beneficial effects: By processing multi-view images of the same area, the first two-dimensional coordinates of the spherical tabletop and the second two-dimensional coordinates of the moving sphere are obtained in each viewpoint image. The first three-dimensional coordinates of the spherical tabletop are then obtained. Based on these first two-dimensional and third-dimensional coordinates, a first projection matrix of the spherical tabletop at each viewpoint is determined. The second three-dimensional coordinates of the moving sphere are then determined based on the first projection matrix and the second two-dimensional coordinates. The three-dimensional trajectory and landing position of the moving sphere are determined based on the second three-dimensional coordinates. Furthermore, by considering the projection relationship between the first two-dimensional and first three-dimensional coordinates of the spherical tabletop, the second three-dimensional coordinates of the moving sphere are determined, thus obtaining the three-dimensional trajectory and landing position. This method eliminates the need for manual calibration of the camera's intrinsic and extrinsic parameters. The second three-dimensional coordinates are determined jointly by the first projection matrix and the second two-dimensional coordinates of the moving sphere at each viewpoint, taking into account the position of the moving sphere at multiple viewpoints. This improves the accuracy of the sphere's three-dimensional coordinates, thereby enhancing the accuracy of its three-dimensional trajectory and landing position. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a method for determining the landing point location provided in an embodiment of this application; Figure 2 A flowchart illustrating an exemplary self-calibrated 3D trajectory and landing point detection system for ping-pong balls, provided for embodiments of this application; Figure 3 A schematic diagram illustrating the setting of key points on an exemplary ping-pong table, provided for an embodiment of this application; Figure 4 A schematic diagram of an exemplary deep learning model architecture for predicting key points of a ping-pong table based on a ResNet structure, provided for embodiments of this application; Figure 5 This is a schematic diagram of the structure of a landing point determination device provided in an embodiment of this application; Figure 6This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0012] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first," "second," etc., used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," etc., may be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] Calculating the 3D trajectory of a ping-pong ball requires multi-view stereo vision matching technology, typically using a binocular vision system that mimics human vision. This system calculates the 3D spatial coordinates of the ping-pong ball by identifying corresponding points from different viewpoints. The calculation process first requires obtaining the camera's intrinsic and extrinsic parameters, commonly achieved through calibration using a calibration board. Furthermore, a large amount of multi-view data is needed to train the ping-pong ball detection model, enabling it to identify the ball from different perspectives. Accurate target detection is fundamental to successful 3D reconstruction. However, calculating the 3D coordinates of the ping-pong ball requires knowledge of the camera's intrinsic and extrinsic parameters. Manual calibration using a calibration board requires specialized knowledge and is tedious, hindering practical application. Secondly, the target detection model needs to be trained to detect high-speed ping-pong balls. Due to the high ball speed, ordinary cameras often produce significant blurring, necessitating specialized high-speed motion cameras, increasing both cost and training difficulty. Finally, the number of cameras significantly impacts the accuracy and complexity of the calculation; both efficiency and precision must be balanced. While monocular vision is more efficient, it uses fewer variables for estimating 3D coordinates, making accuracy difficult to guarantee. Conversely, increasing the number of cameras increases cost and computational complexity.

[0014] Based on this, embodiments of this application provide a method for determining the landing point location. Figure 1This is a flowchart illustrating a method for determining the landing point location provided in an embodiment of this application; as shown below. Figure 1 As shown, the method includes: S101. Perform image processing on the multi-view images to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop contained in each view image and the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image.

[0015] Multi-view images are images taken from multiple perspectives of the same area.

[0016] It should be noted that the moving sphere can be any sphere capable of movement, and the specific type can be determined based on the actual situation. For example, the moving sphere could be a ping-pong ball. The sphere's surface can be understood as the tabletop that matches the moving sphere; the specific type can be determined based on the actual situation. For example, the sphere's surface could be understood as a ping-pong table or a ping-pong table. The same area can be understood as the area containing both the ping-pong ball and the ping-pong table, or it can be understood as the ping-pong playing area.

[0017] It should be noted that multi-view images are images obtained by capturing the same area from multiple perspectives. This can also be understood as multiple images of the same area (the table tennis match area) captured by cameras from multiple angles. The number of angles (or cameras) can be determined based on the actual situation and is not limited here. Before processing the multi-view images, the method also includes acquiring them. This acquisition can be done using multiple cameras. The cameras can be ordinary surveillance cameras, placed on either side of the table tennis table. As an example, three ordinary surveillance cameras are used, placed on either side of the table tennis table to ensure that the cameras' field of view fully covers the entire table surface and movement range. Two cameras are placed on one side at the same horizontal level, approximately 1.3 meters apart. The third camera is placed on the other side of the table, using an appropriate overhead view to collect data, ensuring that all three cameras capture the entire table tennis surface and minimizing the capture of irrelevant scenes.

[0018] It should be noted that the first two-dimensional coordinate information corresponding to the spherical tabletop included in each viewpoint image can be understood as the first two-dimensional coordinate information corresponding to the first key point of the spherical tabletop included in each viewpoint image. The key points of the spherical tabletop can include the four corner points of the table and the four corner points of the net. Specifically, the first key points can be any six or more non-coplanar corner points selected from the four corner points of the table and the four corner points of the net as the key points for determining the first two-dimensional coordinate information. The first two-dimensional coordinate information can be understood as the two-dimensional pixel coordinate information corresponding to the first key point of the spherical tabletop (or ping-pong table). The second two-dimensional coordinate information corresponding to the moving sphere included in each viewpoint image can be understood as the two-dimensional pixel coordinate information corresponding to the ping-pong ball included in each viewpoint image.

[0019] In this embodiment of the application, the process of performing image processing on multi-view images to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop in each view image and the second two-dimensional coordinate information corresponding to the moving sphere in each view image specifically includes: inputting the multi-view images into a first model to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop in each view image; performing image recognition and filtering processing on the multi-view images to obtain the second two-dimensional coordinate information corresponding to the moving sphere in each view image.

[0020] It should be noted that the first model can be understood as any trained deep learning model capable of detecting key points on a spherical tabletop; no limitation is made here. As an example, the first model can be a trained ResNet model. Inputting multi-view images into the first model yields the first two-dimensional coordinate information of the spherical tabletop in each viewpoint image. This can be understood as inputting multi-view images into the ResNet model for keypoint detection, obtaining the two-dimensional pixel coordinate information corresponding to the first key point of the spherical tabletop in each viewpoint image. When there are multiple first key points, it is necessary to determine the two-dimensional pixel coordinate information corresponding to each first key point.

[0021] It should be noted that the above method also includes a manual correction function, allowing operators to adjust the model output to ensure the accuracy of the two-dimensional pixel coordinates corresponding to the final key points.

[0022] In this embodiment of the application, the process of performing image recognition and filtering on multi-view images to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image specifically includes: acquiring a background image containing the sphere's tabletop; performing image recognition on the multi-view images based on the background image to obtain the first pixel corresponding to the moving sphere contained in each view image; and performing filtering on each view image based on the first pixel to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image.

[0023] It should be noted that the background image can be understood as a static background image. Obtaining a background image containing the ping-pong ball on the table can be understood as taking a static background image from a fixed table and camera position (the same position as when capturing multi-view images). The first pixel can be understood as any newly added or dynamic pixels in the multi-view images besides the background image, including at least the ping-pong ball, player, racket, and noise (such as changes in lighting and slight shaking). Image recognition of the multi-view images based on the background image to obtain the first pixel corresponding to the moving ball in each view image can be understood as identifying the first pixel (including the aforementioned newly added and dynamic pixels) by subtracting the multi-view images from the background image. Filtering each view image based on the first pixel to obtain the second two-dimensional coordinate information corresponding to the moving ball in each view image can be understood as applying filtering algorithms (such as median filtering and mean filtering), threshold adjustment, and color constraints to eliminate noise data in the multi-view images, thereby obtaining clearer ping-pong ball pixel information from the first pixel, and determining the two-dimensional pixel coordinate information corresponding to the ping-pong ball based on the pixel position.

[0024] The embodiments of this application employ a neural network model to detect key points on a pool table, which enables efficient identification and localization of key points, improves the efficiency of obtaining the two-dimensional pixel coordinates corresponding to the key points on the pool table, and allows operators to adjust the structure of the model output to ensure the accuracy of the obtained two-dimensional pixel coordinates corresponding to the key points on the pool table.

[0025] S102. Obtain the first three-dimensional coordinate information corresponding to the spherical tabletop.

[0026] It should be noted that the first three-dimensional coordinate information corresponding to the spherical tabletop can be understood as the three-dimensional coordinate information corresponding to the first key point of the spherical tabletop. There is a correspondence between obtaining the first three-dimensional coordinate information corresponding to the first key point of the spherical tabletop and the first two-dimensional coordinate information corresponding to the first key point of the spherical tabletop obtained above; that is, when obtaining the two-dimensional coordinates of a key point on the spherical tabletop, the three-dimensional coordinates of that key point are obtained. Since the length, width, and height of a standard ping-pong table are fixed, obtaining the first three-dimensional coordinate information corresponding to the spherical tabletop can be understood as establishing a three-dimensional coordinate system with the X-axis along the length of the tabletop, the Y-axis along the width of the tabletop, and the Z-axis perpendicular to the tabletop upwards. By arbitrarily selecting a key point from the key points contained in the spherical tabletop as the origin, the three-dimensional coordinate information corresponding to all key points contained in the spherical tabletop can be obtained. As an example, the origin can be located at the lower left corner (front left corner point) of the table.

[0027] S103. Determine the first projection matrix of the spherical tabletop at each viewpoint based on the first two-dimensional coordinate information and the first three-dimensional coordinate information.

[0028] It should be noted that the first projection matrix of the spherical tabletop at each viewpoint can be understood as the projection matrix between the two-dimensional pixel coordinates and the three-dimensional coordinates of the key points of the spherical tabletop captured by the camera during shooting. In practical applications, the first projection matrix can also be understood as the camera matrix corresponding to each camera.

[0029] In this embodiment of the application, the process of determining the first projection matrix of the spherical tabletop at each viewpoint based on the first two-dimensional coordinate information and the first three-dimensional coordinate information specifically includes: constructing a first correlation matrix based on the first two-dimensional coordinate information and the first three-dimensional coordinate information; performing singular value decomposition on the first correlation matrix to obtain the second projection matrix corresponding to the spherical tabletop; and performing correction processing on the second projection matrix to obtain the first projection matrix of the spherical tabletop at each viewpoint.

[0030] It should be noted that the first correlation matrix can be understood as an correlation matrix established based on the correspondence between the two-dimensional pixel coordinates and the three-dimensional coordinates of the spherical tabletop. Constructing the first correlation matrix based on the first two-dimensional coordinate information and the first three-dimensional coordinate information can be understood as follows: normalizing the first two-dimensional coordinate information corresponding to the spherical tabletop from each viewpoint to obtain normalized first two-dimensional coordinate information; constructing a sub-matrix for each viewpoint (or each camera (or video camera)) based on the normalized first two-dimensional coordinate information and the corresponding first three-dimensional coordinate information; and stacking the sub-matrices of each viewpoint into a first correlation matrix. In practical applications, the first correlation matrix can be denoted as M.

[0031] It should be noted that the second projection matrix can be understood as the initial projection matrix or the initial camera matrix corresponding to the camera. Performing singular value decomposition (SVD) on the first incidence matrix to obtain the second projection matrix corresponding to the spherical tabletop can be understood as follows: based on the first incidence matrix, a linear system is constructed; Singular Value Decomposition (SVD) is used to decompose the first incidence matrix into three matrices; the vectors corresponding to the smallest singular values ​​are extracted; and the vectors calculated using SVD are used to construct the second projection matrix. For ease of understanding, an example is provided here: for the constructed first incidence matrix M, the linear system Mx=0 is solved; SVD is used to decompose matrix M into three matrices u, s, and v; the vectors corresponding to the smallest singular values ​​are extracted; and the vectors calculated using SVD are used to construct the second projection matrix. In practical applications, the second projection matrix can be denoted as P.

[0032] In this embodiment, the process of correcting the second projection matrix to obtain the first projection matrix corresponding to the spherical tabletop at each viewpoint specifically includes: performing orientation correction processing on the second projection matrix based on the first three-dimensional coordinate information to obtain the third projection matrix corresponding to the spherical tabletop at each viewpoint; constructing the first normalized matrix corresponding to the spherical tabletop based on the first two-dimensional coordinate information of the spherical tabletop contained in the image of each viewpoint; and determining the first projection matrix corresponding to the spherical tabletop at each viewpoint based on the third projection matrix and the first normalized matrix corresponding to the spherical tabletop at each viewpoint.

[0033] It should be noted that the third projection matrix can be understood as the projection matrix obtained by directional correction of the second projection matrix. If the second projection matrix does not require directional correction, it is equivalent to the third projection matrix. Based on the first 3D coordinate information, the second projection matrix is ​​directionally corrected to obtain the third projection matrix corresponding to the spherical tabletop at each viewpoint. This can be understood as follows: for each viewpoint (or each camera), the corresponding second projection matrix is ​​used to project the 3D coordinate information onto a 2D plane, and the sign of the z-coordinate is detected. If the z-coordinate sign is incorrect (e.g., not on the table or off-table), the sign of the second projection matrix is ​​reversed (e.g., the second projection matrix is ​​reversed) to obtain the third projection matrix.

[0034] It should be noted that constructing the first normalized matrix corresponding to the spherical tabletop based on the first two-dimensional coordinate information of the spherical tabletop contained in each viewpoint image can be understood as calculating the mean and standard deviation of the first two-dimensional coordinate information of the spherical tabletop contained in each viewpoint image, requiring the mean to be 0 and the standard deviation to be 1, and constructing the first normalized matrix based on the mean and standard deviation. In practical applications, the first normalized matrix can be denoted as N. The first projection matrix can be understood as the projection matrix (or camera matrix) obtained by restoring the third projection matrix to world coordinates. Determining the first projection matrix corresponding to the spherical tabletop in each viewpoint based on the third projection matrix and the first normalized matrix can be understood as multiplying the inverse of the first normalized matrix corresponding to the spherical tabletop in each viewpoint by the third projection matrix to obtain the first projection matrix corresponding to the spherical tabletop in each viewpoint.

[0035] The solution in this application uses the correspondence between key points on a ping-pong table in two-dimensional and three-dimensional space to establish a projection relationship (or association relationship) between the world coordinate system and the pixel coordinate system, so as to calculate a high-precision three-dimensional trajectory of the ping-pong ball in combination with a multi-view camera array; it has the characteristics of high computational efficiency and strong robustness.

[0036] S104. Determine the second three-dimensional coordinate information corresponding to the moving sphere based on the first projection matrix and the second two-dimensional coordinate information; and determine the three-dimensional motion trajectory and landing position of the moving sphere based on the second three-dimensional coordinate information.

[0037] It should be noted that the second three-dimensional coordinate information corresponding to the moving sphere can be understood as the three-dimensional coordinates of the ping-pong ball.

[0038] In this embodiment of the application, the process of determining the second three-dimensional coordinate information corresponding to the moving sphere based on the first projection matrix and the second two-dimensional coordinate information specifically includes: constructing the projection equation of the moving sphere at each viewpoint based on the first projection matrix and the second two-dimensional coordinate information; stacking the projection equations at each viewpoint to obtain the second correlation matrix corresponding to the moving sphere; and performing singular value decomposition on the second correlation matrix to obtain the second three-dimensional coordinate information corresponding to the moving sphere.

[0039] It should be noted that the projection equation of the moving sphere at each viewpoint can be understood as the projection equation of the intersection of the viewpoints at each viewpoint. Constructing the projection equation of the moving sphere at each viewpoint based on the first projection matrix and the second two-dimensional coordinate information can be understood as normalizing the second two-dimensional coordinate information to obtain normalized second two-dimensional coordinate information; and constructing the projection equation (or linear system) of the intersection of the three camera viewpoints based on the normalized second two-dimensional coordinate information and the first projection matrix. Specifically, normalizing the second two-dimensional coordinate information can be understood as using the decomposition result of the first correlation matrix to construct the intrinsic parameter matrix for each viewpoint (or each camera), and normalizing the corresponding second two-dimensional coordinate information for each viewpoint based on the intrinsic parameter matrix of each viewpoint.

[0040] It should be noted that the second incidence matrix can be understood as a matrix formed by stacking the projection equations of all viewpoints. Stacking the projection equations corresponding to each viewpoint yields the second incidence matrix for the moving sphere. In practical applications, the second incidence matrix can be denoted as Q. Performing singular value decomposition (SVD) on the second incidence matrix yields the second three-dimensional coordinate information of the moving sphere. This can be understood as performing SVD on the second incidence matrix to identify the minimum singular value vector, extracting the column vector corresponding to the minimum singular value as the homogeneous 3D coordinates of the ping-pong ball, and converting the homogeneous 3D coordinates to Euclidean coordinates, i.e., the second three-dimensional coordinate information.

[0041] In this embodiment of the application, the process of determining the three-dimensional motion trajectory and landing position of the moving ball based on the second three-dimensional coordinate information specifically includes: determining the three-dimensional motion trajectory of the moving ball based on the second three-dimensional coordinate information; and determining the landing position of the moving ball based on the three-dimensional motion trajectory and the first coordinate value.

[0042] It should be noted that the three-dimensional trajectory of the moving ball can be understood as the three-dimensional trajectory of a ping-pong ball. Determining the three-dimensional trajectory of the moving ball based on the second three-dimensional coordinate information can be understood as acquiring multiple second three-dimensional coordinate pieces in chronological order, constructing the three-dimensional trajectory of the ping-pong ball based on these multiple pieces of information, and thus obtaining the direction of motion of the ping-pong ball's three-dimensional trajectory. The first coordinate value can be understood as the Z-coordinate value in the three-dimensional coordinate system; determining the landing position of the moving ball based on the three-dimensional trajectory and the first coordinate value can be understood as determining the direction of motion of the ping-pong ball's three-dimensional trajectory and setting the Z-coordinate value to 0 to obtain the landing position of the ping-pong ball.

[0043] The solution in this application determines the second three-dimensional coordinate information of the moving sphere based on the projection relationship between the first two-dimensional coordinate information and the first three-dimensional coordinate information corresponding to the sphere's tabletop, thereby obtaining the three-dimensional motion trajectory and landing position. This eliminates the need for manual calibration of the camera's intrinsic and extrinsic parameters. Furthermore, by jointly determining the second three-dimensional coordinate information based on the first projection matrix corresponding to each viewpoint and the second two-dimensional coordinate information corresponding to the moving sphere, the solution considers the position of the moving sphere at multiple viewpoints, which improves the accuracy of the sphere's three-dimensional coordinates and thus enhances the accuracy of the sphere's three-dimensional trajectory and landing position.

[0044] To facilitate understanding, an example of the above scheme is provided here. In practical applications, the above method for determining the landing point can be exemplified as a self-calibrated 3D trajectory and landing point detection method for a ping-pong ball. By automatically detecting key point models combined with a self-calibration method, high efficiency and convenience are achieved. Simultaneously, multi-view cameras are used to calculate a more accurate 3D spatial position. Using three ordinary cameras (non-high-speed cameras) combined with the physical structure and size of the ping-pong table, the mapping relationship between 3D coordinates in world space and 2D coordinates in pixel space is learned. Based on this mapping relationship, the 3D coordinates of the ping-pong ball are calculated, and the direction of motion of the 3D trajectory is obtained for calculating the ball's landing point. Specifically, this includes the following: (1) Use three ordinary cameras to obtain image information of the table tennis table and the table tennis ball from different perspectives. Call the ResNet model to detect the key points of the table. Use the detected two-dimensional pixel coordinates of the table and the three-dimensional coordinates of the selected key points of the table to obtain the camera projection matrix between the 2D and 3D of the key points of the table. Calculate the landing position of the table tennis ball based on the camera projection matrix and the key points of the table tennis ball.

[0045] (2) Method for obtaining the relationship between the world coordinate system and the pixel coordinate system: normalize the 2D coordinates of the key points of the table in the pixel space; establish the relationship matrix M based on the correspondence between points; perform singular value decomposition on the relationship matrix M, establish the camera matrix, and perform projection direction correction.

[0046] The solution proposed in this application embodiment uses three ordinary cameras (non-high-speed cameras) combined with the physical structure and size of a ping-pong table to learn the mapping relationship between three-dimensional coordinates in world space and two-dimensional coordinates in pixel space. Based on this mapping relationship, the three-dimensional coordinates of the ping-pong ball are calculated, and the motion direction of the three-dimensional trajectory is obtained for calculating the landing position of the ball.

[0047] Unlike traditional methods that require high-speed cameras to collect data and train a ping-pong ball detection model, this embodiment uses a background subtraction algorithm combined with filtering and color constraint operations to segment the high-speed moving ping-pong ball and obtain its coordinates from three cameras. During the learning of the mapping relationship between two-dimensional and three-dimensional space, key points of the ping-pong table are designated as corresponding points, including the four corners of the table and the four corners of the net. Since ping-pong tables have standard manufacturing specifications, the length and width of the table and the height and width of the net are fixed in the real world; therefore, by specifying the origin of the world coordinate system, the coordinates of the key points can be obtained. The acquisition of pixel-space key point coordinates uses a semi-automatic method to ensure the accuracy of the annotation. This embodiment trains a table key point detection model to achieve automatic annotation while retaining the ability to manually correct the annotation results, ensuring the robustness of the learned mapping relationship. Finally, the three-dimensional spatial coordinates of the ping-pong ball are calculated based on the mapping relationship, the ball's trajectory is drawn, and the landing position is determined based on the direction of the trajectory. This operation is faster and less costly than the traditional method using high-speed cameras.

[0048] For ease of understanding, Figure 2 A flowchart illustrating an exemplary self-calibrated 3D trajectory and landing point detection system for a ping-pong ball is provided as an embodiment of this application; Figure 2 As shown, the specific steps are as follows: 1. Data reading module.

[0049] 2. Human body recognition.

[0050] 3. Racket recognition.

[0051] 4. Key points of 2D pool table.

[0052] 5. Key points of the 3D pool table.

[0053] 6. 2D coordinates of a ping-pong ball.

[0054] It should be noted that the 2D coordinates of the ping-pong ball are obtained by combining human body recognition and racket recognition with filtering, background subtraction, and color constraints.

[0055] 7. Projection matrix.

[0056] It should be noted that the projection matrix is ​​obtained from the 2D key points and 3D key points of the pool table.

[0057] 8. 3D coordinates of a table tennis ball.

[0058] It should be noted that the 3D coordinates of the ping-pong ball are obtained from the 2D coordinates and the projection matrix.

[0059] 9. Calculate the landing point.

[0060] It should be noted that the landing point is calculated based on the 3D coordinates of the ping-pong ball.

[0061] 10. Draw a 3D trajectory.

[0062] It should be noted that the 3D trajectory is drawn based on the 3D coordinates of the table tennis ball.

[0063] The detailed steps of the embodiments of this application are as follows.

[0064] 1. Camera setup.

[0065] Three standard surveillance cameras were placed on either side of the ping-pong table, ensuring that the cameras' fields of view fully covered the entire table surface and movement area. Two cameras were placed on one side, at the same horizontal level, approximately 1.3 meters apart. The third camera was placed on the other side of the table, using a suitable overhead view to collect data. The goal was to ensure that all three cameras captured the entire ping-pong table surface, minimizing the capture of irrelevant scenes.

[0066] 2. Setting key points on the pool table.

[0067] Selection of standard key points: Key points are set on the surface of the ping-pong table, including: (1) The four corners of the pool table; (2) The four corners of the net.

[0068] It should be noted that the number of key points can be dynamically adjusted, but in actual use, it should be ensured that there are no fewer than 4 key points. Figure 3 A schematic diagram illustrating the setting of key points on an exemplary ping-pong table, provided as an embodiment of this application; such as Figure 3 As shown, you can choose the four corners of the table or the four corners of the net as key points.

[0069] Establishment of the coordinate system: According to the International Table Tennis Federation (ITTF) standard, the length of a table tennis table is 2.74 meters, the width is 1.525 meters, and the height is 0.76 meters. Establish a three-dimensional coordinate system with its origin located at the lower left corner (front left corner point) of the table. The X-axis is along the length of the table, the Y-axis is along the width of the table, and the Z-axis is perpendicular to the table and pointing upwards.

[0070] 3. Pool table key point detection model.

[0071] Dataset creation: Collect table tennis match data from multiple perspectives simultaneously, mark the coordinates of key points on the table according to the set criteria, and set the saving order of the coordinates to start from the top left corner of the table to ensure that the key points of the table are fully visible in each perspective.

[0072] ResNet Model Training: A deep learning model based on the ResNet architecture was developed and trained. The basic components of this architecture include an input layer, multiple residual blocks, pooling layers, and an output layer. Specifically, the residual blocks consist of a series of convolutional layers, each followed by normalization and a LeakyReLU activation function. Skip connections are used to directly add the input signal to the convolutionally transformed signal to form a residual mapping. Finally, the model was trained using the ping-pong table keypoint dataset collected in the previous step. This model aims to efficiently identify and locate keypoints on the table, improving the efficiency of automatic calibration. Figure 4 This application provides an exemplary deep learning model architecture diagram for predicting key points on a ping-pong table based on a ResNet structure, as shown in the following embodiment. Figure 4 As shown, the 640×360 image is sequentially processed through convolution + Leaky ReLU + Batch Normalization, max pooling, upsampling, and softmax normalization, and finally the key points of the table are detected.

[0073] Calibration process: The camera captures images of the ping-pong table, and the ResNet model is used for keypoint detection. The model outputs the pixel coordinates of each keypoint. This will be used for subsequent two-dimensional and three-dimensional coordinate mapping learning. A manual correction function is also set up to allow operators to adjust the model output results to ensure the accuracy of the final calibration results.

[0074] 4. Table tennis ball test.

[0075] Step 1: Background image acquisition.

[0076] A static background image was taken at a fixed position on the pool table and at the camera location to serve as a background template.

[0077] Step 2: Extract new pixels.

[0078] Based on step 1, by performing difference analysis between the real-time captured image and the background image, newly added pixels and dynamic pixels are identified, including the ping-pong ball, player, racket, and noise (such as changes in light and shadow and slight shaking).

[0079] Step 3: Noise data filtering.

[0080] By applying filtering algorithms (such as median filtering and mean filtering), threshold adjustment, and color constraints, noise data in the image is eliminated to obtain clearer ping-pong ball pixel information, and two-dimensional image coordinates are calculated based on pixel positions.

[0081] Step 4: Trajectory tracking.

[0082] Based on the clear two-dimensional pixel coordinates of the ping-pong ball obtained in step 3, the ball's trajectory has a certain regularity. By combining the Kalman filter algorithm, the direction of the ping-pong ball's movement is predicted, thereby enhancing the accuracy of the detection results.

[0083] 5. Two-dimensional and three-dimensional mapping relationship.

[0084] This paper utilizes a multi-view geometric calculation method to calculate the mapping relationship between 2D and 3D images from multiple perspectives. Given that the dimensions of a standard ping-pong table in the real world are fixed, the 3D coordinates of selected key points on the table can be represented by normalized data. Combined with the 2D pixel coordinates of the table detected by the model, the projection matrix between the 2D and 3D key points is obtained. Based on this projection relationship, the 3D coordinates of the ping-pong ball can be estimated from the detected 2D pixel coordinates. The camera projection matrix representing the correspondence between 3D and 2D points is denoted by P. The calculation process employs normalization, linear system construction, singular value decomposition (SVD), and projection direction correction to ensure the reliability and accuracy of the obtained camera matrix. The camera matrix is ​​crucial for various computer vision applications, enabling accurate 3D-to-2D mapping and its inverse process. The specific steps are as follows: Step 1: Normalization of 2D points on the pool table.

[0085] To improve the numerical stability of subsequent calculations, the 2D coordinates of the key points on the table are normalized in pixel space to unify the scaling of data from different perspectives. The normalization process involves two main steps: (1) Calculation of mean and standard deviation: For each camera k, calculate the mean and standard deviation of the normalized 2D coordinate values ​​of the table key points. The mean is required to be 0 and the standard deviation is 1.

[0086] (2) Construction of normalization matrix: Based on the mean and standard deviation, construct a normalization matrix N to store the standardized 2D coordinates.

[0087] Step 2: Construct the association matrix.

[0088] An association matrix is ​​established based on the point-to-point correspondence, integrating information from multiple cameras into a unified association matrix to describe the correspondence between 2D and 3D points. For each camera k, a submatrix M_k is initialized, containing the coordinates of 3D points and the normalized 2D points from step 1, used to encode the projection constraints of that camera. Finally, the submatrices of each camera are stacked into an association matrix M.

[0089] Step 3, Singular Value Decomposition (SVD).

[0090] Based on the matrix M constructed in step 2, the linear system Mx=0 is solved. SVD is used to decompose matrix M into three matrices u, s, and v. The vector corresponding to the minimum singular value is extracted, and this vector contains elements of the camera matrix P. Multi-view data improves the accuracy of matrix P by averaging the error of a single viewpoint.

[0091] Step 4: Establish the camera matrix.

[0092] Using the vectors obtained from SVD in step 3, construct a camera matrix P to represent the initial estimate of the camera projection matrix, mapping 3D points to 2D points.

[0093] Step 5: Projection direction correction.

[0094] Based on step 4, verify the correctness of the camera matrix orientation by calculating the consistency of the projection orientations of multiple cameras and performing the following sign check: (1) Projection check: For each camera k, project the 3D point onto the 2D plane using its projection matrix and check the sign of its z coordinate.

[0095] (2) Sign correction: If the sign of the z coordinate is incorrect, then reverse the sign of the camera matrix.

[0096] Step 6: Restore world coordinates.

[0097] For each camera k, the camera matrix corrected in step 5 is multiplied by the inverse of the normalized matrix N to obtain the final camera matrix in the original coordinate system. This step ensures that the camera matrix accurately maps the original 3D coordinates to 2D coordinates.

[0098] Step 7: Normalize the table tennis ball coordinates.

[0099] For each camera k, the intrinsic parameter matrix is ​​constructed using the matrix decomposition result from step 3 to normalize the 2D points and align the points with the camera's intrinsic parameter coordinate system.

[0100] Step 8: Construct the correlation matrix related to spherical coordinates.

[0101] Multi-view computation describes the projection constraints of the ping-pong ball by constructing a joint matrix. The specific steps are as follows: (1) Projection equation: For each camera k, a linear system of the intersection points of the three camera viewpoints is constructed based on the normalized coordinates and the projection matrix constructed in step 6.

[0102] (2) Joint matrix: The projection equations of all cameras are stacked into an incidence matrix Q.

[0103] Step 9: Decompose the matrix constructed from spherical coordinates.

[0104] Multi-view computation utilizes SVD decomposition to extract the 3D coordinates of the ping-pong ball from the incidence matrix. Specifically, SVD decomposition Q identifies the minimum singular value vector, from which the column vector corresponding to the minimum singular value is extracted as the homogeneous 3D coordinates of the ping-pong ball.

[0105] Step 10: Convert to Euclidean coordinates.

[0106] From homogeneous coordinates to Euclidean coordinates, the homogeneous coordinates are reconstructed and normalized to provide a physically meaningful 3D spatial position of the ping-pong ball (i.e., the coordinates of the ping-pong ball in the final output).

[0107] 6. Calculation of the landing point of the ping-pong ball.

[0108] This application provides a method for calculating the landing point of a ping-pong ball strike. By systematically processing multiple sets of input motion trajectory data, this method can accurately identify the landing point of the ball.

[0109] Using the coordinates of the ping-pong ball output above, we can directly determine the lowest point, which is the landing point on the table.

[0110] First, continuous ping-pong ball motion trajectories are acquired, assuming a frame count greater than 10 to ensure sufficient data for analysis. Next, a local minimum detection method is applied to identify the minimum position in the Z-coordinate (perpendicular to the table surface) to determine the possible landing point. When processing the initial shot (usually a serve), this method identifies two minimum values, extracts the corresponding motion data points, and adjusts their Z-coordinate to 0 to clearly define the landing point. In subsequent shots, a single minimum value is identified to extract the corresponding landing point position, again with the Z-coordinate set to 0 to ensure accuracy.

[0111] This application also provides a device for determining the landing point location. Figure 5 This is a schematic diagram of the structure of a landing point determination device provided in an embodiment of this application; as shown below. Figure 5 As shown, the landing point determination device 500 includes: The image processing unit 501 is used to perform image processing on the multi-view images to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop contained in each view image and the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image; the multi-view images are multi-view images of the same area. Acquisition unit 502 is used to acquire the first three-dimensional coordinate information corresponding to the spherical tabletop; The first determining unit 503 is used to determine the first projection matrix of the spherical tabletop at each viewpoint based on the first two-dimensional coordinate information and the first three-dimensional coordinate information. The second determining unit 504 is used to determine the second three-dimensional coordinate information corresponding to the moving sphere based on the first projection matrix and the second two-dimensional coordinate information; and to determine the three-dimensional motion trajectory and landing position of the moving sphere based on the second three-dimensional coordinate information.

[0112] In one embodiment, the first determining unit 503 is further configured to construct a first correlation matrix based on the first two-dimensional coordinate information and the first three-dimensional coordinate information; perform singular value decomposition on the first correlation matrix to obtain a second projection matrix corresponding to the spherical tabletop; and perform correction processing on the second projection matrix to obtain the first projection matrix corresponding to the spherical tabletop at each viewpoint.

[0113] In one embodiment, the first determining unit 503 is further configured to perform orientation correction processing on the second projection matrix based on the first three-dimensional coordinate information to obtain a third projection matrix corresponding to the spherical tabletop in each viewpoint; construct a first normalized matrix corresponding to the spherical tabletop based on the first two-dimensional coordinate information of the spherical tabletop contained in the image of each viewpoint; and determine the first projection matrix corresponding to the spherical tabletop in each viewpoint based on the third projection matrix corresponding to the spherical tabletop in each viewpoint and the first normalized matrix.

[0114] In one embodiment, the second determining unit 504 is further configured to construct the projection equation of the moving sphere at each viewpoint based on the first projection matrix and the second two-dimensional coordinate information; stack the projection equations at each viewpoint to obtain the second correlation matrix corresponding to the moving sphere; and perform singular value decomposition on the second correlation matrix to obtain the second three-dimensional coordinate information corresponding to the moving sphere.

[0115] In one embodiment, the image processing unit 501 is further configured to input the multi-view images into a first model to obtain the first two-dimensional coordinate information corresponding to the spherical desktop contained in each view image; and to perform image recognition and filtering processing on the multi-view images to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image.

[0116] In one embodiment, the image processing unit 501 is further configured to acquire a background image containing the spherical tabletop; perform image recognition on the multi-view images based on the background image to obtain a first pixel corresponding to the moving sphere contained in each view image; and perform filtering processing on each view image based on the first pixel to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image.

[0117] In one embodiment, the second determining unit 504 is further configured to determine the three-dimensional motion trajectory corresponding to the moving sphere based on the second three-dimensional coordinate information; and to determine the landing position of the moving sphere according to the three-dimensional motion trajectory and the first coordinate value.

[0118] This application also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; as shown below. Figure 6 As shown, the electronic device 600 includes a processor 601 and a memory 603. Optionally, the electronic device 600 may also include a communication bus 602.

[0119] In specific embodiments, the processor 601 described above can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and this embodiment does not specifically limit it.

[0120] In this embodiment, the communication bus 602 is used to establish communication between the processor 601 and the memory 603; when the processor 601 executes the running program stored in the memory 603, it implements the following method for determining the landing point position: Image processing is performed on the multi-view images to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop in each view image and the second two-dimensional coordinate information corresponding to the moving sphere in each view image; the multi-view images are multi-view images of the same area; the first three-dimensional coordinate information corresponding to the spherical tabletop is obtained; the first projection matrix of the spherical tabletop in each view is determined based on the first two-dimensional coordinate information and the first three-dimensional coordinate information; the second three-dimensional coordinate information corresponding to the moving sphere is determined according to the first projection matrix and the second two-dimensional coordinate information; and the three-dimensional motion trajectory and landing position of the moving sphere are determined based on the second three-dimensional coordinate information.

[0121] Furthermore, the processor 601 is also configured to construct a first correlation matrix based on the first two-dimensional coordinate information and the first three-dimensional coordinate information; perform singular value decomposition on the first correlation matrix to obtain a second projection matrix corresponding to the spherical tabletop; and perform correction processing on the second projection matrix to obtain the first projection matrix corresponding to the spherical tabletop at each viewpoint.

[0122] Furthermore, the processor 601 is also configured to perform orientation correction processing on the second projection matrix based on the first three-dimensional coordinate information to obtain a third projection matrix corresponding to the spherical tabletop in each viewpoint; construct a first normalized matrix corresponding to the spherical tabletop based on the first two-dimensional coordinate information of the spherical tabletop contained in the image of each viewpoint; and determine the first projection matrix corresponding to the spherical tabletop in each viewpoint based on the third projection matrix corresponding to the spherical tabletop in each viewpoint and the first normalized matrix.

[0123] Furthermore, the processor 601 is also configured to construct the projection equation of the moving sphere at each viewpoint based on the first projection matrix and the second two-dimensional coordinate information; stack the projection equations at each viewpoint to obtain the second correlation matrix corresponding to the moving sphere; and perform singular value decomposition on the second correlation matrix to obtain the second three-dimensional coordinate information corresponding to the moving sphere.

[0124] Furthermore, the processor 601 is also configured to input the multi-view images into the first model to obtain the first two-dimensional coordinate information corresponding to the spherical desktop contained in each view image; and to perform image recognition and filtering processing on the multi-view images to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image.

[0125] Furthermore, the processor 601 is also configured to acquire a background image containing the spherical tabletop; perform image recognition on the multi-view images based on the background image to obtain a first pixel corresponding to the moving sphere contained in each view image; and perform filtering processing on each view image based on the first pixel to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image.

[0126] Furthermore, the processor 601 is also configured to determine the three-dimensional motion trajectory corresponding to the moving sphere based on the second three-dimensional coordinate information; and to determine the landing position of the moving sphere according to the three-dimensional motion trajectory and the first coordinate value.

[0127] This application provides a storage medium storing a computer program thereon. The computer-readable storage medium stores one or more programs, which can be executed by one or more processors. The computer program implements the landing point location determination method described above.

[0128] Based on the above embodiments, this application provides a computer program product, including a computer program that can be executed by one or more processors, and the computer program implements the landing point determination method as described above.

[0129] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause an image display device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0131] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for determining the landing point location, characterized in that, The method includes: Image processing is performed on the multi-view images to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop contained in each view image and the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image; the multi-view images are multi-view images of the same area. Obtain the first three-dimensional coordinate information corresponding to the spherical tabletop; Based on the first two-dimensional coordinate information and the first three-dimensional coordinate information, determine the first projection matrix of the spherical tabletop at each viewpoint; The second three-dimensional coordinate information corresponding to the moving sphere is determined based on the first projection matrix and the second two-dimensional coordinate information; and the three-dimensional motion trajectory and landing position of the moving sphere are determined based on the second three-dimensional coordinate information.

2. The method according to claim 1, characterized in that, The step of determining the first projection matrix of the spherical tabletop at each viewpoint based on the first two-dimensional coordinate information and the first three-dimensional coordinate information includes: A first correlation matrix is ​​constructed based on the first two-dimensional coordinate information and the first three-dimensional coordinate information; Singular value decomposition is performed on the first correlation matrix to obtain the second projection matrix corresponding to the spherical tabletop. The second projection matrix is ​​corrected to obtain the first projection matrix of the spherical tabletop at each viewpoint.

3. The method according to claim 2, characterized in that, The step of correcting the second projection matrix to obtain the first projection matrix of the spherical tabletop at each viewing angle includes: Based on the first three-dimensional coordinate information, the second projection matrix is ​​subjected to orientation correction processing to obtain the third projection matrix of the spherical tabletop at each viewpoint. Construct the first normalized matrix corresponding to the spherical tabletop based on the first two-dimensional coordinate information of the spherical tabletop contained in each viewpoint image; The first projection matrix of the spherical tabletop at each viewpoint is determined based on the third projection matrix of the spherical tabletop at each viewpoint and the first normalized matrix.

4. The method according to claim 1, characterized in that, Determining the second three-dimensional coordinate information corresponding to the moving sphere based on the first projection matrix and the second two-dimensional coordinate information includes: Based on the first projection matrix and the second two-dimensional coordinate information, construct the projection equation of the moving sphere at each viewpoint; The projection equations corresponding to each viewpoint are stacked to obtain the second correlation matrix corresponding to the moving sphere. The second correlation matrix is ​​subjected to singular value decomposition to obtain the second three-dimensional coordinate information corresponding to the moving sphere.

5. The method according to claim 1, characterized in that, The image processing of the multi-view images to obtain the first two-dimensional coordinate information corresponding to the spherical tabletop in each view image and the second two-dimensional coordinate information corresponding to the moving sphere in each view image includes: The multi-view images are input into the first model to obtain the first two-dimensional coordinate information corresponding to the spherical desktop contained in each view image; The multi-view images are subjected to image recognition and filtering to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image.

6. The method according to claim 5, characterized in that, The step of performing image recognition and filtering on the multi-view images to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each view image includes: Obtain a background image containing the spherical tabletop; Image recognition is performed on the multi-view images based on the background image to obtain the first pixel corresponding to the moving sphere contained in each view image; Based on the first pixel, each viewpoint image is filtered to obtain the second two-dimensional coordinate information corresponding to the moving sphere contained in each viewpoint image.

7. The method according to claim 1, characterized in that, Determining the three-dimensional motion trajectory and landing point of the moving sphere based on the second three-dimensional coordinate information includes: The three-dimensional motion trajectory of the moving sphere is determined based on the second three-dimensional coordinate information. The landing point of the moving sphere is determined based on the three-dimensional motion trajectory and the first coordinate value.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images