Flight trajectory prediction method and system

Abstract

The application provides a flight trajectory prediction method and system, and relates to the field of image motion analysis. The method comprises: collecting planar coordinates of a target to be predicted from at least two non-collinear perspectives; mapping the planar coordinates of the target to a world coordinate system to convert into spatial coordinates of the target; and predicting a second spatiotemporal data group of the target according to a plurality of first spatiotemporal data groups by a convolutional neural network model, the second spatiotemporal data group comprising spatial coordinates of the target at two adjacent moments before and after the target passes through a first plane and a second moment corresponding to the second spatiotemporal data group; wherein each first spatiotemporal data group comprises spatial coordinates of the target at two adjacent moments and a first moment corresponding to the first spatiotemporal data group. Based on the scheme, the influence of positioning errors on subsequent prediction is avoided, and the accuracy of flight trajectory prediction is improved.

Description

Technical Field

[0001] This application relates to the field of image motion analysis, and more specifically, to a method and system for predicting flight trajectories. Background Technology

[0002] Real-time prediction and analysis of ball trajectories have high practical value, especially in fields such as intelligent sports training, match strategy analysis, and robot tug-of-war systems. Accurately capturing and predicting the ball's flight trajectory requires a high degree of integration of computer vision, machine learning, and real-time control. Table tennis, as a sport involving a small ball with high flight speed, suffers from limitations imposed by traditional binocular vision systems, which rely on parallax to calculate depth. These limitations result in poor localization accuracy for small, high-speed targets and relatively slow processing speeds. Furthermore, in table tennis trajectory prediction scenarios, issues such as occlusion and limited field of view make it difficult for binocular localization to accurately pinpoint the ball's location, and algorithmic limitations further restrict processing speed.

[0003] Therefore, improving the accuracy of flight trajectory prediction is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a flight trajectory prediction method and system. It uses a convolutional neural network to predict the spatial coordinates of a target before and after it passes through a first plane based on spatial coordinate data collected from at least two non-collinear viewpoints. This eliminates the need to fit the overall trajectory of the target, avoids the impact of positioning errors on subsequent predictions, and improves the accuracy of flight trajectory prediction.

[0005] In a first aspect, a flight trajectory prediction method is provided. The method includes acquiring the planar coordinates of the target to be predicted from at least two non-collinear perspectives; mapping the planar coordinates of the target to a world coordinate system to convert them into the spatial coordinates of the target; and predicting a second set of spatiotemporal data of the target based on multiple sets of first spatiotemporal data using a convolutional neural network model. The second set of spatiotemporal data includes the spatial coordinates of the target at two adjacent moments before and after the target passes through the first plane and a second moment corresponding to the second set of spatiotemporal data. Each set of first spatiotemporal data includes the spatial coordinates of the target at two adjacent moments and a first moment corresponding to the first set of spatiotemporal data.

[0006] In this embodiment, the planar coordinates of the target to be predicted are acquired through at least two non-collinear perspectives to compensate for the lack of monocular depth through perspective complementarity. The planar coordinates are converted into spatial coordinates to establish a unified spatiotemporal coordinate benchmark. A convolutional neural network model is used to predict the spatial coordinates and time of the target at two adjacent moments before and after it crosses the first plane based on the spatial coordinates and time values. This enables the direct prediction of the time and coordinates before and after crossing the plane from the initial or middle flight data, avoiding complex trajectory fitting, eliminating the influence of positioning deviation on trajectory prediction, and improving the accuracy of flight trajectory prediction.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: predicting the trajectory of the target across the first plane based on the second spatiotemporal data set.

[0008] In this embodiment of the application, two spatial coordinates in the second spatiotemporal data group can represent the target's flight direction. By predicting the target's trajectory across the first plane using the second time point and the flight direction, the accuracy of flight trajectory prediction is improved.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the offset of the target's coordinates relative to the preset center point at two adjacent moments before and after it passes through the first plane is predicted by a convolutional neural network model based on multiple sets of first spatiotemporal data. The second spatiotemporal data set is then obtained based on the offset and the coordinates of the center point.

[0010] In this embodiment, instead of directly predicting absolute coordinates, the offset relative to the center point is predicted. The numerical range is standardized and the physical meaning is clear, which enables the convolutional neural network model to be trained stably, converge quickly, and improve the accuracy of flight trajectory prediction.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the convolutional neural network model is configured to: extract and aggregate features from multiple sets of first spatiotemporal data using a feature extractor; decode the aggregated features using a fully connected decoder to obtain an original prediction data set, the original prediction data set including the offset of the coordinates relative to a preset center point at two adjacent time points after normalization and the time value; and output a second spatiotemporal data set after decoding the original prediction data set.

[0012] In this embodiment, the convolutional neural network model includes a feature extractor and a fully connected decoder. The feature extractor extracts and aggregates features from multiple sets of first spatiotemporal data. The fully connected decoder then decodes the features, outputting the spatial coordinates and time values ​​of the target at two adjacent moments before and after it crosses the first plane. This directly predicts the moment of crossing the plane and the coordinates before and after, avoiding the complex trajectory fitting process in traditional methods, reducing the impact of positioning errors on prediction, and significantly improving the accuracy of flight trajectory prediction.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: normalizing the multiple sets of first spatiotemporal data input, wherein the spatial coordinates are normalized according to preset minimum spatial coordinate parameters and spatial range parameters; and the first moment is normalized according to the ratio of the first moment to the maximum possible time.

[0014] In this embodiment, the spatial coordinates and time values ​​are normalized respectively. The normalization of spatial coordinates is based on preset minimum spatial coordinate parameters and spatial range parameters; the normalization of time values ​​is based on the ratio of the first time point to the maximum possible time. The normalization process makes data of different dimensions and ranges comparable in model training, avoids training instability caused by differences in data range, and accelerates the convergence speed of the model.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, the planar coordinates of the target are mapped to the world coordinate system through a multilayer perceptron to convert them into the spatial coordinates of the target. The multilayer perceptron is trained using a first coordinate dataset, which includes the spatial coordinates and planar coordinates of multiple labeled points in space.

[0016] In this embodiment, the multilayer perceptron is trained using a first coordinate dataset, which includes the spatial coordinates and planar coordinates of multiple labeled points in space. This maintains the physical rationality of the data and improves the coordinate transformation performance of the multilayer perceptron.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, the multilayer perceptron is also trained using a second coordinate dataset, which is obtained by augmenting the first coordinate dataset. The data augmentation methods include augmentation based on physical constraints, augmentation based on geometric transformations, augmentation based on data mixing, or augmentation based on model characteristics.

[0018] In this embodiment, the multilayer perceptron is also trained using a second coordinate dataset, which is obtained from the first coordinate dataset through data augmentation. The augmentation methods include methods based on physical constraints, geometric transformations, data mixing, or model characteristics. The data augmentation methods significantly expand the diversity of training data while maintaining the physical rationality of the data, thereby improving the model's adaptability and generalization performance to complex scenarios.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, a first loss function is used during the training of the convolutional neural network model to compare the difference between the predicted value and the target value output by the convolutional neural network model. The first loss function includes offset loss, boundary condition loss, orientation constraint loss and range constraint loss to make the predicted value meet the preset physical constraints.

[0020] In this embodiment of the application, during the training of the convolutional neural network model, a loss function is used to compare the difference between the predicted value and the target value, and optimization is specifically performed on the prediction accuracy of the offset. Through a specially designed multi-task weighted loss function, including boundary condition loss, orientation constraint loss, and range constraint loss, the prediction of the offset can be optimized more accurately, ensuring high accuracy of the prediction results.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the multiple moments corresponding to the spatial coordinates in multiple sets of first spatiotemporal data are consecutive moments. Alternatively, the multiple first moments corresponding to multiple sets of first spatiotemporal data are discontinuous moments, and each set of first spatiotemporal data includes the corresponding first moment to characterize the temporal relationship of multiple sets of first spatiotemporal data.

[0022] In this embodiment of the application, by constructing a set of input time data and introducing time series information, the model can better capture the motion pattern of the target and maintain the accuracy and consistency of the prediction even in the case of non-continuous time data.

[0023] Secondly, a flight trajectory prediction system is provided. The system includes an image acquisition module configured to acquire the planar coordinates of a target from at least two non-collinear viewpoints; a coordinate transformation module configured to map the planar coordinates of the target to a world coordinate system to convert them into the spatial coordinates of the target; and a trajectory prediction module configured to use a convolutional neural network model to predict a second set of spatiotemporal data of the target based on multiple sets of first spatiotemporal data. The second set of spatiotemporal data includes the spatial coordinates of the target at two adjacent moments before and after the target passes through a first plane and a second moment corresponding to the second set of spatiotemporal data. Each set of first spatiotemporal data includes the spatial coordinates of the target at two adjacent moments and a first moment corresponding to the first set of spatiotemporal data.

[0024] Thirdly, a flight trajectory prediction system is provided, which is a module or unit for performing the methods described in the first aspect and its various implementations.

[0025] Fourthly, an apparatus is provided, comprising a processor and a memory, wherein the processor and the memory are connected together, wherein the memory is used to store program code, and the processor is used to invoke the program code to execute the methods described in the first aspect and its various implementations.

[0026] Fifthly, a computer-readable storage medium is provided storing a computer program that is executed by a processor to implement the method in any possible implementation of the method design of the first aspect.

[0027] In a sixth aspect, a computer program product is provided, including instructions that, when executed by a processor, cause a computer to perform any possible implementation of the method design of the first aspect described above.

[0028] Other beneficial effects can be found in the description of the first aspect, and will not be repeated here. Attached Figure Description

[0029] Figure 1This is a schematic diagram of the image acquisition module setup provided in an embodiment of this application; Figure 2 This is a schematic diagram of the transformation network for converting planar coordinates to spatial coordinates provided in an embodiment of this application; Figure 3 This is a top view schematic diagram of the spatial grid annotation points provided in an embodiment of this application; Figure 4 This is a schematic diagram of the training process of the convolutional neural network model provided in the embodiments of this application; Figure 5 This is a schematic diagram of the loss function calculation process provided in the embodiments of this application; Figure 6 This is a schematic diagram illustrating the mapping relationship between input and output data provided in an embodiment of this application; Figure 7 This is a schematic diagram of the convolutional neural network structure provided in the embodiments of this application; Figure 8 This is a schematic diagram of the output decoding process provided in an embodiment of this application; Figure 9 This is a schematic diagram summarizing the flight trajectory prediction method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the flight trajectory prediction system provided in the embodiments of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0031] The "range" disclosed in this application is defined by a lower limit and an upper limit. A given range is defined by selecting a lower limit and an upper limit, which define the boundaries of a specific range. Ranges defined in this way include endpoint values ​​and can be arbitrarily combined; that is, any lower limit can be combined with any upper limit to form a range. For example, if ranges of 60-120 and 80-110 are listed for a specific parameter, it is also expected that ranges of 60-110 and 80-120 are also included. Furthermore, if the minimum range values ​​are listed as 1 and 2, and the maximum range values ​​are listed as 3, 4, and 5, then the following ranges are all expected: 1-3, 1-4, 1-5, 2-3, 2-4, and 2-5. In this application, unless otherwise stated, the numerical range "ab" represents a shortened representation of any combination of real numbers between a and b, where a and b are real numbers. For example, the numerical range "0-5" means that all real numbers between "0-5" have been listed herein; "0-5" is simply a shortened representation of these numerical combinations. Furthermore, when a parameter is described as an integer greater than or equal to 2, it is equivalent to disclosing that the parameter is, for example, an integer such as 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, etc.

[0032] Unless otherwise specified, all embodiments and optional embodiments of this application can be combined with each other to form new technical solutions.

[0033] Unless otherwise specified, all technical features and optional technical features of this application may be combined to form new technical solutions.

[0034] The terminology used in the following embodiments is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to also include expressions such as “one or more,” unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, “at least one” and “one or more” refer to one, two, or more than two. The term “and / or” is used to describe the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can indicate: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character “ / ” generally indicates that the preceding and following related objects are in an “or” relationship.

[0035] References to "one embodiment," "some embodiments," "one example," or "some examples" used in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0036] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different descriptive objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0037] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. For clarity, the various parts in the drawings are not drawn to scale. Furthermore, some well-known parts may not be shown in the drawings.

[0038] Real-time prediction and analysis of a ball's trajectory has extremely high practical value, especially in fields such as intelligent sports training, competition strategy analysis, and robot combat systems. Accurately capturing and predicting the ball's flight trajectory requires a high degree of integration of computer vision, machine learning, and real-time control.

[0039] Taking table tennis as an example, the ball is only 40 mm in diameter and can travel at speeds of 30-50 m / s. Traditional binocular vision systems rely on parallax to calculate depth. Under the high-speed motion of the ball, the synchronization error of the binocular cameras and the delay in parallax calculation lead to significant positioning deviations. Furthermore, the triangulation principle of binocular systems requires a reasonable ratio between the baseline distance and the target distance, but the spatial constraints of the table setting limit the baseline distance, further compressing the physical upper limit of parallax resolution. Using dual cameras for positioning and tracking also suffers from systematic 3D bias; the trajectories of multiple positioning points exhibit oscillations perpendicular to the flight direction, making it impossible to use fitting methods to fit and predict the trajectory of the positioning points.

[0040] To address the aforementioned issues, this application provides a flight trajectory prediction method and system. It utilizes a convolutional neural network to predict the spatial coordinates of a target before and after it crosses a first plane based on spatial coordinate data collected from at least two non-collinear perspectives. This eliminates the need to fit the overall trajectory of the target, avoids the impact of positioning errors on subsequent predictions, and improves the accuracy of flight trajectory prediction.

[0041] This application uses table tennis as an example for illustration. However, the flight trajectory prediction method and system can also be applied to trajectory prediction scenarios for other high-speed, small targets, such as badminton, tennis, and other ball sports, or drones, vehicles, and other objects. In other scenarios, adaptive modifications can be made by referring to the description of this application's embodiments, which will not be elaborated further below.

[0042] The flight trajectory prediction system is used to execute flight trajectory prediction methods, including an image acquisition module, a coordinate transformation module, and a trajectory prediction module.

[0043] First, the planar coordinates of the target are acquired from at least two non-collinear viewpoints using the image acquisition module.

[0044] Non-collinear perspectives refer to intersecting perspectives where multiple cameras or observation points are not on the same straight line. Their optical centers form a plane with the target rather than a straight line, thus allowing them to surround the target from different directions and obtain more complete information.

[0045] For example, a system employing dual-monocular joint positioning is used for image acquisition, and the image acquisition module includes at least two cameras. For instance... Figure 1 As shown, camera 1 is placed on the side of the table tennis table where the ball is hit, and camera 2 is placed on the extension line of the net. This allows the two cameras to intersect, enabling them to capture images from a non-collinear perspective and thus obtain the coordinates of the ping-pong ball. Preferably, camera 1 and camera 2 form a 90° angle. Preferably, camera 1 and camera 2 should be at as similar height as possible and adjusted to be horizontal.

[0046] The table tennis hitting robot will also be equipped with a robotic arm that controls one side of the racket to hit the ball. Camera 1 can be located behind the robotic arm, and its height will be higher than the robotic arm to avoid the robotic arm obstructing its view.

[0047] It should be understood that the target captured from each viewpoint is on a two-dimensional plane, and the target has its own planar coordinates in each viewpoint. For example, in each frame captured by camera 1 and camera 2, the target has its own pixel coordinates, which are planar coordinates, in each frame captured by the camera. The target has planar coordinates (w1, h1) in the frame captured by camera 1, and planar coordinates (w2, h2) in the frame captured by camera 2.

[0048] Then, the coordinate transformation module maps the target's planar coordinates to the world coordinate system to convert them into the target's spatial coordinates.

[0049] The spatial coordinates of a target can be determined by acquiring its planar coordinates from at least two non-collinear viewpoints. The coordinate transformation module combines the planar coordinates acquired from multiple non-collinear viewpoints to obtain the target's spatial coordinates. The coordinate transformation module can employ a multilayer perceptron (MLP), coordinate convolution (CoordConv), Fourier features, or other algorithms for coordinate transformation.

[0050] For example, the coordinate transformation module uses a multilayer perceptron for coordinate transformation. Figure 2 As shown, a multilayer perceptron includes an input layer, hidden layers, and an output layer. The input layer receives the data input, with each node corresponding to a component of the input feature. Hidden layers are located between the input and output layers and typically contain one or more. Each hidden layer consists of multiple neurons (nodes). Each neuron is connected to all neurons in the previous layer via a set of weights and biases. The output layer outputs the results. Specifically, in a coordinate transformation scenario, the input layer contains pixel coordinates (w1, h1) and (w2, h2), which, after passing through several hidden layers, directly perform regression prediction on the spatial coordinate values ​​(x, y, z) in the world coordinate system.

[0051] In one embodiment, the planar coordinates of the target are mapped to the world coordinate system by a multilayer perceptron to convert them into the spatial coordinates of the target. The multilayer perceptron is trained using a first coordinate dataset, which includes the spatial coordinates and planar coordinates of multiple labeled points in space.

[0052] Training a multilayer perceptron can significantly improve the performance of this coordinate transformation module. Training is performed using a dataset of coordinates from multiple spatial points.

[0053] In one embodiment, as follows: Figure 3The six horizontal and six vertical intersection points shown are labeled using a spatial grid. At the same height, there are 6×6=36 labeled points. Labeling is performed at five heights: the surface of the ball table, 10cm above the ball table, 20cm above the ball table, 30cm above the ball table, and 40cm above the ball table, resulting in a total of 180 labeled points. Using the center point of the robotic arm base as the origin of the world coordinate system (0, 0, 0), the coordinates (x, y, z) of these 180 labeled points in the world coordinate system can be obtained. Each labeled point has its own pixel coordinates (w1, h1) and (w2, h2) in the two cameras. Therefore, the process of establishing the spatial coordinate system is the transformation from pixel coordinates (w1, h1) and (w2, h2) to spatial coordinates (x, y, z) in the world coordinate system. These 180 sets of data serve as the training dataset for the neural network.

[0054] In one implementation, the multilayer perceptron is also trained using a second coordinate dataset, which is obtained by augmenting the first coordinate dataset. The data augmentation methods include augmentation based on physical constraints, augmentation based on geometric transformations, augmentation based on data mixing, or augmentation based on model characteristics.

[0055] The data in the first coordinate dataset is transformed to obtain a new second coordinate dataset, and the coordinate transformation module is trained using the new second coordinate dataset.

[0056] There are several ways to transform the data in the first coordinate dataset to obtain a new second coordinate dataset. For example, augmentation methods based on physical constraints, geometric transformations, data fusion, or model characteristics can be used to augment the data in the coordinate dataset, thereby obtaining a new coordinate dataset.

[0057] In one embodiment, a physical constraint-based method is used to augment the training data.

[0058] For example, a pixel is randomly perturbed in the left image, and a matching point is generated on the corresponding epipolar line in the right image based on the epipolar constraint (assuming a horizontal epipolar line). A small-range random offset is added to simulate matching error. By utilizing the epipolar geometry of binocular vision, the physical plausibility of the labeled point pairs in the generated new dataset is ensured, thereby enhancing the robustness of the coordinate transformation module to matching inaccuracies.

[0059] For example, based on the existing labeled points, the disparity value is slightly and randomly scaled, and the corresponding point in another view is recalculated according to the new disparity. The physical relationship between depth and disparity is maintained. By keeping the disparity of the same object within a reasonable range, the coordinate transformation module can understand the correspondence between depth and disparity.

[0060] For example, by selecting spatially adjacent groups of labeled points and applying similar changes to the depth of these points (such as scaling), new samples with consistent depth are generated. Since adjacent points in space should have similar depth values, the coordinate transformation module can learn spatial smoothing constraints.

[0061] In another embodiment, a geometric transformation-based method is used to augment the training data.

[0062] For example, pixel-level noise, such as Gaussian noise (to simulate sensor noise), salt-and-pepper noise (to simulate pixel damage), and quantization noise (to simulate resolution limitations), is injected into existing data. The added noise amplitude is 0.1-2 pixels, and the proportion of added noise is 1%-10% of the pixels. This improves the tolerance of the coordinate transformation module to measurement errors by injecting new data with pixel-level noise.

[0063] For example, randomly occluding a portion of pixels and filling it with the average of adjacent pixels or NaN can simulate scenes where parts of an object are occluded (such as a finger blocking the view), bright spots caused by reflections, or lens smudges. The shape of the occlusion can be rectangular, circular, or irregular. This new data, obtained by randomly occluding a portion of pixels, improves the coordinate transformation module's ability to handle incomplete data.

[0064] For example, affine transformations can be performed on existing data, such as translation (±2 pixels), rotation (±1°), or scaling (98-102%), or perspective transformations can be performed on existing data, to simulate slight camera shake, minor shifts in camera mounting position, lens deformation due to temperature changes, or slight changes in viewing angle. This improves the adaptability of the coordinate transformation module to changes in camera position.

[0065] In another embodiment, the training data is augmented using a data fusion-based approach.

[0066] For example, interpolation is performed on two samples in the feature space, interpolating both pixel coordinates and spatial coordinates while maintaining physical consistency constraints, thereby generating intermediate pose samples.

[0067] For example, two samples are randomly selected, and the input features and output labels are linearly mixed. A mixing coefficient λ: λ ~ Beta(α, α) is sampled from a Beta distribution for MixUp augmentation. Feature space augmentation encourages the coordinate transformation module to learn a smoother decision boundary.

[0068] For example, by randomizing parameters such as changes in lighting conditions, texture changes, or background interference, the coordinate transformation module can focus on geometric relationships and ignore irrelevant features.

[0069] In another embodiment, the training data is augmented based on model characteristics.

[0070] For example, the model gradient of the coordinate transformation module is calculated, and small perturbations are added along the gradient direction to generate "hard" samples. By generating samples that are challenging to the current coordinate transformation module, the performance of the coordinate transformation module at the boundary is proactively improved.

[0071] For example, more samples are generated for regions where the coordinate transformation module's prediction is uncertain, multiple coordinate transformation modules are trained (integrated), regions with large prediction variance are identified, and augmentation is concentrated in these regions, thereby enhancing data through uncertainty sampling to efficiently utilize augmentation resources.

[0072] According to embodiments of this application, a multi-level, multi-strategy data augmentation method can significantly improve the performance of the binocular vision coordinate mapping model in the coordinate transformation module, especially when labeled data is limited. A balance is struck between augmentation diversity and physical plausibility, ensuring that augmented samples both contribute to generalization and do not introduce unreasonable noise.

[0073] In the table tennis robot project, dual cameras are used for positioning and following. However, each positioning point has a three-dimensional deviation, and the trajectory of multiple positioning points will show oscillations perpendicular to the flight direction. It is impossible to use fitting methods to fit and predict the trajectory of the positioning points.

[0074] Therefore, in this embodiment, a convolutional neural network model predicts a second set of spatiotemporal data for the target based on multiple sets of first spatiotemporal data. Each set of first spatiotemporal data includes the spatial coordinates of the target at two adjacent time points and a first time point corresponding to the first set of spatiotemporal data. The second set of spatiotemporal data includes the spatial coordinates of the target at two adjacent time points before and after the target crosses the first plane and a second time point corresponding to the second set of spatiotemporal data.

[0075] The first moment can be either the earlier of two adjacent moments in the first spatiotemporal data set, or the later of two adjacent moments. The second moment can be either the last moment before the target crosses the first plane, or the first moment after the target crosses the first plane.

[0076] The first plane can be the striking plane or any other plane that is set.

[0077] It should be understood that for a ping-pong ball-hitting robot, the hitting plane can be set, therefore it is necessary to predict the position of the ping-pong ball on that plane to complete the hit. The ball's flight direction can be approximated by the ball's position at the last moment before crossing the hitting plane and the first moment after crossing the hitting plane, thereby further obtaining the racket's hitting posture, such as hitting perpendicular to the flight direction.

[0078] The flight trajectory prediction method provided in this application does not make excessive estimates or considerations about the target's flight process. It only predicts the moment when the target crosses the first plane, as well as the spatial coordinates of the last moment before the target crosses the hitting plane and the spatial coordinates of the first moment after the target crosses the hitting plane, based on multiple sets of data from the initial or mid-flight phase of the target.

[0079] In one embodiment, a convolutional neural network model predicts the offset of the target's coordinates relative to a preset center point at two adjacent moments before and after the target passes through the first plane based on multiple sets of first spatiotemporal data. A second spatiotemporal data set is then obtained based on the offset and the coordinates of the center point.

[0080] Furthermore, the trajectory of the target across the first plane is predicted using a convolutional neural network model based on the second spatiotemporal data set.

[0081] In one embodiment, the spatial coordinates in multiple sets of first spatiotemporal data groups correspond to multiple consecutive moments.

[0082] For example, Table 1 shows the construction method of the first spatiotemporal data input, with each row representing a set of first spatiotemporal data. Each row of data contains two spatial coordinate information and one time information, where the time information represents the time value of the first spatial coordinate information. The first spatiotemporal data set in the first row includes the spatial coordinates (x1, y1, z1) of the target at time 1 and the spatial coordinates (x2, y2, z2) of the target at time 2, along with time 1. The first spatiotemporal data sets in the subsequent five rows are constructed and understood with reference to the first row.

[0083] Table 1. Construction method of the first spatiotemporal data group (continuous time).

[0084] In another embodiment, the multiple first moments corresponding to the multiple sets of first spatiotemporal data groups are discontinuous moments, and each set of first spatiotemporal data groups includes the corresponding first moment to characterize the temporal relationship of the multiple sets of first spatiotemporal data groups.

[0085] It should be understood that Table 1 uses coordinate information from 12 consecutive time points to construct the data. These 12 data points may not be consecutive, but they need to be represented by the time value at the end of the row.

[0086] For example, as shown in Table 2, the input data is constructed in the case of discontinuous data, with 12 time points: 1, 2, 5, 6, 7, 8, 9, 10, 11, 12, 15, and 16. Discontinuous data is also indicated by the end-of-line data.

[0087] Table 2. Construction method of the first spatiotemporal data group (non-continuous time).

[0088] The second spatiotemporal data group can be constructed with reference to the format of the first spatiotemporal data group.

[0089] As shown in Table 3, the second spatiotemporal data group of the output is constructed in the same way, which consists of two spatial coordinate information and one time information. Time m and time m+1 are consecutive times. Time m is the last time before crossing the first plane, and time m+1 is the first time after crossing the first plane. The data at the end of the row is the time m corresponding to the first spatial coordinate.

[0090] Table 3. Construction method of the second spatiotemporal data group

[0091] For example, based on 12 sets of data (including time values) at the initial or mid-flight stage, the next hitting time m, as well as the ball positions at times m and m+1, are directly predicted and output. The flight direction can be approximated from the ball positions at times m and m+1, thereby further obtaining the racket's hitting posture.

[0092] The convolutional neural network model establishes multiple mappings from a first spatiotemporal data set to a second spatiotemporal data set. Training the convolutional neural network model can improve the accuracy of regression.

[0093] The training process of a convolutional neural network model is as follows: Figure 4 As shown, it is divided into data preparation, model training, optimization strategy and result saving.

[0094] In order to improve the accuracy of regression and avoid blindly predicting the final result by the convolutional neural network model, constraints are added to the regression coordinates at time m and time m+1, based on the actual scenario of ping-pong ball flight trajectory positioning.

[0095] See also Figure 1 A coordinate system is established using the ball table as the coordinate origin, with the fulcrum of the robotic arm as the origin. The x-axis points horizontally upwards, the y-axis points horizontally to the left, and the z-axis points vertically upwards from the xy-plane. Assuming the incoming ball travels from left to right, the predicted y-coordinate of the first plane the ball will pass through is 0.2 m. The output of the convolutional neural network model is the predicted coordinates of the ball at the nearest moment *m* before passing through the first plane and the coordinates at the first moment (*m+1*) after passing through the first plane, along with the time value *m*, which represents the spatial coordinates of the two moments before and after passing through the first plane, and the time value *m* before passing through the plane. Based on the actual flight range, at time *m*, the y-value is slightly greater than 0.2, and at time (*m+1*), the y-value is slightly less than 0.2. The x-axis is limited to ±0.5 within the above coordinate system, and the z-value is limited to 0.5 above the ball table. The range of the output coordinates is shown in Table 4. Time *m* represents the last moment before passing through the plane. Because multiple sets of input data are based on the time of the first data point, *m* is no exception; therefore, *m* represents the time of the first data point in a relative set of data.

[0096] Table 4. Predicted Output Coordinate Range

[0097] During the data preparation phase, trajectory data is read from Excel. For each set of trajectory data, the transition point of y from ≥0.2 to <0.2 is detected. Based on the relative time and the prediction time interval (more than 10 time values), training samples are constructed, noise is added, and data augmentation is performed.

[0098] For example, 68 sets of trajectory data are loaded from Excel, 65 sets of training data are extracted, the transition point of y from ≥0.2 to <0.2 is detected, training samples are constructed, multiple samples are generated, and Gaussian noise is added for data augmentation.

[0099] During the model training phase, the model is first initialized, the optimizer is set, and then a training loop is performed. The loss is calculated by forward propagation, the gradient is updated by backpropagation, and the evaluation and monitoring metrics are verified.

[0100] During the training of a convolutional neural network (CNN) model, the goal is to make the model's output as close as possible to the actual target value. This is achieved by comparing the current predicted value with the true target value and updating the weight vector of each layer based on the difference. For example, if the predicted value is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the CNN model can predict the true target value or a value very close to it. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values.

[0101] In one embodiment, during the training of the convolutional neural network model, a first loss function is used to compare the difference between the predicted value and the target value output by the convolutional neural network model. The first loss function includes offset loss, boundary condition loss, orientation constraint loss and range constraint loss to make the predicted value meet the preset physical constraints.

[0102] like Figure 5 As shown, the output of the convolutional neural network model includes two sets of coordinates and one time value. The loss function design includes coordinate loss, offset loss (core), time loss, boundary condition loss, orientation constraint loss, and range constraint loss. The total loss is obtained by weighting the individual losses: Total loss = a*coordinate loss + b*offset loss + c*time loss + d*boundary condition loss + e*orientation constraint loss + f*range constraint loss.

[0103] For example, the various loss functions are weighted to obtain the final total loss function. Currently, the weighting coefficients used are 1.0, 2.0, 0.5, 1.5, 0.3, and 0.1, which can be adjusted according to the actual training results. Total loss = 1.0 * coord_loss + 2.0 * offset_loss + 0.5 * time_loss + 1.5 * boundary_loss + 0.3 * direction_loss + 0.1 * range_loss.

[0104] The coordinate loss is used to ensure that the output coordinates are close to the true coordinates. The coordinate loss `coord_loss = SmoothL1Loss(pred_coords, target_coords)`. The offset loss, as the core supervision signal, ensures accurate offset prediction. The offset loss `offset_loss = MSE(tanh(x1_pred), x1_target) + MSE(tanh(y1_pred), y1_target) + MSE(sigmoid(y2_pred), y2_target)`. The boundary loss is used to enforce the physical constraints: y1 ≥ 0.2, y2 < 0.2; the boundary loss `boundary_loss = ReLU(0.2 - y1_pred + 0.02)`. 2 +Relu(y2_pred-0.2+0.02) 2 y1 cannot be too small, and y2 cannot be too large. Time loss is used to ensure accurate time prediction; time_loss = L1Loss(pred_normalized_time, target_normalized_time). Since the y-value decreases after transformation, direction constraint loss is used to ensure y2 is less than y1; direction constraint loss = ReLU(y2_pred - y1_pred + 0.05). 2 .

[0105] During training, the loss gradually converges to near its minimum, and the model learns the mapping relationship from input to output. Early stopping, learning rate scheduling, and gradient pruning are employed to optimize the model training phase.

[0106] For example, the optimizer uses weight decay (AdamW), the learning rate (lr) is set to 0.0005, and the training cycle is set to 200 epochs. The early stopping tolerance epoch is set to 50.

[0107] During the results saving phase, the best model is saved, along with all metrics from the training history. Visual charts of the training process are generated, and configuration file hyperparameters are saved.

[0108] Convolutional neural network model design faces three challenges: inconsistent coordinate scales (x, y, z coordinates in spatial coordinates have different physical ranges and distributions), special time characteristics (time has monotonically increasing and relative properties), and complex output constraints (it needs to satisfy physical boundary conditions (y≥0.2, y<0.2)).

[0109] Therefore, as shown in Table 5, a hierarchical normalization strategy is designed, which adopts different normalization methods for different types of data, and handles input coordinate normalization, output coordinate normalization, and relative time normalization respectively.

[0110] In one embodiment, the input multiple sets of first spatiotemporal data are normalized, wherein the spatial coordinates are normalized according to preset minimum spatial coordinate parameters and spatial range parameters; and the first moment is normalized according to the ratio of the first moment to the maximum possible time.

[0111] Table 5. Normalization Method

[0112] In one embodiment, the input spatial coordinates are normalized using Min-Max.

[0113] For example, statistical calculations are performed to compute the minimum and maximum values ​​for each coordinate column across the entire training set. The minimum value, `coord_min`, is a 6-dimensional vector corresponding to the minimum value of [x1, y1, z1, x2, y2, z2]. The maximum value, `coord_max`, is a 6-dimensional vector corresponding to the maximum value of [x1, y1, z1, x2, y2, z2]. The range, `coord_range`, is the maximum value `coord_max` minus `coord_min` plus ε (to prevent division by zero). Therefore, the normalized coordinates = (original spatial coordinates - `coord_min`) / `coord_range`.

[0114] According to embodiments of this application, training set statistics are used, and the same parameters are reused during inference. Each coordinate column is independently normalized, respecting its respective data distribution. The normalized coordinates are within the range of [0, 1]. By mapping the input features to the range of [0, 1], it is easier for the convolutional neural network model to process them, thus accelerating model convergence.

[0115] The input spatial coordinates can also be normalized in other ways, and the normalized coordinates can be mapped to the range of [0, 1].

[0116] In one embodiment, the output spatial coordinates are normalized using center point offset.

[0117] For example, the design incorporates a physical model, setting the center point x=0 (center of the table), center point y=0.2 (boundary threshold), and center point z=0.5 (midpoint height). To control the offset range, a scaling factor of 0.5 is set. For the point before transformation, i.e., the spatial coordinates at time m (x1, y1, z1), then the offset of x1 = (x1 - 0.0) / 0.5; the offset of y1 = (y1 - 0.2) / 0.5; and the offset of z1 = (z1 - 0.5) / 0.5. For the point after transformation, i.e., the spatial coordinates at time m+1 (x2, y2, z2), y2 is specially treated to ensure it is positive, and the offset of y2 = (0.2 - y2) / 0.5.

[0118] According to embodiments of this application, implicit physical constraints are designed by offset calculation. The offset is usually in the range of [-1, 1]. The offset intuitively reflects the degree of deviation from the center, so that the convolutional neural network model does not directly predict the absolute coordinates, but predicts the offset relative to the theoretical center point.

[0119] In one embodiment, time is uniformly normalized. Considering the special nature of time data, all time values ​​can be divided by the range 100.0. 100 is a decimal-friendly number, easy to understand and debug. For input time value normalization, each time point in the input sequence has a relative time, relative time = current time index - starting index, so regardless of the starting point, the normalized value of the input time is the same (0.0, 0.02, 0.04, ...). For output time values, output time = transition point index - starting index, normalized: output time / 100.0. Theoretically, with an observation window length of 12, the maximum possible output time is 12, and the normalization is 12 / 100 = 0.12, safely within the range [0, 1]. In actual training data, the maximum output time is 44, and the normalization is 44 / 100 = 0.44, still with a sufficient safety margin.

[0120] For example, if the transition point is at absolute index 40, then when the starting point is 1, the relative time is 39 and the normalization is 0.39; when the starting point is 11, the relative time is 29 and the normalization is 0.29; when the starting point is 21, the relative time is 19 and the normalization is 0.19.

[0121] According to embodiments of this application, the output time varies with the starting point, reflecting the essence of "relative time," and the convolutional neural network model learns to predict different output times from the same input pattern.

[0122] like Figure 6 and Figure 7 As shown, multiple sets of first spatiotemporal data are input into the convolutional neural network model. Features are extracted by the feature extractor, then the features are aggregated, and finally decoded by the fully connected decoder to obtain the second spatiotemporal data set.

[0123] The convolutional neural network model is configured to: extract and aggregate features from multiple sets of first spatiotemporal data using a feature extractor; decode the aggregated features using a fully connected decoder to obtain the original prediction data set, which includes the offset of the coordinates relative to the preset center point and the time value at two adjacent time points after normalization; and output the second spatiotemporal data set after decoding the original prediction data set.

[0124] For example, such as Figure 6 As shown, the original spatial coordinate data is sampled, with each sample containing 6 time point pairs, thus outputting 6 sets of first spatiotemporal data. Each set of first spatiotemporal data is a 7-dimensional feature vector, which is then input into the convolutional neural network model in batches.

[0125] For example, such as Figure 7 As shown, the input data is processed by a feature extractor and then reshaped. In Convolution Block 1, a 3×3 convolutional kernel transforms the 1-channel input feature map into a 32-channel output feature map while maintaining the spatial resolution. Then, batch normalization (BN) and rectified linear units (ReLU) are used for further processing. Subsequent steps are similar and will not be elaborated further. Next, the fully connected decoder transforms the high-dimensional feature maps extracted by the convolution into the final required output form. The FC 896→256 layer linearly maps the flattened 896-dimensional convolutional features to a 256-dimensional compact latent space through a fully connected layer, and introduces element-wise nonlinearity and a dropout rate (Dropout=0.3) for regularization to suppress overfitting. Subsequent steps are similar and will not be elaborated further. Finally, output decoding is performed.

[0126] Because the input data has been normalized and the offset has been predicted, the output of the convolutional neural network model needs to be decoded. The process is as follows: Figure 8 As shown.

[0127] For example, the 7-dimensional vector output by the convolutional neural network model is [x1_raw, y1_raw, z1_raw, x2_raw, y2_raw, z2_raw, time_raw]. Then, offset decoding is performed: the relative time is time = time_raw * 100, and the time is directly obtained by inverse normalization. The offset x1_offset is obtained by multiplying x1_raw by 0.5 after applying tanh, restoring it to [-0.5, 0.5]; the offset y1_offset is obtained by multiplying y1_raw by 0.5 after applying tanh, restoring it to [-0.5, 0.5], then activated by ReLU to ensure it is non-negative, and then 0.01 is added to ensure y1_offset ≥ 0.01; the offset z1_offset is obtained by multiplying z1_raw by 0.5 after applying tanh, restoring it to [-0.5, 0.5]. The offset x2_offset is obtained by multiplying x2_raw by 0.5 after applying tanh, restoring it to [-0.5, 0.5]; the offset y2_offset is obtained by multiplying y2_raw by 0.5 after applying sigmoid, constraining y2_offset to (0, 0.5]; the offset z2_offset is obtained by multiplying z2_raw by 0.5 after applying tanh, restoring it to [-0.5, 0.5]. After obtaining the offsets, the spatial coordinates of the output point before transformation (x1, y1, z1) and the spatial coordinates after transformation (x2, y2, z2) can be calculated. Where: x1 = 0.0 + x1_offset; y1 = 0.2 + y1_offset; z1 = 0.5 + z1_offset; x2 = 0.0 + x2_offset; y2 = 0.2 - y2_offset; z2 = 0.5 + z2_offset.

[0128] For a table tennis hitting robot, the flight trajectory prediction results based on the output of a convolutional neural network model can generate the hitting time, hitting point, and hitting posture.

[0129] The spatial coordinates of the two points at time m and time m+1 can roughly represent the direction of the incoming ball. The opposite direction of the line connecting the two coordinates can represent the direction of the shot. The plane perpendicular to the line is the plane in which the racket contacts the ball. Combining this with time m, we can determine when, in what posture, and from which position the ball is hit.

[0130] In summary, such as Figure 9As shown, the flight trajectory prediction method provided in this application differs from traditional methods. Traditional methods directly predict absolute coordinates, resulting in a large numerical range and inconsistencies across different scales, leading to difficulties in model training and limited prediction accuracy. In contrast, the scheme in this application uses neutral point offset representation to standardize the numerical offset range [-1, 1], unifies time normalization to clarify the physical meaning, and utilizes intelligent boundary constraints to achieve strong constraint guarantees, ensuring 100% satisfaction of boundary conditions. This enables stable training and rapid convergence of the convolutional neural network model.

[0131] In practical applications, the y2 error predicted by the scheme in this application is 0.0125, which is much smaller than the 0.12 of the traditional method. The boundary satisfaction rate is 100%, the time error is 1.64 steps, and the generalization ability is strong.

[0132] According to this scheme, for coordinate data containing error information, the feature extraction capability of convolutional neural networks is used to extract flight trajectory features, and then subsequent trajectory data is predicted. The positioning deviation is used as noise in the network training to eliminate the influence of positioning deviation on trajectory prediction. This solves the problems of trajectory fitting difficulties and subsequent decision failures caused by positioning deviation and inaccurate initial positioning, and improves the accuracy of flight trajectory prediction.

[0133] Furthermore, based on the flight trajectory prediction method proposed in the embodiments of this application, the embodiments of this application also propose a system for implementing the flight trajectory prediction.

[0134] like Figure 10 The flight trajectory prediction system includes an image acquisition module configured to acquire the planar coordinates of the target from at least two non-collinear viewpoints.

[0135] The coordinate transformation module is configured to map the planar coordinates of the target to the world coordinate system to convert them into the target's spatial coordinates.

[0136] The trajectory prediction module is configured to use a convolutional neural network model to predict a second set of spatiotemporal data of the target based on multiple sets of first spatiotemporal data. The second set of spatiotemporal data includes the spatial coordinates of the target at two adjacent moments before and after the target passes through the first plane and the second moment corresponding to the second set of spatiotemporal data. Each set of first spatiotemporal data includes the spatial coordinates of the target at two adjacent moments and the first moment corresponding to the first set of spatiotemporal data.

[0137] In some possible embodiments, the trajectory prediction module is also configured to predict the trajectory of the target across the first plane based on the second spatiotemporal data set.

[0138] In some possible embodiments, the trajectory prediction module uses a convolutional neural network model to predict the offset of the target's coordinates relative to a preset center point at two adjacent moments before and after it crosses the first plane, based on multiple sets of first spatiotemporal data. The second spatiotemporal data set is then obtained based on the offset and the coordinates of the center point.

[0139] In some possible embodiments, the convolutional neural network model is configured to: extract and aggregate features from multiple sets of first spatiotemporal data using a feature extractor; decode the aggregated features using a fully connected decoder to obtain an original prediction data set, the original prediction data set including the offset of the coordinates relative to a preset center point at two adjacent time points after normalization and the time value; and output a second spatiotemporal data set after decoding the original prediction data set.

[0140] In some possible embodiments, the trajectory prediction module further includes: normalizing multiple sets of first spatiotemporal data, wherein the spatial coordinates are normalized according to preset minimum spatial coordinate parameters and spatial range parameters; and the first moment is normalized according to the ratio of the first moment to the maximum possible time.

[0141] In some possible embodiments, the planar coordinates of the target are mapped to the world coordinate system by a multilayer perceptron to convert them into the spatial coordinates of the target. The multilayer perceptron is trained using a first coordinate dataset, which includes the spatial and planar coordinates of multiple labeled points in space.

[0142] In some possible embodiments, the multilayer perceptron is also trained using a second coordinate dataset, which is obtained by augmenting the first coordinate dataset. The data augmentation methods include augmentation based on physical constraints, augmentation based on geometric transformations, augmentation based on data mixing, or augmentation based on model characteristics.

[0143] In some possible embodiments, a first loss function is used during the training of the convolutional neural network model to compare the difference between the predicted value and the target value output by the convolutional neural network model. The first loss function includes offset loss, boundary condition loss, orientation constraint loss and range constraint loss to make the predicted value meet the preset physical constraints.

[0144] In some possible embodiments, the spatial coordinates in multiple sets of first spatiotemporal data groups correspond to multiple consecutive moments. Alternatively, the multiple first moments corresponding to multiple sets of first spatiotemporal data groups are discontinuous moments, and each set of first spatiotemporal data groups includes a corresponding first moment to characterize the temporal relationship of the multiple sets of first spatiotemporal data groups.

[0145] Furthermore, this application also proposes a table tennis robot hitting system, which includes the aforementioned flight trajectory prediction system and hitting module. The hitting module is used to hit the ball according to the predicted trajectory output by the flight trajectory prediction system.

[0146] Furthermore, this application also proposes a flight trajectory prediction device, which includes a processor and a memory connected together. The memory is used to store program code, and the processor is used to call the program code to execute any of the flight trajectory prediction methods proposed in this application.

[0147] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0148] This application also provides a computer-readable medium storing program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0149] Although this application has been described with reference to preferred embodiments, various modifications can be made thereto and components or steps can be replaced with equivalents without departing from the scope of this application. In particular, the technical features mentioned in the various embodiments can be combined in any manner, provided there is no conflict in structure or method steps. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

[0150] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A flight trajectory prediction method, characterized in that, The method includes: The planar coordinates of the target to be predicted are acquired from at least two non-collinear viewpoints; Map the planar coordinates of the target to the world coordinate system to convert them into the spatial coordinates of the target; A convolutional neural network model predicts a second set of spatiotemporal data for the target based on multiple sets of first spatiotemporal data. The second set of spatiotemporal data includes the spatial coordinates of the target at two adjacent moments before and after the target crosses the first plane, and the second moment corresponding to the second set of spatiotemporal data. Wherein: Each first spatiotemporal data group includes the spatial coordinates of the target at two adjacent times and the first time corresponding to the first spatiotemporal data group.

2. The method according to claim 1, characterized in that, The method further includes: The trajectory of the target across the first plane is predicted based on the second spatiotemporal data set.

3. The method according to claim 1 or 2, characterized in that, The step of predicting the second spatiotemporal data set of the target based on multiple first spatiotemporal data sets using a convolutional neural network model specifically includes: The offset of the target's coordinates relative to a preset center point at two adjacent moments before and after it crosses the first plane is predicted by a convolutional neural network model based on the multiple sets of first spatiotemporal data. The second spatiotemporal data group is obtained based on the offset and the coordinates of the center point.

4. The method according to any one of claims 1-3, characterized in that, The convolutional neural network model is configured as follows: The input multiple sets of first spatiotemporal data are subjected to feature extraction and aggregation processing using a feature extractor. The aggregated features are decoded by a fully connected decoder to obtain the original prediction data set, which includes the offset of the coordinates relative to the preset center point and the time value at two adjacent time points after normalization. The second spatiotemporal data group is output after decoding the original prediction data group.

5. The method according to claim 4, characterized in that, The method further includes: normalizing the input multiple sets of first spatiotemporal data groups, wherein, The spatial coordinates are normalized according to the preset minimum spatial coordinate parameters and spatial range parameters; The first moment is normalized based on the ratio of the first moment to the maximum possible time.

6. The method according to any one of claims 1-5, characterized in that, The step of mapping the planar coordinates of the target to the world coordinate system to convert them into the spatial coordinates of the target includes: The planar coordinates of the target are mapped to the world coordinate system by a multilayer perceptron to convert them into the spatial coordinates of the target. The multilayer perceptron is trained using a first coordinate dataset, which includes the spatial coordinates and planar coordinates of multiple labeled points in space.

7. The method according to claim 6, characterized in that, The multilayer perceptron is also trained using a second coordinate dataset, which is obtained by augmenting the first coordinate dataset. The data augmentation methods include augmentation based on physical constraints, augmentation based on geometric transformations, augmentation based on data mixing, or augmentation based on model characteristics.

8. The method according to any one of claims 3-7, characterized in that, During the training of the convolutional neural network model, a first loss function is used to compare the difference between the predicted value and the target value output by the convolutional neural network model. The first loss function includes offset loss, boundary condition loss, orientation constraint loss and range constraint loss to ensure that the predicted value meets the preset physical constraints.

9. The method according to any one of claims 1-8, characterized in that, Multiple time points corresponding to the spatial coordinates in the first spatiotemporal data group are consecutive time points; or The multiple first time points corresponding to the multiple sets of first spatiotemporal data groups are discontinuous time points, and each set of first spatiotemporal data groups includes the corresponding first time point to characterize the temporal relationship of the multiple sets of first spatiotemporal data groups.

10. A flight trajectory prediction system, characterized in that, Includes modules or units for performing the method as described in any one of claims 1 to 9.

Images