Trajectory prediction method and computing system for trajectory prediction

Abstract

The present application provides a trajectory prediction method and a computing system for trajectory prediction. In the method, the historical trajectories of a plurality of target objects are respectively processed by an encoder to obtain features to generate first trajectory information of the target objects. The first trajectory information of the target objects is processed by a pooling to generate second trajectory information of the target objects. The second trajectory information of each target object includes a positional relationship with respect to other target objects. Third trajectory information is obtained from the historical trajectories of the target objects. The third trajectory information includes a moving direction, scene information and / or a moving pattern. The second trajectory information and the third trajectory information of the target objects are processed by a decoder to generate a predicted trajectory of the target objects. In this way, the accuracy of the prediction can be improved.

Description

Technical Field

[0001] This invention relates to a trajectory processing technique, and more particularly to a trajectory prediction method and a computational system for trajectory prediction. Background Technology

[0002] Trajectory prediction is a common component in many smart applications. The prediction results can be used as a reference to automate alerts or adjust parameters. The trajectory prediction task includes object detection and identification processes. Through these processes, target objects can be further filtered out, and trajectory prediction can be performed only on these target objects, thereby improving application efficiency.

[0003] Traditional object detection and recognition methods rely on two-dimensional (2D) information acquired by cameras. However, relying solely on color information from 2D data can easily lead to performance instability in many applications. For example, misjudging human-shaped standees or misjudging volume due to angle. Therefore, in recent years, sensors capable of acquiring three-dimensional (3D) information have been introduced in applications requiring high detection and recognition stability (e.g., smart roadside devices, robots, and autonomous vehicles). However, the sheer volume of 3D information increases the system's workload and makes it difficult to meet the real-time requirements of applications.

[0004] It is also worth noting that, in addition to the problem that existing literature focuses on pedestrians as the target object, existing literature also fails to achieve accurate trajectory prediction for multiple target objects. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a trajectory prediction method and a computing system for trajectory prediction, which can be applied to various types of objects and can improve prediction efficiency.

[0006] This invention proposes a trajectory prediction method based on a prediction model. The prediction model includes a generator network, and the trajectory prediction method includes (but is not limited to) the following steps: First trajectory information of the target objects is generated by extracting features from the historical trajectories of multiple target objects using an encoder. The encoder is used to obtain hidden states representing the movement characteristics reflected by the historical trajectories, and the first trajectory information includes these hidden states. Second trajectory information of the target objects is generated by pooling the first trajectory information. The pooling process is used to obtain the relationship between the first trajectory information of the target objects, and the second trajectory information of each target object includes its positional relationship relative to other target objects. Third trajectory information is obtained from the historical trajectories of the target objects. The third trajectory information includes movement direction, scene information, and / or movement pattern. The predicted trajectory of the target objects is generated by decoding the second and third trajectory information using a decoder.

[0007] The computational system for trajectory prediction according to embodiments of the present invention includes (but is not limited to) a memory and a processor. The memory stores program code. The processor is coupled to the memory. The processor is configured to load and execute the program code to execute a prediction model. The prediction model includes a generator network. The processor is further configured to: acquire features from the historical trajectories of multiple target objects using an encoder to generate first trajectory information of the target objects; generate second trajectory information of the target objects by pooling the first trajectory information; obtain third trajectory information from the historical trajectories of the target objects; and generate a predicted trajectory of the target objects by decoding the second and third trajectory information of the target objects. The encoder is used to obtain hidden states representing the movement characteristics reflected by the historical trajectories, and the first trajectory information includes the hidden states. The pooling process is used to obtain the relationship between the first trajectory information of the target objects, and the second trajectory information of each target object includes its positional relationship relative to other target objects. The third trajectory information includes movement direction, scene information, and / or movement pattern.

[0008] Based on the above, the trajectory prediction method and the computational system for trajectory prediction in this embodiment of the invention refer to more trajectory information, thus making them applicable to trajectory prediction of multiple target objects.

[0009] To make the above features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a block diagram of a computing system according to an embodiment of the present invention.

[0011] Figure 2 This is a flowchart of a trajectory prediction method according to an embodiment of the present invention.

[0012] Figure 3A This is a schematic diagram of a generator network according to an embodiment of the present invention.

[0013] Figure 3B This is a schematic diagram of a generator network and a discriminator network according to an embodiment of the present invention.

[0014] Figure 4 This is a flowchart of trajectory generation according to an embodiment of the present invention.

[0015] Figure 5 This is a flowchart of point cloud information processing according to an embodiment of the present invention.

[0016] Figure 6 This is a flowchart illustrating the determination of the size of a three-dimensional pixel (voxel) based on a dimension according to an embodiment of the present invention.

[0017] Figure 7This is a flowchart of training stereo pixel size according to an embodiment of the present invention.

[0018] Figure 8A and Figure 8B These are point cloud images under two stereo pixel sizes according to an embodiment of the present invention.

[0019] Figure 9 This is a flowchart of a depth-based stereo pixel size determination according to an embodiment of the present invention.

[0020] Figure 10A and Figure 10B This is a schematic diagram of depth clustering according to an embodiment of the present invention.

[0021] Figures 11A to 11C These are point cloud images with different stereo pixel sizes according to an embodiment of the present invention.

[0022] Figure 12A This is a schematic diagram of a trajectory according to an embodiment of the present invention.

[0023] Figure 12B This is a schematic diagram of angular grouping according to an embodiment of the present invention.

[0024] Figure 13 This is a flowchart generated based on scene information according to an embodiment of the present invention.

[0025] Figure 14 This is a schematic diagram of the projection result according to an embodiment of the present invention.

[0026] Figure 15 This is a schematic diagram of the trajectories of multiple movement modes according to an embodiment of the present invention.

[0027] Figure 16 This is a schematic diagram of trajectory filtering according to an embodiment of the present invention.

[0028] Figure 17A This is a schematic diagram of the probability distribution of noise corresponding to a left-turn mode according to an embodiment of the present invention.

[0029] Figure 17B This is a schematic diagram of the probability distribution of noise corresponding to the non-moving mode according to an embodiment of the present invention.

[0030] Figure 17C This is a schematic diagram of the probability distribution of noise corresponding to the straight-running mode according to an embodiment of the present invention.

[0031] Symbol explanation:

[0032] 10: Arithmetic unit;

[0033] 11: Memory;

[0034] 12: Processor;

[0035] S210~S240, S410~S430, S510~S570, S610~S620, S611~S612, S910~S920, S131~S133: Steps;

[0036] 30: Generator Networks;

[0037] 31: Encoder;

[0038] 32: Decoder;

[0039] 33: Pooling module;

[0040] 34: Direction module;

[0041] 35: Scene Module;

[0042] 36: Distribution Calculator;

[0043] I1: First trajectory information;

[0044] I2: Second trajectory information;

[0045] I3: Third trajectory information;

[0046] 40: Discriminator Network;

[0047] O1, O2: Target objects;

[0048] ROI1, ROI2: Regions of Interest;

[0049] PC: 3D point cloud map;

[0050] VP: Viewing position;

[0051] FOV: Field of View;

[0052] P1~P5: Sampling points;

[0053] Obs1~Obs4: Observe the trajectory. Detailed Implementation

[0054] Figure 1 This is a block diagram of the components of a computing system 10 according to an embodiment of the present invention. Please refer to... Figure 1 The computing system 10 includes (but is not limited to) one or more memories 11 and one or more processors 12. The computing system 10 may be one or more desktop computers, laptop computers, smartphones, tablets, servers or other electronic devices.

[0055] The memory 11 can be any form of fixed or removable random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), or similar component. In one embodiment, the memory 11 is used to store program code, software modules, configuration settings, data, or files (e.g., trajectory information, model parameters, image information, or scale).

[0056] Processor 12 is coupled to memory 11. Processor 12 may be a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), or other programmable general-purpose or special-purpose microprocessors, digital signal processors (DSPs), programmable controllers, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), neural network accelerators, or other similar components or combinations thereof. In one embodiment, processor 12 is used to execute all or part of the operations of computing system 10, and can load and execute various program codes, software modules, files, and data stored in memory 11. In some embodiments, some operations in the methods of this embodiment may be implemented by different or the same processor 12.

[0057] The methods described in the embodiments of the present invention will be described below in conjunction with the various devices, components, and modules in the computing system 10. The various processes of this method may be adjusted according to the implementation, and are not limited thereto. In some embodiments, some processes may be implemented by different processors 12.

[0058] Figure 2 This is a flowchart of a trajectory prediction method according to an embodiment of the present invention. Please refer to... Figure 2The processor 12 extracts features from the historical trajectories of one or more target objects using an encoder to generate first trajectory information for the target objects (step S210). Specifically, the target objects can be people, other animals, vehicles, living creatures, or other objects. Multiple target objects may have the same or different object types. The historical trajectory of each target object includes its historical location. For example, at a given time point t, the historical trajectory could be the locations at time points t-5, t-4, t-3, t-2, and t-1. The locations arranged chronologically in the trajectory may be at the same or different intervals. Furthermore, these locations can be latitude and longitude, relative positions, or coordinates in a specific coordinate system.

[0059] On the other hand, processor 12 executes the prediction model stored in memory 11. The prediction model is used to generate subsequent predicted trajectories based on historical trajectories. The prediction model is based on a social generative adversarial network (GAN) architecture. This prediction model includes a generator network. Figure 3A This is a schematic diagram of a generator network 30 according to an embodiment of the present invention. Please refer to... Figure 3A The generator network 30 includes an encoder 31 and a decoder 32, i.e., an encoder-decoder architecture. Historical trajectories are input to the encoder 31. In one embodiment, the encoder 31 is a Long Short-Term Memory (LSTM) module for one or more target objects. The LSTM model is a Recurrent Neural Network (RNN) used for feature extraction. In some applications, the LSTM module is suitable for processing and predicting important events with long intervals and delays in time series, such as trajectory prediction. The encoder 31 is used to obtain hidden states representing the movement characteristics reflected by the historical trajectory, and the first trajectory information includes the hidden states. For example, the latent features of the i-th person (i.e., a target object) in the scene captured by the LSTM module at time t are the hidden states. The hidden states at different times capture the movement characteristics that change over time. Movement characteristics can be speed, acceleration, stride length, and / or direction (i.e., the features obtained from feature extraction).

[0060] In other embodiments, encoder 31 may use other neural networks. For example, RNNs, Continuous Recurrent Neural Networks (C-RNNs), or other networks suitable for acquiring temporal features. In some embodiments, processor 12 may also form fixed-length positions in historical trajectories using word embeddings before inputting them to an LSTM or other module for feature acquisition.

[0061] Further details of the generator network 30 will be described in subsequent embodiments. It is also worth noting that the output of the generator network 30 is used by the discriminator network to determine authenticity. Authenticity is, for example, true or false. Figure 3B This is a schematic diagram of a generator network 30 and a discriminator network 40 according to an embodiment of the present invention. Please refer to... Figure 3B The output of generator network 30 is the predicted trajectory. Discriminator network 40 may include an encoder. This encoder may be an LSTM or other model. The trained discriminator network 40 can determine whether the predicted trajectory is true or false.

[0062] On the other hand, the location in historical trajectories can be obtained not only through satellite positioning and base station positioning, but also through image positioning. Figure 4 This is a flowchart illustrating trajectory generation according to an embodiment of the present invention. Please refer to... Figure 4 The processor 12 can acquire one or more images acquired by one or more image acquisition devices (not shown). The image acquisition device can be a two-dimensional or three-dimensional camera, video camera, three-dimensional scanner, depth sensor, monitor, smartphone, dashcam, or roadside unit or mobile robot with image acquisition capabilities, and acquire two-dimensional and / or three-dimensional images within a specified field of view.

[0063] Processor 12 determines the position of the target object in three-dimensional space (step S410). Taking a three-dimensional point cloud map (i.e., a three-dimensional image) as an example, this point cloud map includes one or more sensing points of the target object. Each sensing point records the depth or distance of a local area of ​​the target object relative to the viewing position. The viewing position can be the location of the image acquisition device. Processor 12 can identify the object type using image recognition technology and determine a specific object type accordingly. For example, a pet, an adult, or a child. Next, processor 12 can compare the image recognition results with the three-dimensional point cloud map and select a region of interest (RoI) or bounding box for this object type in the three-dimensional point cloud map. In the three-dimensional point cloud map, this region of interest or bounding box encompasses the target object and can represent the target object's position in three-dimensional space. For two-dimensional images, processor 12 can convert the target object's position in the image (derived from the image recognition results) into its position in three-dimensional space using homography transformation or other camera space transformations.

[0064] It is worth noting that point cloud information processing methods can be divided into "voxel-based" and "point-based" methods. Point-based methods can be further divided into "ordered point clouds" and "unordered point clouds." Voxel-based methods downsample the 3D point cloud image by setting the size of the voxel (hereinafter referred to as the voxel size). If the voxel size is small, the memory space used in memory 11 is quite large (possibly a cubic growth), but the loss is small, and the resolution is high. If the voxel size is large, the loss is large, but the resolution may be reduced, and the memory space used in memory 11 is relatively small. Compared to voxel-based methods, point-based methods may have additional space consumption or irregular access issues.

[0065] It is also worth noting that the Point-Voxel Convolution Neural Network (PVCNN) combines point-based and 3D pixel-based operations. Figure 5 This is a flowchart of point cloud information processing according to an embodiment of the present invention. Please refer to... Figure 5 The upper branch of the PVCNN architecture performs basic operations on stereo pixels to establish neighborhood relationships between stereo pixels and construct neighborhood features accordingly. On the other hand, the lower branch of this architecture performs basic operations on points to perform feature transformation on point cloud information through a multilayer perceptron (MLP).

[0066] While PVCNN can achieve a balance between point-based and stereo pixel-based operations, the stereo pixel size setting in the stereo pixel-based operation part is inflexible and has redundant space usage. Therefore, this invention proposes an adaptive improvement mechanism.

[0067] Specifically, the processor 12 inputs a three-dimensional point cloud map (step S510) and normalizes the three-dimensional point cloud map (step S520) to adjust the information of different point cloud scales to the same reference benchmark input.

[0068] Before performing voxelization, the processor 12 may first determine the voxel size (step S530). The voxel size refers to the specific size of one or more pixels within a voxel region as a voxel. Unlike a fixed voxel size, the voxel size in this embodiment of the invention can be changed based on the analysis and judgment results of the current situation.

[0069] In one embodiment, for the analysis size (step S531), Figure 6 This is a flowchart illustrating the determination of 3D pixel dimensions based on a size according to an embodiment of the present invention. Please refer to... Figure 6 The processor 12 can determine the size ratio of one or more target objects in the image acquired by the image acquisition device (step S610). The size ratio is the ratio of the length, width, and height of the bounding box of the target object or object type. For example, the width, height, and length ratio of a person is approximately 1:2:1, while the ratio of a car is approximately 1:1:2. Therefore, the bounding box is usually set to 64 pixels * 128 pixels. The processor 12 can determine the voxel size based on the size ratio (step S620). For example, the processor 12 can look up a lookup table or use a formula to find the correspondence between the size ratio and the voxel size.

[0070] A suitable stereo pixel size can save computational resources without affecting localization performance. To train a suitable stereo size ratio, training data is required. The training data consists of labeled point cloud data. "Labeled" means that the target objects in the point cloud are marked with bounding boxes, regions of interest, or 3D object boxes. Figure 7 This is a flowchart illustrating the training of stereo pixel dimensions according to an embodiment of the present invention. Please refer to... Figure 7 For a specific object type, the processor 12 can normalize the length, width, and height of the target object of that object type in the training data based on the largest of the size proportions (step S611). For example, the processor 12 can select the largest value of the length, width, and height in its own labeled 3D bounding box, and then divide the length, width, and height by this largest value to complete the normalization.

[0071] Processor 12 can determine the normalized length, width, and height statistics of the target object of this object type (step S612). The statistics are, for example, the mean, median, or mode. Assuming there are three people, processor 12 can calculate the normalized average length, width, and height of these three people. These normalized length, width, and height statistics serve as a reference for the stereo pixel size. For example, the proportions formed by the statistics are directly used as the stereo pixel size. Alternatively, weights are assigned to the proportions formed by the statistics, and these weights are used to determine the stereo pixel size. Since the statistics are specific to different object types, the corresponding stereo pixel size may also differ depending on the object type. In some embodiments, memory 11 may also store statistics and / or stereo pixel sizes provided by other devices for use by processor 12.

[0072] Figure 8A and Figure 8B These are point cloud images under two stereoscopic pixel sizes according to an embodiment of the present invention. Please refer to... Figure 8A The aspect ratio of width, height, and length is 1:1:1, and the 3D pixel size is 2×2×2. Please refer to... Figure 8B The aspect ratio of its width, height, and length is 1:4:1, and the 3D pixel size is 2×8×2. Although Figure 8B The 3D pixel density is lower, but the overall features (e.g., contours, colors, etc.) are more consistent with... Figure 8A The differences are not significant. However, Figure 8B The lower information content speeds up the 3D pixelation process and reduces the space required for recording information.

[0073] In one embodiment, regarding the analysis depth (step S532), Figure 9 This is a flowchart illustrating the determination of depth-based 3D pixel size according to an embodiment of the present invention. Please refer to... Figure 9 The processor 12 can determine the depth of one or more target objects relative to the scanning position in the image acquired by the image acquisition device (step S910). That is, the image is the image scanned by the image acquisition device at this scanning position. Depth is the distance between the scanning position and the target object. In one embodiment, sensing points or pixels in a 3D point cloud map can record depth (values). In another embodiment, sensing points or pixels in a depth map can record depth.

[0074] In one embodiment, the processor 12 can cluster the depth (values) using a clustering algorithm (e.g., k-means, Gaussian Mixture Model, or DBSCAN algorithm, which uses density-based spatial clustering of applications with noise) and determine specific point cloud regions as candidate target objects based on the clustering results. For example, Figure 10A and Figure 10B This is a schematic diagram of depth clustering according to an embodiment of the present invention. Please refer to... Figure 10A and Figure 10B , Figure 10A This refers to two-dimensional images (e.g., recording red, green, and blue information). Figure 10B This is a depth map image. Pixels at the same depth in the depth map use the same pixel value or grayscale value. Figure 10B It can be seen that if pixels belong to the same object, their depths are roughly the same or within a certain range. Using the clustering method, the two target objects circled in dashed lines can be identified.

[0075] Please refer to Figure 9 The processor 12 can determine the stereo pixel size based on the depth (step S920). In one embodiment, the greater the depth, the smaller the corresponding stereo pixel size, and the smaller the depth, the larger the corresponding stereo pixel size. That is, the stereo pixel size corresponding to a closer target object is larger than the stereo pixel size corresponding to a more distant target object, thereby reducing the amount of information. For example, the stereo pixel size used for a more distant target object is 1×1×1, while the stereo pixel size used for a closer target object is 4×4×4. Although the stereo pixel size is larger, the output of stereo pixelation still retains enough information to be recognized.

[0076] The processor 12 can assign corresponding stereo pixel sizes to different depth ranges. For example, the depth ranges from 0.5 meters (m) to 4 meters, with 0.5 meters as the unit of calculation. If the depth of the grouped target objects falls between 0.5 meters and 1 meter, then their stereo pixel size is 2×2×2. If the depth of the grouped target objects falls between 1 meter and 1.5 meters, then their stereo pixel size is 4×4×4, and so on.

[0077] Taking practical field applications as an example, Figures 11A to 11C These are point cloud images with different stereoscopic pixel sizes according to an embodiment of the present invention. Please refer to... Figures 11A to 11C These are point cloud images from different perspectives, and the 3D pixel size is changed according to the corresponding depth of the target objects O1 and O2. Figure 11ANote that closer target objects O1 have a larger information content, but do not need to be described in detail. To ensure that target objects O1 and O2 within the field of view have the same resolution, a more detailed or smaller stereo pixel size can be provided for the more distant target object O2, while a coarser or larger stereo pixel size can be provided for the closer target object.

[0078] Please refer to Figure 5 The analysis of size and depth can be performed simultaneously or selectively. After determining the stereo pixel size (step S530), the processor 12 can perform stereo pixelation on the input 3D point cloud map according to the determined stereo pixel size (step S540) to obtain stereo pixels. After obtaining the stereo pixels, the processor 12 can complete the feature integration through three-dimensional volumetric convolution operation (step S550), and convert the stereo pixels back into a point cloud map through de-stereo pixelation (step S560) to fuse with the feature-converted point cloud map (step S570).

[0079] Please refer to Figure 4 The processor 12 can transform the position of the target object from its position in three-dimensional space to its position in a top view (step S420). That is, top-view projection. The processor 12 can calculate the representative position of the target object in the top view. For example, the representative position is the center point, the centroid point, or any point on the target object. By continuously recording the representative positions of the target object, these representative positions can be used as those positions in the historical trajectory (step S430).

[0080] Please refer to Figure 2 and Figure 3A The processor 12 uses the pooling module 33 to process the first trajectory information I1 of multiple target objects to generate the second trajectory information I2 of the target objects (step S220). Specifically, the pooling process is used to obtain the relationship between the first trajectory information I1 of multiple target objects. For example, the processor 12 concatenates the current position of a target object with the current positions of other target objects after word embedding, and then inputs this into the multilayer perceptron and takes the maximum value. Finally, the positional relationship between each target object and other target objects can be obtained. For example, the position of the first target object relative to the second target object, the position relative to the third target object, and so on. The second trajectory information I2 of each target object includes the positional relationship relative to other target objects.

[0081] Please refer to Figure 2 and Figure 3AThe processor 12 obtains third trajectory information I3 from the historical trajectory of the target object (step S230). Specifically, in addition to considering the social relationships between target objects, the present invention also provides other auxiliary trajectory judgment models. The third trajectory information I3 may include movement direction, scene information and / or movement mode.

[0082] In one embodiment, the generator network 30 includes a direction module 34. The historical trajectory includes multiple directions. The processor 12 can determine the direction of movement from one location to another. For example, the processor 12 can calculate the relative position difference between the positions of the trajectory points sampled at different time points and derive the angle based on the relative position difference using trigonometric functions.

[0083] Taking practical applications as an example, Figure 12A This is a schematic diagram of a trajectory according to an embodiment of the present invention. Please refer to... Figure 12A The positions of the preceding and following trajectory points can be connected by a line. The direction of movement from the first trajectory point to the second trajectory point is 168 degrees, the direction of movement from the second trajectory point to the third trajectory point is 167 degrees, and the direction of movement from the third trajectory point to the fourth trajectory point is 175 degrees.

[0084] Processor 12 can arrange the movement directions corresponding to those positions into movement trends according to time sequence. A movement trend is the change of multiple movement directions arranged according to time sequence. The third trajectory information includes this movement trend.

[0085] In one embodiment, processor 12 can encode movement trends into binary vectors. For example, one-hot encoding can be used to form a sequence of 0s or 1s. Multiple binary vectors representing movement directions are concatenated to form a movement trend feature. These binary vectors are then input to decoder 132.

[0086] For example, Figure 12B This is a schematic diagram of angle grouping according to an embodiment of the present invention. Please refer to... Figure 12B The processor 12 can first define the angle difference d in the encoding rules (taking d = 30 as an example), and divide 360 ​​degrees equally using the angle difference d. The processor 12 can generate a one-hot code C based on the position of the movement direction. Taking a movement direction of 275 degrees as an example, the encoded sequence is 0000000000100, which is a 360 / d + 1 bit sequence. This +1 part is used to record the non-movement state. Furthermore, the smaller the angle difference d, the more detailed the obtained features will be. However, considering space cost or other factors, the angle difference d can still be adjusted according to requirements.

[0087] For example, please refer to Figure 12A and Figure 12BThe four trajectory point vectors on the right move in directions of 168 degrees, 167 degrees, and 175 degrees. Since these three directions of movement all fall within the same interval, their binary vectors are all 0000001000000. If the object moves while stationary, its binary vector is 1000000000000, and so on.

[0088] It should be noted that the specific time difference between the aforementioned time points is related to the sampling frequency during data acquisition. Taking 30 FPS (frames per second) as an example, the time difference between the preceding and following trajectory points obtained at this sampling frequency is approximately 1 / 30 ≈ 0.033 seconds. However, if the sampling frequency is 5 FPS, the time difference is 1 / 5 = 0.2 seconds. Therefore, the time difference between the preceding and following time points can still be changed according to actual needs.

[0089] Please refer to Figure 3A In one embodiment, the generator network 30 includes a scene module 35. The processor 12 can acquire image information. This image information includes one or more scene objects from two-dimensional or three-dimensional images acquired by an image acquisition device. Each image is acquired from a viewing position and within a field of view in a historical trajectory, and those scene objects are objects viewed from that viewing position and within the field of view. This module primarily generates third trajectory information based on the scene objects surrounding the location of the target object. The type of scene object may vary depending on the application. For example, in a mobile robot application, scene objects might be walls, beams, and tables and chairs, but pedestrians are not scene objects. The processor 12 can predefine the type of scene object. The third trajectory information may be specific to the defined scene objects and may exclude other bit-defined objects.

[0090] The processor 12 can generate scene information based on scene objects. The scene information includes the relative positions and / or image areas of scene objects obtained from images acquired by the image acquisition device, where the relative positions are relative to the viewing position of the image acquisition device. Specifically, Figure 13 This is a flowchart illustrating scene information generation according to an embodiment of the present invention. Please refer to... Figure 13 Taking a 3D point cloud image PC as an example, the processor 12 can identify the location (selected by regions of interest ROI1 and ROI2) and type of scene objects in the 3D point cloud image PC through object detection algorithms (e.g., feature matching or neural network inference) (step S131). The processor 12 can switch the image's viewpoint to a top view (step S132) to project the regions of interest ROI1 and ROI2 onto the top view. For example, Figure 14 This is a schematic diagram of the projection result according to an embodiment of the present invention. Please refer to... Figure 14This image shows a cost map for a mobile robot application. From the viewing position VP, the mobile robot sees the scene objects framed by the regions of interest (ROIs) 1 and 2 within its field of view (FOV, using a triangular area as an example). The FOV serves as a reference range for scene information. It's worth noting that, in addition to scene objects identified through image detection, the processor 12 can also identify other scene objects such as walls or beams within the FOV from the top-down view (in this embodiment, the cost map). It should be noted that the reference range is not limited to the FOV and may also be based on the geometric or irregularly shaped area formed by the viewing position. Furthermore, the top-down view may also be based on a plan view, object configuration diagram, or other diagrams designed or planned from a top-down perspective.

[0091] Next, taking object type as the unit, the processor 12 can count the area ratio of the region of interest contained in each category in the screen and / or its relative position relative to the target object (the observation position), and concatenate the area ratio and the statistical relative position or select one of them to obtain scene information (step S133).

[0092] In one embodiment, processor 12 can determine the area proportion of the image area of ​​all scene objects of a certain object type within a reference range in the top view. The reference range is an area selected from the top view (e.g., Figure 14 The field of view (FOV), but not limited to this, and scene information includes area proportions. For example, the area proportion is the sum of the image areas of the regions of interest of three tables within the field of view, divided by the area of ​​the field of view. Similarly, if there are other object types, the processor 12 determines the area proportion of that object type. In some embodiments, the area can be determined using screen resolution statistics.

[0093] In one embodiment, processor 12 may determine the statistical relative positions of all scene objects of a given object type. Scene information includes these statistical relative positions. The relative position may be the distance or number of pixels between the target object and scene objects on the horizontal and vertical axes. For example, the area ratio is the sum of the relative positions of two chairs within the field of view divided by the total number of chairs (i.e., two). That is, the average of the relative positions. Similarly, if there are other object types, processor 12 determines the statistical relative positions of those object types. In some embodiments, the statistical relative position may be a weighted average of relative positions, the median, or the mode of the relative positions.

[0094] In one embodiment, if the area ratio R and the statistical relative position (Δx, Δy) are to be concatenated, the concatenation result for a certain object type can be "R, Δx, Δy" or "Δx, Δy, R".

[0095] Please refer to Figure 3AIn one embodiment, the generator network 30 includes a distribution calculator 36. The processor 12 can determine the corresponding noise distribution based on the movement pattern. Specifically, existing social adversarial generative networks (GANs) provide the same pre-defined Gaussian noise distribution to all input data for adversarial generation. However, the distribution calculator 36 can provide different noise to the prediction model for different movement patterns to train the generator network 30. The movement pattern can be straight walking, turning left, turning right, or standing still, but is not limited to these. For example, Figure 15 This is a schematic diagram of the trajectories of multiple movement modes according to an embodiment of the present invention. Please refer to... Figure 15 Different trajectories (formed by connecting the points in the diagram) may represent different movement patterns. For example, the top, left, and right trajectories represent straight movement, while the bottom trajectory represents a left turn.

[0096] In one embodiment, the movement pattern information can be based on the relative distance between sampling points in a historical trajectory. Since the movement pattern of the target object may change at any time, the processor 12 can set a relationship filtering value (which can be determined based on experience and the sampling frequency of the data) and filter the trajectory point positions accordingly to distinguish between different movement patterns. For example, five sampling points could represent five trajectory point positions in a historical trajectory, such as the positions of five steps taken by a pedestrian. On the other hand, regarding data filtering, if the distance between trajectory point positions at different times is greater than the relationship filtering value, the information of these two trajectory point positions will not be used by the distribution calculator 36. If the distance between trajectory point positions at different times is not greater than the relationship filtering value, the information of these two trajectory point positions can be used by the distribution calculator 36.

[0097] For example, Figure 16 This is a schematic diagram of trajectory filtering according to an embodiment of the present invention. Please refer to... Figure 16 Assuming the relationship filtering value is 20 cm, the distance between sampling point P1 and sampling point P2 differs by more than 20 cm, therefore processor 12 can ignore / delete / exclude the trajectory from sampling point P1 to sampling point P2. However, the distances between sampling points P2 and P3, P3 and P4, and P4 and P5 differ by less than 20 cm, therefore these trajectories can be used by the distribution calculator 36.

[0098] The relative positions of the sampling points in these historical trajectories can be used to establish a symmetric matrix. That is, the elements of the symmetric matrix are the relative positions of each sampling point with other sampling points. After the processor 12 groups this symmetric matrix, it can obtain the grouping results. These groups are the defined different movement patterns. For these different movement patterns, the processor 12 can determine the noise corresponding to the randomness referenced by the decoder 32 based on the noise distribution corresponding to the movement pattern. By providing different noise, the prediction model can achieve better prediction results for different movement patterns.

[0099] In one embodiment, processor 12 can obtain a noise distribution by applying a Gaussian distribution to the center (as average) and standard deviation of a cluster after the historical trajectory has been grouped. However, the type of noise distribution is not limited to a Gaussian distribution. Processor 12 can determine randomly selected noise based on this Gaussian distribution.

[0100] For example, Figure 17A This is a schematic diagram of the probability distribution of noise corresponding to a left-turn mode according to an embodiment of the present invention. Figure 17B This is a schematic diagram of the probability distribution of noise corresponding to a non-moving mode (or stationary mode) according to an embodiment of the present invention, and Figure 17C This is a schematic diagram showing the probability distribution of noise corresponding to a straight-line mode according to an embodiment of the present invention. Please refer to... Figures 17A-17C Let's assume that the observed trajectories Obs1 to Obs4 represent the trajectories of a pedestrian taking one, two, three, and four steps, respectively. Therefore, it can be concluded that the Gaussian distribution curves for different movement patterns may differ.

[0101] Please refer to Figure 2 and Figure 3A The processor 12 uses the second trajectory information I2 and the third trajectory information I3 of the target object to generate the predicted trajectory of the target object through the decoder 32 (step S240). Specifically, the decoder 32 is an LSTM model and generates the predicted trajectory of each target object based on the second trajectory information I2, the third trajectory information I3, and noise. For example, the decoder 32 can convert the hidden state into the coordinates of a position in the future trajectory. In other embodiments, the decoder 32 may use other neural networks, such as RNNs, C-RNNs, or other networks suitable for processing temporal features.

[0102] In one embodiment, processor 12 can integrate second trajectory information I2 and third trajectory information I3 of multiple target objects through multi-head attention to generate integrated information. For example, processor 12 can perform self-attention processing on the second trajectory information I2 and the third trajectory information I3 respectively, and then concatenate the results of each set of self-attention and perform a linear transformation to obtain the integrated information. Self-attention is, for example, scaled dot-product attention. The attention mechanism can be described as the process of mapping a query and a series of key-value pairs to an output vector, where the output vector is the weighted sum of the weights calculated based on the query and the key, applied to the values. Furthermore, this integrated information is used as input to decoder 32.

[0103] In summary, the trajectory prediction method and computational system for trajectory prediction in this embodiment of the invention provide social relationships and additional auxiliary information (e.g., movement direction, scene information, and / or movement pattern) to the generator network in the prediction model, so as to...

[0104] This improves the prediction accuracy of the prediction model. Furthermore, the prediction model can reference 3D information and process 3D point cloud maps using appropriate stereo pixel sizes to improve processing efficiency and memory space utilization.

[0105] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Anyone skilled in the art can make some modifications and refinements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A trajectory prediction method based on a prediction model, wherein the prediction model includes a generator network, and the trajectory prediction method includes: The historical trajectories of multiple target objects are respectively processed by an encoder to obtain features to generate the first trajectory information of the target objects. The encoder is used to obtain a hidden state representing the movement characteristics reflected by the historical trajectory, and the first trajectory information includes the hidden state. The first trajectory information of the target object is processed by pooling to generate the second trajectory information of the target object, wherein the pooling process is used to obtain the relationship between the first trajectory information of the target objects, and the second trajectory information of each target object includes the positional relationship relative to other target objects; Obtain third trajectory information from the historical trajectory of the target object, wherein the third trajectory information includes at least one of movement direction, scene information and movement mode; and generate a predicted trajectory of the target object by decoding the second trajectory information and the third trajectory information of the target object.

2. The trajectory prediction method as described in claim 1, wherein the historical trajectory includes multiple locations, and the step of obtaining the third trajectory information from the historical trajectory of the target object includes: Determine the direction of movement from one of the said positions to the other; as well as The movement directions corresponding to the positions are arranged into movement trends according to time sequence, wherein the third trajectory information includes the movement trends.

3. The trajectory prediction method as described in claim 2, further comprising: The movement trend is encoded into a binary vector, which is then input into the decoder.

4. The trajectory prediction method as described in claim 1, wherein the step of obtaining the third trajectory information from the historical trajectory of the target object includes: Acquire image information, wherein the image information includes at least one scene object in a first image, the first image being acquired from a viewing position in the historical trajectory and viewed from the field of view, and the at least one scene object being an object viewed from the viewing position and viewed from the field of view. as well as The scene information is generated based on the at least one scene object, wherein the scene information includes at least one of the relative position of the at least one scene object and the image area known from the first image, and the relative position is the position relative to the viewing position.

5. The trajectory prediction method as described in claim 4, wherein the at least one scene object is a first object type, and the step of generating the scene information based on the at least one scene object includes: Determine the statistical relative positions of the relative positions of all scene objects of the first object type, wherein the scene information includes the statistical relative positions; And / or The viewpoint of the first image is switched to a top view, and the area ratio of the image area of ​​all scene objects of the first object type in the top view is determined within a reference range, wherein the reference range is a region selected from the top view, and the scene information includes the area ratio.

6. The trajectory prediction method as described in claim 1, wherein the step of obtaining the third trajectory information from the historical trajectory of the target object includes: The corresponding noise distribution is determined based on the movement pattern; as well as The noise corresponding to randomness is determined based on the noise distribution.

7. The trajectory prediction method as described in claim 1, wherein the step of generating the predicted trajectory of the target object by using the decoder from the second trajectory information and the third trajectory information of the target object includes: The second trajectory information and the third trajectory information of the target object are integrated by multi-head attention to generate integrated information, wherein the integrated information is used as input to the decoder, and the output of the generator network is used by the discriminator network to identify authenticity.

8. The trajectory prediction method as described in claim 1, further comprising: Determine the size ratio of at least one target object in the second image, wherein the size ratio is the ratio of the length, width and height of the at least one target object; The size of the 3D pixels is determined based on the aforementioned size ratio; as well as The three-dimensional point cloud map is stereo-pixelated according to the stereo pixel size, wherein the three-dimensional point cloud map includes the sensing points of the at least one target object. The stereo-pixelated three-dimensional point cloud map is used to determine the position of the at least one target object in three-dimensional space. The position in the three-dimensional space is converted to the position on the top view as one of the multiple positions in the historical trajectory.

9. The trajectory prediction method as described in claim 8, wherein the plurality of target objects are of a second object type, and the step of determining the 3D pixel size according to the size ratio: The length, width, and height of at least one target object of the second object type in the training data are normalized based on the largest of the said size ratios; and Determine the normalized length, width, and height statistics of the target object of the second object type, wherein the normalized length, width, and height statistics serve as the reference for the stereo pixel size.

10. The trajectory prediction method as described in claim 1, further comprising: Determine the depth of at least one target object in a third image relative to a scan position, wherein the third image is an image scanned at the scan position; The size of the stereo pixels is determined based on the depth, wherein the greater the depth, the smaller the size of the stereo pixels, and the smaller the depth, the larger the size of the stereo pixels. as well as The three-dimensional point cloud map is stereo-pixelated based on the stereo pixel size. The stereo-pixelated three-dimensional point cloud map is used to determine the position of the at least one target object in three-dimensional space. The position in the three-dimensional space is converted to the position in the top view as one of the multiple positions in the historical trajectory.

11. A computational system for trajectory prediction, comprising: Memory, which stores program code; as well as A processor, coupled to the memory, is configured to load and execute the program code to: Execute a prediction model, wherein the prediction model includes a generator network; The historical trajectories of multiple target objects are respectively processed by an encoder to obtain features to generate the first trajectory information of the target objects, wherein the encoder is used to obtain a hidden state representing the movement characteristics reflected by the historical trajectory, and the first trajectory information includes the hidden state. The first trajectory information of the target object is processed by a pooling process to generate the second trajectory information of the target object, wherein the pooling process is used to obtain the relationship between the first trajectory information of the target objects, and the second trajectory information of each target object includes the positional relationship relative to other target objects; Third trajectory information is obtained from the historical trajectory of the target object, wherein the third trajectory information includes at least one of movement direction, scene information, and movement mode; and The second trajectory information and the third trajectory information of the target object are used to generate the predicted trajectory of the target object through a decoder.

12. The computational system for trajectory prediction as claimed in claim 11, wherein the historical trajectory includes multiple locations, and the processor is further configured to: Determine the direction of movement from one of the positions to the other; and The movement directions corresponding to the positions are arranged into movement trends according to time sequence, wherein the third trajectory information includes the movement trends.

13. The computational system for trajectory prediction as described in claim 12, further comprising: The movement trend is encoded into a binary vector, which is then input into the decoder.

14. The computational system for trajectory prediction as claimed in claim 11, wherein the processor is further configured to: Acquire image information, wherein the image information includes at least one scene object in a first image, the first image being acquired from a viewing position along the historical trajectory and viewed within the field of view, and the at least one scene object being an object viewed from the viewing position and within the field of view; and The scene information is generated based on the at least one scene object, wherein the scene information includes at least one of the relative position of the at least one scene object and the image area known from the first image, and the relative position is the position relative to the viewing position.

15. The computational system for trajectory prediction as described in claim 14, wherein the at least one scene object is a first object type, and the processor is further configured to: Determine the statistical relative positions of the relative positions of all scene objects of the first object type, wherein the scene information includes the statistical relative positions; and / or The viewpoint of the first image is switched to a top view, and the area ratio of the image area of ​​all scene objects of the first object type in the top view is determined within a reference range, wherein the reference range is a region selected from the top view, and the scene information includes the area ratio.

16. The computational system for trajectory prediction as claimed in claim 11, wherein the processor is further configured to: Determine the corresponding noise distribution based on the movement pattern; and The noise corresponding to randomness is determined based on the noise distribution.

17. The computational system for trajectory prediction as claimed in claim 11, wherein the processor is further configured to: The second trajectory information and the third trajectory information of the target object are integrated by multi-head attention to generate integrated information, wherein the integrated information is used as input to the decoder, and the output of the generator network is used by the discriminator network to identify authenticity.

18. The computational system for trajectory prediction as claimed in claim 11, wherein the processor is further configured to: Determine the size ratio of at least one target object in the second image, wherein the size ratio is the ratio of the length, width and height of the at least one target object; The size of the 3D pixels is determined based on the aforementioned size ratio; as well as The three-dimensional point cloud map is stereo-pixelated according to the stereo pixel size, wherein the three-dimensional point cloud map includes the sensing points of the at least one target object. The stereo-pixelated three-dimensional point cloud map is used to determine the position of the at least one target object in three-dimensional space. The position in the three-dimensional space is converted to the position on the top view as one of the multiple positions in the historical trajectory.

19. The computational system for trajectory prediction as claimed in claim 18, wherein the plurality of target objects are of a second object type, and the processor is further configured to: The length, width, and height of at least one target object of the second object type in the training data are normalized based on the largest of the said size ratios; and Determine the normalized length, width, and height statistics of the target object of the second object type, wherein the normalized length, width, and height statistics serve as the reference for the stereo pixel size.

20. The computational system for trajectory prediction as claimed in claim 11, wherein the processor is further configured to: Determine the depth of at least one target object in a third image relative to a scan position, wherein the third image is an image scanned at the scan position; The 3D pixel size is determined based on the depth, wherein a larger depth corresponds to a smaller 3D pixel size, and a smaller depth corresponds to a larger 3D pixel size; and The three-dimensional point cloud map is stereo-pixelated according to the stereo pixel size, wherein the three-dimensional point cloud map includes the sensing points of the at least one target object. The stereo-pixelated three-dimensional point cloud map is used to determine the position of the at least one target object in three-dimensional space. The position in the three-dimensional space is converted to the position on the top view as one of the multiple positions in the historical trajectory.

Images