A trajectory prediction method and system based on multi-modal perception

Abstract

The application relates to the fields of intelligent traffic and edge computing technology, and discloses a trajectory prediction method and system based on multi-modal perception, wherein the trajectory prediction method based on multi-modal perception comprises the following steps: collecting multi-modal perception data, generating an initial calculation graph structure; analyzing scene complexity, dynamically reconstructing the calculation graph structure in combination with the scene complexity and the device resource state; performing task perception pruning, generating an optimized calculation graph; evaluating the calculation task priority, determining the execution positions of each node on the edge device and the cloud; performing scene-adaptive model decomposition, decomposing the trajectory prediction model into edge-side and cloud-side components; and generating the trajectory prediction result of the target object; through the dynamic reconfigurable calculation graph structure and the task-perception calculation graph pruning, the application dynamically adjusts the calculation resource allocation according to the scene complexity and the task characteristics, removes the calculation nodes with smaller contribution to the current task, and reduces the calculation complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation and edge computing, more particularly, to a trajectory prediction method and system based on multi-modal perception. BACKGROUND

[0002] With the rapid development of smart cities, a large number of edge devices such as intelligent cameras and roadside units are widely deployed in urban transportation networks. These devices need to collaboratively process heterogeneous data from multiple perception channels and perform real-time trajectory prediction tasks based on these multi-modal data to support traffic management, safety warning and autonomous driving assistance and other applications.

[0003] However, edge devices usually face the challenge of limited computing resources. Existing trajectory prediction methods often have high computational complexity, and their complex multi-modal fusion operations and spatio-temporal interaction modeling processes are difficult to efficiently run on edge devices with limited computing resources. Although simple compression models can be used to reduce computational complexity, this often leads to a decline in the multi-modal fusion ability and spatio-temporal interaction modeling ability of the model, which cannot meet the accuracy requirements of trajectory prediction in complex scenarios.

[0004] Therefore, there is an urgent need for a technology that can reduce the computational complexity of multi-modal trajectory prediction under the condition of limited computing resources of edge devices while maintaining prediction accuracy. In addition, it is also necessary to dynamically adjust the allocation of computing resources according to the complexity of different scenarios and task requirements, to balance the computing efficiency and prediction accuracy. In the case where edge devices cannot independently complete complex computing tasks, how to effectively use cloud resources for collaborative computing, and how to adaptively decompose the model structure according to the characteristics of the scene, so that it is more suitable for execution on edge devices with limited resources, are all problems that need to be solved in the current technical field. SUMMARY

[0005] The present application provides a trajectory prediction method and system based on multi-modal perception, which solves the technical problem of low computing efficiency and difficult balance between prediction accuracy when performing complex multi-modal trajectory prediction tasks on resource-limited edge devices in related technologies.

[0006] The present application provides a trajectory prediction method based on multi-modal perception, comprising:

[0007] Collecting multi-modal perception data to generate an initial computation graph structure;

[0008] Analyzing the complexity of the scene based on the initial computation graph structure, and dynamically reconstructing the computation graph structure in combination with the complexity of the scene and the state of the device resources;

[0009] Performing task-aware pruning on the reconstructed computation graph structure, removing computation nodes with a contribution degree lower than a preset threshold, and generating an optimized computation graph;

[0010] The computing task priority is evaluated for optimizing the nodes in the computing graph, and the execution positions of the nodes in the edge device and the cloud are determined;

[0011] According to the execution position, the execution scene adaptive model decomposition is divided, and the trajectory prediction model is decomposed into edge side and cloud side components;

[0012] The respective components are executed by the edge device and the cloud in cooperation, the computing results of both sides are integrated, and the trajectory prediction result of the target object is generated.

[0013] Further, the step of analyzing the scene complexity comprises: calculating the number of target objects in the scene, the motion mode diversity index and the occlusion degree index; normalizing the number of target objects in the scene, the motion mode diversity index and the occlusion degree index, and mapping them to a unified numerical range; obtaining the scene complexity score through weighted calculation; determining the calculation complexity level based on the scene complexity score.

[0014] Further, the step of dynamically reconstructing the computing graph structure comprises: obtaining the current device resource state, including the calculation resource availability, the memory capacity and the power level; converting the device resource state into a resource constraint coefficient; determining the calculation graph complexity target value; selecting the calculation graph template closest to the target value from the pre-defined calculation graph template library; refining the selected calculation graph template according to the specific characteristics of the current scene to generate the final calculation graph structure.

[0015] Further, the step of evaluating the computing task priority comprises: feature extraction for each computing task in the optimized computing graph; calculation of the time sensitivity index of the task; estimation of the resource demand of the task on the edge device; calculation of the data transmission cost of the task; standardization processing of the task features, the time sensitivity index, the resource demand and the data transmission cost; obtaining the task priority score through weighted calculation; based on the priority score, classifying the tasks into tasks executed locally on the edge device and tasks migrated to the cloud for execution.

[0016] Further, the step of executing task-aware pruning comprises: task feature analysis of the current trajectory prediction task to determine the task type, precision requirement and time limit; performing sensitivity analysis on each node in the reconstructed computing graph to calculate the node influence degree; measuring the calculation cost of each computing node; calculating the task relevance adjustment factor of each node; calculating the contribution score of each node; determining the contribution threshold; removing the nodes with a contribution score lower than the threshold; reconstructing the connection relationship of the retained node set to ensure the continuity of the data flow.

[0017] Further, the step of performing scene-adaptive model decomposition comprises: analyzing the data dependency relationship between nodes in the computation graph, constructing a dependency graph; calculating the time sensitivity index of each node; estimating the resource demand of each node for edge device and cloud execution; estimating the communication overhead of data transmission between nodes; constructing a model decomposition optimization objective function; normalizing different dimension cost indicators; solving the optimization problem to obtain the optimal node mapping scheme; and dividing the computation graph nodes into edge-side node set and cloud-side node set according to the mapping scheme.

[0018] Further, the collaborative execution comprises: edge device executing the computing task of edge-side component; transmitting necessary intermediate results to the cloud; cloud executing the computing task of cloud-side component; returning the cloud computing result to the edge device; standardizing the computing results of the edge device and the cloud; fusing the real-time features of the edge device and the deep features of the cloud to generate enhanced feature representation; and calculating the future trajectory based on the enhanced features.

[0019] Further, the multi-modal perception data comprises: visual data including images and video sequences; point cloud data for three-dimensional space information acquisition; radar data for target object distance and speed measurement; wherein the multi-modal perception data is preprocessed after acquisition, including data standardization and feature extraction.

[0020] Further, the step of generating the trajectory prediction result of the target object comprises: calculating the position coordinates of the key time points; using a trajectory interpolation algorithm to generate a complete continuous trajectory; calculating the uncertainty of the trajectory; generating trajectory hypotheses and their confidence; and post-processing the generated trajectory, including smoothing, physical constraint checking and scene consistency verification.

[0021] The present application provides a multi-modal perception-based trajectory prediction system for executing the above-mentioned multi-modal perception-based trajectory prediction method, comprising:

[0022] The data perception and graph structure generation module is used for acquiring multi-source heterogeneous perception data and constructing an initial computation graph.

[0023] The scene perception and graph structure reconstruction module is used for analyzing the environmental complexity and dynamically adjusting the computation graph based on the device resource state.

[0024] The task perception pruning module is used for removing low-contribution computing nodes and optimizing the computing process.

[0025] The computing task scheduling module is used for evaluating the task priority and determining the best execution location.

[0026] The model adaptive decomposition module is used for dividing the prediction model into edge-side and cloud-side components according to the scene characteristics.

[0027] The collaborative computing and results generation module is used to integrate edge cloud computing results and output target trajectory predictions.

[0028] The beneficial effects of this invention are as follows: By using a dynamically reconfigurable computation graph structure and task-aware computation graph pruning, the allocation of computing resources is dynamically adjusted according to scene complexity and task characteristics, and computation nodes that contribute little to the current task are removed, thereby reducing computational complexity. Because the computational operations are more streamlined and targeted, the computational burden on edge devices is effectively reduced.

[0029] The trajectory prediction method based on multimodal perception can dynamically adjust the calculation strategy according to the resource conditions such as the computing power, memory capacity, and power status of edge devices, so that the same trajectory prediction model can run efficiently on devices with different resource configurations, thereby enhancing the adaptability and scalability of the system.

[0030] By selectively retaining computational nodes that contribute significantly to the prediction accuracy of the current task and discarding nodes that contribute less, the trajectory prediction method based on multimodal perception achieves a good balance between computational efficiency and prediction accuracy, avoiding the accuracy degradation problem caused by simple compression models.

[0031] When the computing power of edge devices is insufficient to independently complete complex computing tasks, the trajectory prediction method based on multimodal perception migrates low-priority computationally intensive tasks to the cloud for execution through the edge cloud collaborative reasoning framework, thereby optimizing the allocation of computing resources and improving the overall performance of the system.

[0032] Through a scenario-adaptive model decomposition strategy, the model structure and task allocation methods can be automatically adjusted according to the characteristics of different scenarios and computing platforms, enabling the system to maintain efficient operation in complex and ever-changing environments.

[0033] Unlike simple compression models, trajectory prediction methods based on multimodal perception reduce computational complexity while retaining key multimodal fusion and spatiotemporal interaction modeling capabilities, ensuring trajectory prediction accuracy in complex scenarios. Attached Figure Description

[0034] Figure 1 This is a flowchart of a trajectory prediction method based on multimodal perception in this invention;

[0035] Figure 2 It is a scatter plot of the relationship between computing resource states under different scenario complexities;

[0036] Figure 3 It is a bar chart comparing the changes in the number of nodes before and after computation graph pruning for pedestrian trajectory prediction, vehicle trajectory prediction, and hybrid trajectory prediction.

[0037] Figure 4is a two-axis chart contrasting the task allocation of different components in the edge cloud collaborative execution architecture and the processing delay thereof;

[0038] Figure 5 is a bar chart contrasting the computational efficiency of the method with the full model and the simplified model under different scenario complexities;

[0039] Figure 6 is an area chart of the balance between resource utilization and prediction accuracy of the method under different scenario complexities. DETAILED DESCRIPTION

[0040] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that the discussion of these implementations is merely meant to provide a better understanding of the subject matter described herein and can include changes, modifications, additions or omissions of the functions and arrangements of the elements discussed without departing from the scope of the present specification. Various examples can omit, substitute, or add various procedures or components as appropriate, and the examples described herein can be combined together to form additional examples. Additionally, some features described in the examples can be combined in a single example.

[0041] In at least one embodiment of the present disclosure, a multi-modal perception based trajectory prediction method is disclosed, as shown in Figure 1 comprises:

[0042] Step 1, collect multi-modal perception data and generate an initial computational graph structure;

[0043] First, multi-modal perception data is collected by various sensors of the edge device, including visual data, point cloud data, radar data, etc., and an initial computational graph structure is generated based on a preset trajectory prediction model. The initial computational graph structure includes multiple computational nodes and edges connecting these nodes, where each computational node represents a specific data processing operation, and the edge represents the data flow direction.

[0044] The trajectory prediction model is composed of the following components: a multi-modal feature extraction component, including a visual feature extraction unit, a point cloud feature extraction unit, and a radar feature extraction unit, respectively responsible for extracting feature representations from different modal sensor data; a feature fusion component that receives the outputs of the modal feature extraction units and fuses multiple heterogeneous features into a unified feature representation; a time series modeling component for capturing the time dependence of the target historical trajectory, including an attention mechanism based time series relationship modeling unit; a context understanding component for analyzing the target surrounding environment information, including a scene semantic analysis unit and an interaction relationship modeling unit; a trajectory generation component that predicts future trajectories based on fused features and time series and context information, including a decision prediction unit and a trajectory interpolation unit.

[0045] The initial computational graph structure can be represented as a directed graph:

[0046] ;

[0047] wherein denotes a computation graph, denotes a set of nodes, denotes a set of directed edges between nodes.

[0048] A set of nodes is denoted as:

[0049] ;

[0050] wherein denotes a set of nodes, , , denote the 1st, 2nd, 3rd node, respectively, is the total number of nodes.

[0051] A set of directed edges between nodes is denoted as:

[0052] ;

[0053] wherein denotes a set of directed edges between nodes, denotes the data flow from node to node .

[0054] Each node is associated with a computation operation that transforms input data into output data:

[0055] ;

[0056] wherein is the input data of node , is the output data of node , is the computation operation of node .

[0057] For a convolution node, the computation operation is denoted as:

[0058] ;

[0059] wherein is the input data of node , is the 1st a convolution kernel parameter of the i-th computing node, a bias parameter of the i-th computing node, an activation function, represents a convolution operation.

[0060] For the fusion node, the computing operation adopts a weighted fusion manner: the fusion node performs weighted summation on the features from different modalities, where each modality feature is assigned with a corresponding weight coefficient, and the fused feature representation is obtained by accumulating the products of all modality features and their corresponding weight coefficients. Specifically, the fusion node receives feature inputs of multiple modalities, multiplies each modality feature by its weight coefficient, and then adds all the weighted features to form a unified multi-modal fusion feature output.

[0061] Step 2, analyze the scene complexity and dynamically reconstruct the computation graph structure;

[0062] According to the collected multi-modal perception data, the current scene complexity is analyzed, and the computation graph structure is dynamically adjusted based on the scene complexity. The scene complexity can be evaluated by factors such as the number of targets, motion pattern diversity, and occlusion situation. The higher the complexity, the more computing resources are needed.

[0063] wherein the scene complexity evaluation can be calculated by the following formula:

[0064] ;

[0065] wherein, represents the scene complexity score, represents the number of target objects in the scene, represents the motion pattern diversity index, represents the occlusion degree index, , , respectively represent the weight coefficients of the number of target objects, the motion pattern diversity, and the occlusion degree.

[0066] Before calculating the scene complexity, the three different dimension indicators, , and are normalized to map them to the [0, 1] interval, respectively, to ensure the comparability of different indicators in the complexity evaluation calculation.

[0067] In the embodiments of the present application, in order to more accurately adjust the computation graph structure, the current computing resource status of the edge device is also considered in the dynamic reconstruction process, including CPU occupancy, memory usage, power level, etc. Through the resource perception function ​Real-time monitoring of device resource status and taking the status as one of the input factors of reconstruction decision. For different types of resource indicators such as CPU occupancy, memory usage and power level, standardization processing is carried out before inputting the resource perception function, and they are uniformly converted into availability indicators in the range of [0, 1].

[0068] The algorithm of dynamically reconstructing the calculation graph performs the following steps: according to the scene complexity score Determine the required calculation complexity level , the higher the level, the more calculation resources required; obtain the current device resource status , including the availability of computing resources, memory capacity and power level; calculate the resource constraint coefficient:

[0069] ;

[0070] Wherein, is the current device resource status, is the resource constraint coefficient, which increases with the increase of resource tension degree; is the resource state mapping function, defined as:

[0071] ;

[0072] Wherein , and represent the CPU availability, memory availability and power level at the current time , , and are the weight coefficients of CPU availability, memory availability and power level;

[0073] Determine the calculation graph complexity target value:

[0074] ;

[0075] Wherein is the required calculation complexity level, is the resource constraint coefficient, is the calculation graph complexity target value.

[0076] Select the calculation graph template closest to the target value from the predefined calculation graph template library ; according to the specific characteristics of the current scene, the selected calculation graph template is refined and adjusted to generate the final calculation graph structure ; the initial calculation graph structure is converted into the reconstructed calculation graph structure , to achieve dynamic adaptation.

[0077] As Figure 2 shown, the relationship between the computational resource state under different scene complexity is shown, the horizontal axis represents the scene complexity score, the vertical axis represents the device CPU usage, and the size of the point represents the number of reconstructed computational graph nodes. Through the graph, the correlation between scene complexity and computational resource consumption can be intuitively observed, verifying the effectiveness of the dynamically reconfigurable computational graph structure.

[0078] Step 3, task-aware pruning is performed on the reconstructed computational graph structure, and computational nodes with a contribution degree lower than a preset threshold are removed to generate an optimized computational graph;

[0079] Based on the specific trajectory prediction task characteristics currently being executed, the computational graph structure is pruned to remove computational nodes that contribute less to the current task, generating an optimized computational graph structure and reducing computational complexity.

[0080] The computational graph pruning process is implemented through the following steps:

[0081] For each computational node , its contribution to the current task is calculated ;

[0082] According to the contribution degree threshold , determine the node set that needs to be retained:

[0083] ;

[0084] Where is the node set that needs to be retained, is the contribution score of node , and is the contribution degree threshold.

[0085] Reconnect the retained nodes to form an optimized computational graph structure:

[0086] ;

[0087] Where is the optimized computational graph structure, is the node set that needs to be retained, and is the directed edge set between the retained nodes.

[0088] In the embodiments of the present application, in order to improve the accuracy of computational graph pruning, the contribution degree evaluation function not only considers the computational cost and prediction accuracy of the computational node, but also combines the task characteristic parameters. Specifically, the contribution degree evaluation function is defined as:

[0089] ;

[0090] wherein is the contribution score of node , represents the influence degree of node on prediction accuracy, represents the calculation cost of node , represents the adjustment factor of node to the current task.

[0091] Before calculating the contribution evaluation function, the influence degree and the calculation cost are normalized to map them to a unified numerical range, so as to ensure the rationality and comparability of data of different dimensions in calculation.

[0092] wherein, is defined as:

[0093] ;

[0094] wherein represents the adjustment factor of node to the current task, represents the task feature vector, represents the node function feature vector, is a cosine similarity function, used to measure the matching degree of task demand and node function, is an exponential function.

[0095] wherein, the specific implementation of function is to calculate the similarity between the task feature vector and the node function feature vector. By calculating the cosine value of the included angle between the two vectors, the dot product of the two vectors is first calculated, and then divided by the product of the lengths of the two vectors. The function maps the similarity to -1 to 1, and the value closer to 1 indicates that the directions of the two vectors are more consistent, i.e. the task demand and the node function are more matched, and the value closer to -1 indicates that the directions are more opposite.

[0096] Before using the cosine similarity function, the categorical features such as task type are converted into numerical feature vectors by One-Hot Encoding.

[0097] The task-aware computation graph pruning algorithm performs the following steps: task feature analysis is performed on the current trajectory prediction task to determine the task type (such as pedestrian trajectory prediction, vehicle trajectory prediction), task accuracy requirement and time limit; for each node in the reconstructed computation graph, sensitivity analysis is performed, and the node is temporarily removed, and the change of prediction accuracy is measured on the validation set ;

[0098] Computing node impact degree:

[0099] ;

[0100] wherein represents the node impact degree on prediction accuracy, represents the node change in prediction accuracy after being removed, is the prediction accuracy of the complete model. For each computing node measure its computing cost , including computing time, memory occupation and energy consumption;

[0101] According to the task characteristics, calculate the task relevance adjustment factor of each node:

[0102] Analyze the relevance of the node function to the current task; according to the relevance degree, specify the adjustment factor value, the adjustment factor value of the node with high relevance is large; calculate the contribution score of each node; determine the contribution threshold , which is adaptively adjusted according to the resource limit and the accuracy requirement; remove the nodes with the contribution score lower than the threshold, i.e. nodes; reconstruct the connection relationship for the retained node set to ensure the continuity of the data flow; verify the performance of the optimized computing graph, if it does not meet the requirements, adjust the threshold , continue to remove the nodes with the contribution score lower than the threshold, and reconstruct the connection relationship for the retained node set and verify the performance of the optimized computing graph.

[0103] As shown in Figure 3 , the node number changes before and after the computing graph pruning of three different task types (pedestrian trajectory prediction, vehicle trajectory prediction, and mixed trajectory prediction) are compared, as well as the proportion of high-priority nodes retained by each. Through the graph, the adaptability of the task-aware computing graph pruning algorithm to different types of tasks can be clearly shown, verifying the ability of the algorithm to effectively reduce the computing complexity while maintaining the prediction accuracy.

[0104] Step 4, evaluate the computing task priority for the nodes in the optimized computing graph, and determine the execution location of each node on the edge device and the cloud;

[0105] Perform priority evaluation on each computing task in the optimized computing graph to determine which tasks are executed locally on the edge device and which tasks are migrated to the cloud for execution. The priority evaluation is mainly based on factors such as time sensitivity, resource demand, and data transmission cost of the task.

[0106] where the priority of a computing task is evaluated by the following formula:

[0107] ;

[0108] where, represents the priority score of task , the higher the score, the more suitable it is to be executed on edge devices; represents the time sensitivity indicator; represents the resource demand indicator; represents the data transmission cost indicator; , , represent the time sensitivity weight, resource demand weight and data transmission cost weight, respectively.

[0109] where the function is used to evaluate the degree of real-time requirement of a computing task. The specific implementation considers three main factors: the position of the task in the prediction process (front-end processing tasks are usually more time-sensitive), the timeliness decay of the task result (the rate at which the result value decreases over time), and the dependency relationship of the task (tasks that are dependent on multiple critical tasks are more time-sensitive). By weighting the combination of these factors, a comprehensive time sensitivity score is obtained, with a higher score indicating that the task needs to be executed more timely.

[0110] The function is used to estimate the computing resources required for the task to be executed on the edge device. The specific implementation is based on historical execution data and task characteristics, using a regression model to predict the resource consumption of the task in terms of processor usage, memory occupation and energy consumption, then according to the current device's resource tension degree, different weights are given to calculate the weighted sum as the overall resource demand indicator. The larger the value, the higher the demand for resources.

[0111] The function is used to evaluate the cost of transmitting task data between edge devices and the cloud. The specific implementation is to calculate the transmission time and network resource occupation according to the size of the task input and output data, the current network bandwidth and network delay. The function will divide the data size by the bandwidth to get the theoretical transmission time, and then apply a penalty coefficient according to the current network delay condition, finally get the comprehensive data transmission cost.

[0112] Before calculating the priority of a computing task, the time sensitivity indicator , the resource demand indicator and the data transmission cost indicator are standardized to map them to the [0, 1] interval, ensuring the rationality of different dimension indicators in weighted calculation.

[0113] In this application embodiment, in order to make more effective use of the heterogeneous computing resources of edge devices and the cloud, the task priority evaluation also considers the characteristics of different computing platforms, such as the speedup ratio of the GPU / TPU acceleration capability of the cloud to a specific type of computing task, as well as the real-time status of network bandwidth and latency.

[0114] The computation task priority evaluation algorithm is performed through a series of steps: first, for each computation task in the optimized computation graph... Feature extraction is performed to analyze its computational characteristics (including computational complexity, memory requirements, and parallelization potential) and data dependencies (determining the input data source and the output data destination); then, the time sensitivity index of the task is calculated. The assessment evaluates the impact of task results on system real-time performance and considers the task's position in the prediction process; then, it estimates the resource requirements of the task on edge devices. It predicts the CPU utilization, memory usage, and energy consumption of the task, while considering the current resource availability of the device; then it calculates the data transfer cost of the task. Estimate the size of the task's input and output data, taking into account transmission delay and bandwidth usage under current network conditions; then calculate the task priority score. And determine the priority threshold that will be dynamically adjusted based on the current system load and network conditions. Finally, tasks are categorized (those with priority scores above a threshold). Tasks are executed locally on edge devices, while tasks below a threshold are migrated to the cloud for execution, and a task scheduling plan is generated that includes the task execution order, resource allocation, and communication scheduling.

[0115] Step 5: Based on the execution location, the execution scenario is divided into adaptive model decomposition, and the trajectory prediction model is decomposed into edge-side and cloud-side components;

[0116] Based on the characteristics of the current scenario and the analysis results of the preceding steps, the trajectory prediction model is adaptively decomposed into edge-side and cloud-side components. The edge-side components are responsible for handling tasks that are highly time-sensitive and have relatively low resource requirements, while the cloud-side components are responsible for handling computationally intensive tasks.

[0117] The model decomposition strategy can be represented as a mapping function:

[0118] ;

[0119] in The mapping function will optimize the set of nodes in the computation graph. Mapping to execution location set middle; This indicates that it is executed on an edge device. This indicates that the operation is performed in the cloud.

[0120] mapping function Map each node in the optimized computation graph to its execution location (edge device or cloud). During model decomposition, the execution location of the node is binary coded, and the categorical output is converted to numerical data for processing.

[0121] In the embodiments of the present application, in order to ensure the execution efficiency after model decomposition, the decomposition strategy also considers the data dependency relationship between the edge side and the cloud side components, and tries to reduce the number of data transmission between components while ensuring the smoothness of data flow.

[0122] The edge cloud collaborative architecture is composed of edge side inference components (including lightweight feature extraction units, primary feature processing units and time-sensitive prediction units, which are specially designed as resource-efficient computing modules), cloud processing components (including high-computing-complexity feature fusion units, deep time series analysis units and high-precision prediction units, which have strong computing power), data transmission optimization components (responsible for data compression, transmission scheduling and communication optimization between edge and cloud), task scheduling components (responsible for dynamic allocation and scheduling of computing tasks, including task priority management units and resource allocation units) and result integration components (combining the computing results of edge side and cloud to generate the final trajectory prediction output).

[0123] The scene-adaptive model decomposition strategy performs the following steps: analyzing the data dependency relationship between nodes in the computation graph, and constructing a dependency graph:

[0124] ;

[0125] wherein represents the dependency graph; represents the node set in the optimized computation graph; represents the data dependency relationship between nodes;

[0126] Calculate the time sensitivity index of each node , which reflects the degree of influence of node execution delay on the final result; estimate the resource requirements of each node when executed on the edge device and the cloud and ; estimate the communication overhead of data transmission between nodes , including data volume, network bandwidth and delay; construct a model decomposition optimization objective function, considering execution time, resource utilization and communication overhead; use heuristic algorithm or dynamic programming method to solve the optimization problem to obtain the optimal node mapping scheme ; wherein the objective function of the model decomposition optimization problem is defined as:

[0127] ;

[0128] wherein is a node mapping function; is a set of nodes in the optimized computation graph; is an execution time cost function; is a resource utilization cost function; is a communication cost function; is a set of neighboring nodes of ; and are a resource utilization cost weight coefficient and a weight coefficient, respectively; and are the th and the th node, is a neighboring node of ; denotes a minimization operation; denotes a summation symbol.

[0129] wherein the execution time cost function is used to predict the time overhead of a node in a specified execution location (edge or cloud). The implementation is to first obtain the baseline execution time of a node, and then apply a corresponding speed scaling factor according to the processing power characteristics of the execution location. For example, a computation-intensive node may be several times faster in the cloud than in an edge device, while an IO-intensive node may benefit less. The function outputs the expected execution time of a node in a specific execution location.

[0130] The resource utilization cost function is used to evaluate the degree of resource occupation of a node in a specified execution location. The implementation is to calculate the ratio of the usage amount of a node to the total capacity of the corresponding resource of the execution location for multiple resource types such as processor, memory, and power, and then perform a weighted summation according to the importance of different resource types. This function reflects the contribution of node execution to the device resource pressure, and the higher the value, the more serious the resource occupation.

[0131] The communication cost function is used to calculate the data transmission overhead between two neighboring nodes. The implementation is to determine whether cross-device communication is needed according to the execution location of the nodes: if the two nodes are located in the same execution location (both in the edge or both in the cloud), the communication overhead is zero; if they are located in different locations, the communication cost is calculated according to the data transmission amount between them, the current network bandwidth, and the delay condition. The function also considers the additional impact of network delay on real-time requirements of applications, which is reflected through a penalty coefficient.

[0132] Before calculating the objective function, the execution time cost function , the resource utilization cost function and communication cost function Normalization is performed to scale cost indicators of different dimensions to the same numerical range, ensuring comparability in optimization calculations;

[0133] According to the mapping scheme The computing graph nodes are divided into an edge-side node set and a cloud node set ;

[0134] Insert necessary data communication nodes between the edge-side and cloud node sets to ensure the continuity of data flow;

[0135] According to the current network conditions and device resource status, dynamically adjust the decomposition strategy to adapt to environmental changes.

[0136] As Figure 4 shown, the task allocation of different components and their processing delays in the edge cloud collaborative execution architecture are shown. The column chart represents the number of nodes of each component, and the line chart represents the corresponding processing delay. Through the chart, the computing load distribution of edge-side and cloud components and their respective processing efficiency can be analyzed, verifying the effect of the scene-adaptive model decomposition strategy in optimizing resource utilization and reducing processing delay.

[0137] Step 6, through the edge device and the cloud cooperatively execute the respective components, integrate the computing results of both sides to generate the trajectory prediction result of the target object;

[0138] According to the execution plan generated in the previous steps, the edge-side and cloud-side components are collaboratively executed. The edge device executes the local components and transmits the necessary intermediate results to the cloud; the cloud executes complex computing tasks and returns the computing results to the edge device. Finally, the edge device integrates all the computing results to generate the complete trajectory prediction result.

[0139] Among them, the trajectory prediction result includes the position coordinate sequence of the target object at multiple future time points, represented as:

[0140] ;

[0141] Among them is the prediction time range, and are the horizontal and vertical coordinates of the predicted position coordinates at time.

[0142] In the embodiments of the present application, in order to improve the robustness of the prediction, the trajectory prediction result not only includes the most likely trajectory path, but also includes multiple possible trajectory hypotheses and their confidence levels, forming a probability distribution representation. This representation can more comprehensively reflect the uncertainty of target motion and provide more abundant decision-making basis for downstream applications.

[0143] The trajectory prediction result generation algorithm performs the following steps: obtaining respective calculation results from the edge side and the cloud, including feature representations, intermediate states, and partial prediction results; before converging data from different sources, standardizing the calculation results of the edge device and the cloud to ensure that data generated by different computing platforms are comparable when fused; performing time alignment on the obtained results to address time differences caused by edge cloud communication delays; fusing real-time features of the edge device and deep features of the cloud to generate enhanced feature representations; based on the enhanced features, using a trajectory generation component of a multi-modal trajectory prediction model to calculate future trajectories, first predicting the position coordinates of the trajectory prediction target object at key time points, and then using a trajectory interpolation algorithm to generate complete continuous trajectories; calculating the uncertainty of the trajectories to generate multiple possible trajectory hypotheses, calculating probability distribution parameters for each predicted trajectory point, and sampling from the probability distribution to generate multiple possible trajectory paths; calculating confidence scores for the generated multiple trajectories to form a weighted trajectory set; post-processing the trajectories, including smoothing, physical constraint checking, and scene consistency verification; and generating final trajectory prediction results and outputting them in a format suitable for downstream applications.

[0144] As shown in Figure 5 , the calculation efficiency indicators of the present method and two comparative methods (full model and simplified model) under different scene complexity are compared, including average response time and CPU occupancy. Through the figure, the calculation efficiency advantage of the present method under various scenes can be directly observed, especially the improvement compared with traditional methods in high complexity scenes, verifying the efficiency of the present method under resource limited conditions.

[0145] As shown in Figure 6 , the balance between resource utilization and prediction accuracy of the present method under different scene complexity is shown. The horizontal axis represents the scene complexity score, and the vertical axis represents the normalized values of resource utilization and prediction accuracy. Through the figure, it can be directly observed that as the scene complexity increases, the present method can adaptively adjust resource allocation, optimizing resource utilization while maintaining high prediction accuracy, verifying the adaptive ability and efficiency of the present method.

[0146] A trajectory prediction system based on multi-modal perception for executing the above-mentioned trajectory prediction method based on multi-modal perception, comprising:

[0147] A data perception and graph structure generation module for obtaining multi-source heterogeneous perception data and constructing an initial calculation graph;

[0148] A scene perception and graph structure reconstruction module for analyzing environmental complexity and dynamically adjusting the calculation graph based on device resource status;

[0149] A task-aware pruning module for removing low-contribution computing nodes and optimizing the computing flow;

[0150] A computing task scheduling module for evaluating task priorities and determining optimal execution locations;

[0151] A model self-adaptive decomposition module for dividing the prediction model into edge and cloud components according to scene characteristics;

[0152] A collaborative computing and result generation module for integrating edge cloud computing results and outputting target trajectory predictions.

[0153] Here, an embodiment of the present application is provided:

[0154] The method of this embodiment is applied to an intelligent traffic management system for complex intersections in urban traffic. The system is deployed at five complex traffic intersections in a certain city, and various edge devices are installed at each intersection, including: 8 high-definition cameras (4K resolution, 30 frames / second); 4 millimeter wave radar sensors (detection distance 80 meters); 2 laser radars (16 lines, scanning frequency 10 Hz); 3 roadside processing units (RSU, configured with quad-core processors, 8 GB memory, limited edge computing capability).

[0155] These edge devices need to collaboratively process multi-modal data from different sensors to perform real-time trajectory prediction for traffic participants (including vehicles, pedestrians, and non-motor vehicles) to support intelligent traffic signal control, safety warning, and autonomous driving assistance systems. There are 120-180 vehicles, 50-80 pedestrians, and 30-50 non-motor vehicles passing through the intersection per minute, with a 50% increase in traffic flow and an increase in scene complexity during peak hours.

[0156] The system collects multi-modal perception data in the above-mentioned scenarios, and the multi-modal perception data collection is shown in Table 1:

[0157] Table 1: Multi-modal perception data collection

[0158]

[0159] Based on the collected multi-modal data, the system generates an initial computation graph structure. The computation graph contains 212 computing nodes, and the initial computation graph structure composition is shown in Table 2:

[0160] Table 2: Initial computation graph structure composition

[0161]

[0162] The system analyzes the scene complexity of the current intersection by monitoring the number of target objects, motion pattern diversity, and occlusion level in the scene. At the same time, the system monitors the resource status of the roadside processing unit. The scene complexity analysis results and the calculation graph reconstruction situation are shown in Table 3:

[0163] Table 3: Scene complexity analysis results and calculation graph reconstruction situation

[0164]

[0165] According to the scene complexity score and device resource status, the system dynamically reconstructs the calculation graph structure, reducing the number of calculation nodes during high-load periods (8:00 and 17:45), while retaining more nodes during low-load periods (10:30 and 21:30) to improve prediction accuracy.

[0166] The system performs task-aware calculation graph pruning on the reconstructed calculation graph and evaluates the calculation task priority to determine the execution location of the task. The calculation graph pruning results of three different task types are shown in Table 4:

[0167] Table 4: Calculation graph pruning results of different task types

[0168]

[0169] During peak hours (17:45), the system evaluates the priority of each calculation task in the pruned calculation graph based on time sensitivity, resource demand, and data transmission cost, and determines the execution location of each task.

[0170] According to the task priority evaluation results, the system performs scene-adaptive model decomposition, decomposing the trajectory prediction model into edge-side and cloud components. The model decomposition and edge-cloud collaborative execution situation are shown in Table 5:

[0171] Table 5: Model decomposition and edge-cloud collaborative execution situation

[0172]

[0173] Through the edge-cloud collaborative execution strategy, the system can reduce the calculation delay and resource consumption while maintaining the prediction accuracy. Especially during peak hours, edge devices can focus on time-sensitive calculation tasks, while computation-intensive tasks are migrated to the cloud for processing, achieving a balance between resource utilization and calculation efficiency.

[0174] To verify the effectiveness of the embodiment, we conducted a three-month system deployment test in the urban traffic scenario described above, and compared the method with two comparative methods: Comparative Method 1: Direct deployment of the complete model on the edge device (without optimization); Comparative Method 2: Simplified model through traditional model compression technology (pruning, quantization).

[0175] The comparison results of the computational efficiency of the method under different scene conditions are shown in Table 6:

[0176] Table 6: Comparison results of computational efficiency

[0177]

[0178] The results show that the method achieves computational efficiency improvement in various complexity scenarios. In particular, in the high complexity scenario, the comparative method 1 (complete model) crashes due to insufficient resources, while the method can run stably and has higher efficiency than the comparative method 2 (simplified model) through dynamic computation graph optimization and edge cloud collaboration.

[0179] The comparison results of the prediction accuracy of the method are shown in Table 7:

[0180] Table 7: Comparison results of prediction accuracy

[0181]

[0182] The results show that although the computational complexity of the method is reduced, its prediction accuracy is still close to that of the complete model (comparative method 1), and far superior to that of the model obtained by simple model compression (comparative method 2). In particular, in the complex interaction scenario, the method achieves high prediction accuracy by retaining key multi-modal fusion and spatio-temporal interaction modeling capabilities.

[0183] In summary, the results of Table 6 and Table 7 show that the method of the embodiment achieves a good balance between computational efficiency and prediction accuracy, verifying the effectiveness of the dynamic reconfigurable computation graph structure, task-aware computation graph pruning, edge cloud collaborative inference framework, and other technologies, and providing a practical solution for multi-modal trajectory prediction in resource-constrained environments.

[0184] The above describes embodiments of the present application, but the embodiments are not limited to the specific implementation described above, which is only illustrative and not limiting. Those skilled in the art can make more forms of equivalent embodiments under the inspiration of the embodiments, which are all within the protection scope of the embodiments.

Claims

1. A multi-modal perception based trajectory prediction method, characterized in that, The method comprises the following steps: Collecting multi-modal perception data to generate an initial computational graph structure; The multi-modal perception data includes visual data, point cloud data, and radar data; Generating the initial computational graph structure based on a preset trajectory prediction model; the initial computational graph structure comprises a plurality of computational nodes and edges connecting the nodes, wherein each computational node represents a specific data processing operation, and the edges represent the direction of data flow; The trajectory prediction model is composed of the following components: a multi-modal feature extraction component, including a visual feature extraction unit, a point cloud feature extraction unit, and a radar feature extraction unit, which are responsible for extracting feature representations from different modal sensor data respectively; A feature fusion component receives the outputs of the modal feature extraction units and fuses multiple heterogeneous features into a unified feature representation; A time series modeling component for capturing the time dependence of the target historical trajectory, including an attention mechanism-based time series relationship modeling unit; A context understanding component for analyzing the surrounding environment information of the target, including a scene semantic analysis unit and an interaction relationship modeling unit; a trajectory generation component based on fused features and time series and context information to predict future trajectories, including a decision prediction unit and a trajectory interpolation unit; Analyzing the scene complexity and dynamically reconstructing the computational graph structure based on the scene complexity and device resource state; The step of analyzing the scene complexity comprises: calculating the number of target objects in the scene, the motion pattern diversity index, and the occlusion degree index; normalizing the number of target objects in the scene, the motion pattern diversity index, and the occlusion degree index to map them to a unified numerical range; obtaining the scene complexity score through weighted calculation; determining the computational complexity level based on the scene complexity score; The step of dynamically reconstructing the computational graph structure comprises: obtaining the current device resource state, including the availability of computing resources, the memory capacity, and the power level; converting the device resource state into a resource constraint coefficient; determining the computational graph complexity target value; selecting a computational graph template closest to the target value from a predefined computational graph template library; refining the selected computational graph template based on the specific characteristics of the current scene to generate a reconstructed computational graph structure; Performing task-aware pruning on the reconstructed computational graph structure, removing computational nodes with a contribution degree below a preset threshold to generate an optimized computational graph; Evaluating the computational task priority for the nodes in the optimized computational graph to determine the execution location of each node on the edge device and the cloud; According to the execution location, the model decomposition is divided into execution scene adaptation, and the trajectory prediction model is divided into edge side and cloud side components; The edge device and the cloud cooperatively execute their respective components, integrate the computing results of both sides, and generate the trajectory prediction result of the target object.

2. The method of claim 1, wherein, The step of evaluating the computational task priority comprises: feature extraction of each computational task in the optimized computational graph; calculation of the time sensitivity index of the computational task; estimation of the resource demand of the task on the edge device; calculation of the data transmission cost of the task; standardization processing of the task features, time sensitivity index, resource demand, and data transmission cost; obtaining the task priority score through weighted calculation; based on the priority score, the tasks are classified into tasks executed locally on the edge device and tasks migrated to the cloud for execution.

3. The method of claim 1, wherein, The step of performing task-aware pruning includes: performing task feature analysis on the current trajectory prediction task to determine the task type, precision requirement and time limit; performing sensitivity analysis on each node in the reconstructed computation graph to calculate the node influence degree; measuring the calculation cost of each calculation node; calculating the task relevance adjustment factor of each node; calculating the contribution score of each node; determining the contribution threshold; removing nodes with a contribution score lower than the threshold; reconstructing the connection relationship of the retained node set to ensure the continuity of data flow.

4. The method of claim 1, wherein, The step of performing scene-adaptive model decomposition includes: analyzing the data dependency relationship between nodes in the computation graph to build a dependency graph; calculating the time sensitivity index of each node; estimating the resource requirements of each node for execution on the edge device and the cloud; estimating the communication overhead of data transmission between nodes; constructing a model decomposition optimization objective function; normalizing the cost indicators of different dimensions; solving the optimization problem to obtain the optimal node mapping scheme; and dividing the computation graph nodes into an edge-side node set and a cloud-side node set according to the mapping scheme.

5. The method of claim 1, wherein, The collaborative execution includes: the edge device performing the calculation task of the edge-side component; transmitting the necessary intermediate results to the cloud; the cloud performing the calculation task of the cloud-side component; returning the cloud calculation result to the edge device; standardizing the calculation results of the edge device and the cloud; fusing the real-time features of the edge device and the deep features of the cloud to generate enhanced feature representations; and calculating the future trajectory based on the enhanced features.

6. The method of claim 1, wherein, The step of generating the trajectory prediction result of the target object includes: calculating the position coordinates of the key time points; generating a complete continuous trajectory using a trajectory interpolation algorithm; calculating the uncertainty of the trajectory; generating trajectory hypotheses and their confidence; and post-processing the generated trajectory, including smoothing, physical constraint checking and scene consistency verification.

7. A multi-modal perception based trajectory prediction system, characterized in that, A multi-modal perception-based trajectory prediction method for performing any one of claims 1-6, comprising: a data perception and graph structure generation module for acquiring multi-source heterogeneous perception data and constructing an initial computation graph; a scene perception and graph structure reconstruction module for analyzing environmental complexity and dynamically adjusting the computation graph based on device resource status; a task-aware pruning module for removing low-contribution calculation nodes and optimizing the calculation process; a calculation task scheduling module for evaluating task priority and determining the best execution location; a model adaptive decomposition module for dividing the prediction model into edge-side and cloud-side components according to scene characteristics; a collaborative calculation and result generation module for integrating edge cloud calculation results and outputting the target trajectory prediction.

Images