Method, apparatus, device and storage medium for trajectory generation

Abstract

Embodiments of the present disclosure relate to a method, device, equipment and storage medium for trajectory generation. The method comprises: obtaining a plurality of images of a scene in which a movable target object is located in a historical time period; generating, by using a scene prediction model, a target voxel representation of the scene in a target time period based on at least the plurality of images; and generating, by using a trajectory generation model, a predicted trajectory of the target object in the target time period based on at least the target voxel representation. In this way, the accuracy of the predicted trajectory can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and more specifically, to methods, apparatus, devices, and storage media for trajectory generation. Background Technology

[0002] As the application of mobile intelligent agents in real-world environments continues to expand, trajectory planning technology is facing increasingly complex perceptual inputs and diverse task requirements. Especially in dynamic scenarios, effectively responding to control intentions from decision-making systems while ensuring trajectory rationality and safety has become a key challenge in trajectory generation. In recent years, end-to-end data-driven methods have received widespread attention due to their simple modeling process and good generalization ability. However, existing trajectory generation methods still face many challenges when dealing with complex dynamic environments. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for trajectory generation is provided. The method includes: acquiring multiple images of a scene in which a movable target object is located over a historical time period; generating a target voxel representation of the scene over a target time period using a scene prediction model, based at least on the multiple images; and generating a predicted trajectory for the target object over the target time period using a trajectory generation model, based at least on the target voxel representation.

[0004] In a second aspect of this disclosure, an apparatus for trajectory generation is provided. The apparatus includes: an image acquisition module configured to acquire multiple images of a scene in which a movable target object is located over a historical time period; a target voxel representation generation module configured to generate a target voxel representation of the scene over a target time period using a scene prediction model, based at least on the multiple images; and a predicted trajectory generation module configured to generate a predicted trajectory for the target object over the target time period using a trajectory generation model, based at least on the target voxel representation.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram of an example architecture for generating predicted trajectories according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram of an example architecture for generating target voxel representations according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A flowchart of a process for trajectory generation according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A schematic structural block diagram of an apparatus for trajectory generation according to some embodiments of the present disclosure is shown;

[0015] Figure 6 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0018] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," "machine learning network," "neural network," or simply "network," and these terms are used interchangeably.

[0019] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0023] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown. References Figure 1 One or more movable objects, such as a first robot 110-1, a second robot 110-2, and a third robot 110-3, are deployed in environment 100. First robot 110-1, second robot 110-2, and third robot 110-3 can also be collectively or individually referred to as robot 110. Each robot 110 may be equipped with control devices to control its operation. For example, ... Figure 1 As shown, control devices 120-1, 120-2, and 120-3 are respectively deployed at the first robot 110-1, the second robot 110-2, and the third robot 110-3, which may also be collectively referred to as control device 120 or individually. In some embodiments, only one robot may be deployed in the environment 100, and only the movement path of this one robot needs to be planned. In some embodiments, multiple robots may be deployed in the environment 100. In this case, path planning can be performed simultaneously for multiple robots in the environment 100, or path planning can be performed only for any one robot in the environment 100.

[0026] In embodiments of this disclosure, robot 110 can be used for various suitable purposes, such as a transport robot for delivering goods, and control device 120 can be used to plan the movement path of robot 110. Robot 110 can, for example, perform parcel delivery services or warehouse management services. As an example, robot 110 can operate in indoor environments, such as warehouse environments.

[0027] exist Figure 1 In environment 100, control device 120 can be any type of microcontroller, programmable logic controller, industrial computer, single-board computer, field-programmable gate array, digital signal processor, multi-core processor, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, control device 120 can control robot 110 based on control commands it generates.

[0028] In some embodiments, the operation of robot 110 can be controlled via server 130. Server 130 can receive relevant data for various tasks and remotely control the operation of robot 110. Control device 120 collects data from robot 110 and communicates with server 130 via network 140 to send the data to server 130. Server 130 determines control commands for the robot based on the acquired data and sends the control commands to control device 120. Control device 120 controls robot 110 based on the received control commands. Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0029] In some embodiments, robot 110 can connect to and communicate with server 130 via network 140. Robot 110 can send relevant data on various tasks performed by robot 110 to server 130 in real time.

[0030] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Furthermore, although in Figure 1 The illustration shows a robot, but embodiments of this disclosure can also be applied to other types of movable objects.

[0031] As briefly mentioned earlier, with the widespread application of autonomous mobile systems in fields such as industrial manufacturing, autonomous driving, and robotics, how to achieve safe and efficient path planning and trajectory prediction in complex environments has become one of the important research directions for mobile intelligent systems. The core objective of trajectory prediction is to generate a feasible and reasonable motion path for the moving body based on perceived environmental information and given task objectives.

[0032] Traditional trajectory planning methods typically employ a modular, phased approach. For example, an environment model is first built using a perception module, then a behavior decision module generates target instructions, and finally, a trajectory planning module calculates the specific path. While this segmented "perception-decision-planning" approach offers a degree of engineering control, information transmission bottlenecks exist between the multiple modules. The perception layer may lose some semantic or geometric details from the original image, affecting the accuracy of trajectory prediction. Furthermore, redundant processing and error accumulation between modules increase the overall system complexity and maintenance costs, and make it difficult to quickly respond to sudden changes in dynamic scenarios.

[0033] With the development of deep learning, end-to-end trajectory generation methods have begun to receive widespread attention. These methods attempt to predict motion trajectories directly from perceptual inputs (such as images and state information) using a unified neural network structure, skipping intermediate explicit decision-making logic to achieve a more efficient decision-making loop. However, traditional trajectory prediction methods still use two-dimensional images or their encoded features as perceptual information for the trajectory generation model, lacking explicit modeling of scene geometry and temporal dynamics. This implicit image representation struggles to capture environmental evolution trends such as the appearance of future obstacles and the shrinking of traversable areas, thus affecting the accuracy and robustness of the trajectory.

[0034] Furthermore, generating a reasonable and safe trajectory in complex or dynamic environments often requires models with stronger spatial structure understanding and temporal change perception capabilities. End-to-end trajectory generation methods based primarily on 2D images and their features, while computationally efficient, often lack sensitivity to impending obstacles or changes in passable areas within the scene. In such cases, this can lead to problems such as path deviation, high collision rates, or response lag.

[0035] In view of this, according to embodiments of the present disclosure, an improved scheme for trajectory generation is proposed to at least partially solve one or more of the above-mentioned problems. In this scheme, firstly, multiple images of the scene in which the movable target object is located over a historical time period are acquired. Then, using a scene prediction model, at least based on the multiple images, a target voxel representation of the scene over a target time period is generated. Further, using a trajectory generation model, at least based on the target voxel representation, a predicted trajectory for the target object over the target time period is generated.

[0036] In the embodiments of this disclosure, a voxel representation incorporating a time dimension is introduced as an image-aware feature to replace the traditional trajectory prediction method based on two-dimensional images or their encoded features. This approach enables the trajectory generation model to explicitly model dynamic changes in the future scene. Compared to methods relying solely on static image features, the predicted target voxel representation possesses stronger spatial structure representation and temporal evolution perception capabilities, providing clear information on obstacle distribution, open areas, and environmental change trends during the trajectory generation stage. This significantly improves the accuracy, feasibility, and foresight of the predicted trajectory.

[0037] Some exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0038] Figure 2 A schematic diagram of an example architecture 200 for generating predicted trajectories according to some embodiments of the present disclosure is shown. The following will primarily refer to... Figure 1The example environment described herein is for illustrative purposes only and is not intended to limit the scope of the invention. Embodiments of this disclosure can also be applied to other types of environments. Furthermore, architecture 200 will be described relative to server 130, but this is merely exemplary. Architecture 200 may be implemented on a remote server of the target object (e.g., server 130), or on the target object's own control device (e.g., control device 120), or may be implemented collaboratively by the remote server and the control device.

[0039] like Figure 2 As shown, server 130 can acquire multiple images 201 of the scene in which the movable target object is located. In embodiments of this disclosure, the target object may include various suitable types of movable objects, such as including but not limited to autonomous driving systems (e.g., autonomous vehicles), drones, robots, etc. Such objects are capable of autonomously moving toward a set target in their environment. For example, the target object may be one or more of the first robot 110-1, the second robot 110-2, and the third robot 110-3 described above (collectively or individually referred to as robot 110). Furthermore, for safe and efficient movement, it is necessary to perceive the environment in which the target object is located. Based on the perception results, the next action of the target object can be determined (such as determining the direction of travel, planning a reasonable path, etc.).

[0040] In some embodiments, the scene in which the target object is located can refer to the surrounding environment experienced by the target object within a certain time range, which may include static objects, dynamic objects, etc. Static objects may include, for example, roads, obstacles, buildings, shelves, and other objects that remain still over a relatively long period of time. Dynamic objects may include, for example, people, animals, other movable objects, and other objects that may move in real time.

[0041] During the movement of a target object, information about its surrounding environment over a specific historical time period can be acquired using image acquisition devices (e.g., cameras, vision sensors) to further construct 3D understanding or predict future scenes. Such image acquisition devices can be placed in the surrounding environment or on the target object itself. For example, server 130 can acquire multiple frames of images from one or more perspectives captured by a camera on the target object over a historical time period. These images record the environmental changes experienced by the target object, providing foundational data for subsequent future scene prediction.

[0042] In some embodiments, to support the trajectory generation model's perception of the scene, it typically relies on extracting image features from the acquired images as input to express the semantic and spatial information of the scene. However, image features obtained based on two-dimensional image encoding are often static and implicit vector representations, making it difficult to accurately reflect the structural details and dynamic changes of the environment over time. Especially when dealing with scenes with complex obstacle distributions or dynamic target interference, the representational power and interpretability of image features obtained based on two-dimensional image encoding are limited. In this case, predictive voxel representations that can express the semantics, spatial structure, and temporal evolution characteristics of the scene can be introduced, thereby reflecting the spatial structure of the environment, the passage space, and their changing trends over time.

[0043] refer to Figure 2 The scene prediction model can generate a target voxel representation 202 of the scene within a target time period, based on at least multiple acquired images 201. This target voxel representation 202, generated from multiple images 201 and historical scene information, explicitly expresses the three-dimensional structure of the scene within the target time period and its dynamic evolution over time. In other words, the target voxel representation 202 is a four-dimensional (4D) scene representation that includes a time dimension. It not only possesses the semantic abstraction capabilities of traditional image features but also provides rich prior information on geometric structure and traversable areas, thus making it more conducive to generating safe, reasonable, and controllable predicted trajectories.

[0044] As an example, Figure 3 A schematic diagram of an example architecture 300 for generating target voxel representations according to some embodiments of the present disclosure is shown. Figure 3 As shown, server 130 can determine the historical voxel representation 302 of a scene within a historical time period based on multiple images 301 of the scene within that time period. The scene prediction model 311 can be considered as... Figure 2 In the example of scene prediction model 211, multiple images 301 can be regarded as examples of multiple images 201, and target voxel representation 304 can be regarded as an example of target voxel representation 202.

[0045] For example, server 130 can use a decoder, such as a 3D encoder, to perform 3D modeling on multiple acquired images 301 to determine historical scene features for a historical time period. The encoder can be used to extract spatial structure and geometric information contained in the images and model the scene in a 3D voxel space. For example, the voxel representation can be a regular 3D mesh structure, where each cell can be used to represent whether a corresponding position in space is occupied, the voxel density value, semantic labels, or other attributes. Further, the decoder can be used to convert the historical scene features into a 3D voxel representation, resulting in historical voxel representation 302.

[0046] In some embodiments, the historical voxel representation 302 may include a set of voxel representations for multiple historical moments. This set includes several voxel representations corresponding to multiple frames of images or multiple historical moments. A particular frame voxel representation in the historical voxel representation may correspond to a three-dimensional spatial feature formed after the image acquired at a certain moment has been processed by an encoder.

[0047] As an example, a historical time period can represent a continuous time interval, such as several seconds in the past. Within this time period, server 130 can continuously acquire image frames at a fixed frequency (e.g., several frames per second), with each image frame corresponding to a specific moment within that time period. The multiple moments included in the historical time period can be understood as a collection of these image frame acquisition time points. For example, one image frame corresponds to every 0.2 seconds in the past second, for a total of 5 frames. Each frame represents a specific moment in the past (e.g., t0 = 0s, t1 = 0.2s, t2 = 0.4s, ...). Furthermore, server 130 can construct a corresponding voxel representation based on the image frames corresponding to each moment, that is, each image frame generates a 3D scene representation at the corresponding moment after 3D modeling. The voxel representation of each frame reflects the 3D spatial environment in which the target object is located at the corresponding moment. Therefore, multi-frame voxel representations can be used to describe the continuous 3D modeling results of the scene in which the target object is located within a historical time period.

[0048] Alternatively or additionally, multiple images within a historical time period may not be limited to a sequence of images acquired continuously at fixed time intervals. In other words, multiple images may include images acquired at non-equidistant intervals, such as images acquired at key moments manually set according to application requirements, or images acquired based on detected events. In this case, server 130 can also construct corresponding multi-frame voxel representations based on these non-continuous or non-equidistant images, thereby supporting subsequent scene information prediction and trajectory planning.

[0049] Continue to refer to Figure 3The scene prediction model 311 can generate a target voxel representation 304 of the scene within the target time period, based at least on the historical voxel representation 302 and control information 303 related to the target time period. To make the voxel representation of the future target time period generated by the scene prediction model 311 more closely match the needs of the actual trajectory prediction task, control information 303 related to the predicted target within the target time period can be introduced.

[0050] In practical applications, future environmental changes are diverse and uncertain, and different prediction tasks may correspond to different time ranges or trajectory preferences. The time range, regional focus, or behavioral direction required for trajectory generation tasks vary across different application scenarios. Therefore, relying solely on historical images or historical voxel representations for future scene prediction can easily lead to a lack of specificity or behavioral consistency in the generated target voxel representations. By introducing control information 303, the scene prediction model 311 can not only understand "what the current environment was like in the past," but also clearly define the structure of "when, in which areas, and for what purpose should it be predicted." In this way, it can provide more matching scene semantics and structural support for downstream trajectory generation, thereby improving the accuracy and practicality of the overall trajectory generation task.

[0051] In some embodiments, control information 303 may include time control information. The time control information may indicate a target time period. For example, the time control information may indicate a specific timestep of the target prediction time period, thereby clarifying the target time point or time range to be predicted (e.g., predicting the scene structure in the 1st or 1.5th second of the future). Alternatively or additionally, the specific time control information may be determined based on user input. The introduction of this time control feature allows the model to predict not only based on historical spatial scenes but also to clearly define the target "in which second of the future," avoiding the uncertainty in the time dimension when simply predicting scene information based on historical voxel representations.

[0052] Alternatively or additionally, control information 303 may include trajectory control information. Trajectory control information may indicate one or more target poses of the target object within a target time period. Each target pose may indicate the target object's position parameters (such as translations x and y) and attitude parameters (such as orientation angle and rotation angle) in three-dimensional space. Alternatively or additionally, the specific trajectory control information may be determined based on user input. The introduction of trajectory control information enables the model to explicitly respond to the target object's future motion intentions when generating target scene features of a movable object, thereby achieving precise control over the scene generation process. Through trajectory control information, directional simulation of scene changes can be performed under arbitrary control conditions, thus meeting the flexibility and diversity required for complex behavior-driven decision-making.

[0053] Alternatively or additionally, the control information 303 may include more, less, or different types of control information, such as control information for the task to be performed on the target object, control information for the movement strategy of the target object, etc. The control information 303 can further enhance the flexibility of the scene prediction model 311 in generating the target voxel representation 304, thereby enhancing its adaptability to trajectory generation tasks.

[0054] In some embodiments, the scene prediction model 311 can generate a target voxel representation 304 of the scene within a target time period, guided by control information 303. For example, one strategy for generating the target voxel representation 304 can be based on control information 303 and an attention mechanism, by modulating historical scene features to generate a target scene structure corresponding to the control conditions.

[0055] In some embodiments, the scene prediction model 311 can determine historical scene features at multiple scales over a historical time period based on historical voxel representations. Then, the scene prediction model 311 can obtain control features at multiple scales by extracting features from the control information 303.

[0056] Furthermore, the scene prediction model 311 can determine the target scene features corresponding to the control information for the target time period based on historical scene features at multiple scales and control features at multiple scales. For each scale, the scene prediction model 311 can apply an attention mechanism to the historical scene features at that scale, using the control features as conditions, to obtain the regulated scene features at that scale. Then, the scene prediction model 311 can determine the target voxel features by fusing the regulated scene features determined for each scale. Finally, the scene prediction model 311 can use a decoder to determine the target voxel representation based on the target scene features.

[0057] Alternatively or additionally, another strategy for generating the target voxel representation 304 can be a process of recovering a structured voxel representation that meets the prediction conditions from an initial noise state based on control information 303. In some embodiments, the scene prediction model 311 can generate target scene features by denoising noise data for a target time period based on historical voxel representations and control information. Further, the scene prediction model 311 can utilize a decoder to determine the target voxel representation based on the target scene features.

[0058] In practical implementation, the aforementioned denoising process can employ various modeling methods, including but not limited to flow matching methods based on continuous-time modeling, conditional diffusion models, or other generation mechanisms. Taking the flow matching method as an example, a continuous mapping path from the initial noise state to the target structure state can be learned, and a denoising trajectory can be fitted during the training phase, thereby quickly obtaining a structured output that satisfies the conditional constraints during prediction.

[0059] Return to reference Figure 2 The trajectory generation model 212 can generate a predicted trajectory 205 for a target object within a target time period, based at least on the target voxel representation 202. For example, the trajectory generation model 212 can perceive the spatial structure and dynamic changes of the scene within the target time period based on the target voxel representation 202, and then plan a feasible path for the target object within that time period. The target voxel representation 202 can provide the trajectory generation model 212 with dynamic information about the future scene structure, obstacle distribution, and passage space. Furthermore, the target voxel representation 202 can assist the model in determining the feasible movement path of the target object within the target time period. By introducing the target voxel representation, the trajectory generation model 212 acquires a structured perception capability for future environmental changes, thereby generating a more reasonable, safe, and intentionally consistent trajectory output.

[0060] In end-to-end trajectory generation tasks, models typically rely on raw input information such as images and states to directly predict the complete trajectory. While this approach possesses overall modeling capabilities, it often struggles to accurately understand and execute high-level task instructions in real-world scenarios with diverse task objectives or clear control intentions, resulting in a lack of controllability in trajectory behavior.

[0061] To address this issue, a local target point (or local waypoint) 206 can be introduced as an intermediate control representation. The local target point 206 can be used to explicitly indicate the short-term target position of the object during travel, serving as a spatial representation of the task instruction. By transforming various types of task requirements (e.g., "keep to the right") into explicit spatial target points (e.g., a location point to the right front), the model can obtain more direct and explicit control objectives, thereby improving the goal orientation and behavioral consistency of trajectory generation.

[0062] Continue to refer to Figure 2 The local target point 206 of the target object can be generated based on at least one of the target object's state information 203 or task information 204 and the target voxel representation 202. The state information indicates at least one of the target object's travel speed, current position, and current attitude.

[0063] The state information 203 of the target object is used to describe its current motion state. The state information 203 may include, but is not limited to, the following: the spatial position of the target object in the current coordinate system, used to determine its starting point or the context of its current environment; the traveling speed of the target object, which helps the trajectory prediction model infer its reasonable direction of motion and rate changes; and the current posture of the target object (e.g., direction of travel or orientation angle), etc.

[0064] The target object's task information 204 describes the external goals or constraints that the current trajectory prediction needs to meet. Task information 204 may include at least one of the following: initial constraint information related to the target object's movement in the scene and task description information related to the tasks the target object needs to perform in the scene. Initial constraint information refers to the basic task constraints related to the target object's movement set before the task begins. Initial constraint information may be expressed as a specification of movement patterns or path preferences. Task description information is a control task instruction received from the outside, used to dynamically adjust the target object's behavior strategy. This type of information may include at least one of the task-specific instructions received from the user or task-specific instructions generated using a machine learning model.

[0065] As an example, a target point generation model can be used to determine local target points 206. The target point generation model can determine local target points 206 of the target object during its movement by combining one or more of the aforementioned information with the target voxel representation. Using local target points 206, the movement trend or behavioral intention of the target object can be expressed, and control references can be provided for subsequent trajectory generation.

[0066] Continue to refer to Figure 2 The trajectory generation model 212 can generate a predicted trajectory of the target object based on the target voxel representation 202 and local target points 206. The trajectory generation model 213 can generate a predicted motion trajectory of the target object based on current environmental perception information and trajectory control information. The target voxel representation 202 provides an explicit expression of the future scene structure and dynamic changes, while the local target points 206 provide clear guidance on motion intention. By combining these two types of information, the responsiveness of the trajectory generation process to control intention and the rationality of spatial planning can be effectively improved.

[0067] In embodiments of this disclosure, the trajectory generation model 212 can adopt any structural form suitable for trajectory modeling or trajectory generation tasks. For example, it can be a recurrent neural network, an encoder-decoder structure, a Transformer-based sequence modeling network, a diffusion model, a flow matching model, or a combination of the above structures.

[0068] In some embodiments, the aforementioned process of generating a predicted trajectory of a target object within a target time period based on the target voxel representation can be based on a collaborative architecture consisting of a scene prediction model and a trajectory generation model. This process can be performed during the inference phase. During inference, the scene prediction model first generates a target voxel representation within the target time period based on the perceived image and control information. The trajectory generation model can use this target voxel representation as core input features to infer and plan the future path of the target object within the target time period. Alternatively or additionally, this process can be performed during the training phase of the scene prediction model and the trajectory generation model. Some embodiments are described below regarding the training of the scene prediction model and the trajectory generation model.

[0069] Continue to refer to Figure 2 During training, the scene prediction model 211 can generate a target voxel representation 202 of the scene in the target time period based on multiple images 201 of the scene where the target object is located in the historical time period. The target voxel representation 202 can be used to characterize information such as the occupancy status or accessibility of each spatial location in the scene in the future target time period. Then, the trajectory generation model 212 can generate a predicted trajectory 205 for the target object in the target time period based at least on the target voxel representation 202.

[0070] In some embodiments, server 130 may construct a loss function 213 based on the difference between the predicted trajectory 205 and the reference trajectory 206. Further, server 130 may update the trajectory generation model 212 and the scene prediction model 211 based on the loss function.

[0071] As an example, server 130 can obtain the predicted trajectory 205 and the corresponding reference trajectory 206 (e.g., an expert demonstration trajectory, a manually labeled trajectory, or other high-quality trajectory). Furthermore, server 130 can construct a trajectory error loss function 213 based on the predicted trajectory 205 and the reference trajectory 206. For example, the loss function 213 can be constructed using mean squared error (MSE), Euclidean distance between trajectory points, or more complex sequence distance metrics.

[0072] For example, a backpropagation algorithm can be used to optimize the parameters of the trajectory generation model 212 based on the loss function 213. Furthermore, the input to the trajectory generation model 212 includes the target voxel representation 202 generated by the scene prediction model 211. Therefore, the loss function 213 can be propagated forward to the scene prediction model 211 through the backpropagation path of the trajectory generation model 212. In other words, the loss function 213 is also used to jointly update the parameters of the scene prediction model 211. This joint training method allows the prediction results of the target voxel representation 202 to be more adapted to the overall task of trajectory generation.

[0073] As can be more clearly understood from the description of the various embodiments of this disclosure, in these embodiments, scene voxel representations containing a time dimension are introduced as image perception features to replace trajectory prediction based on two-dimensional images or their encoded features. Target voxel representations possess stronger spatial structure representation capabilities and temporal evolution perception capabilities, providing clear information on obstacle distribution, open areas, and environmental change trends during the trajectory generation stage. In this way, the accuracy and foresight of predicted trajectories can be improved, thereby reducing collision risks and enhancing the overall system's adaptability to dynamic environments.

[0074] Figure 4 A flowchart of a process 400 for trajectory generation according to some embodiments of the present disclosure is shown. Process 400 may be executed by a remote server of the target object (e.g., server 130), or by a control device of the target object itself (e.g., control device 120), or may be performed collaboratively by the remote server and the control device. Reference is made below. Figure 1 Describe the process 400.

[0075] In box 410, server 130 acquires multiple images of the scene in which the movable target object is located during a historical time period.

[0076] In box 420, server 130 uses a scene prediction model to generate a target voxel representation of the scene for a target time period, based on at least multiple images.

[0077] In box 430, server 130 uses a trajectory generation model, based at least on the target voxel representation, to generate a predicted trajectory for the target object within the target time period.

[0078] In some embodiments, generating a target voxel representation of a scene in a target time period based on multiple images includes: determining a historical voxel representation of the scene in a historical time period based on multiple images of the scene in a historical time period; and generating a target voxel representation of the scene in a target time period using a scene prediction model, based at least on the historical voxel representation and control information related to the target time period.

[0079] In some embodiments, the control information includes at least one of time control information or trajectory control information, wherein the time control information indicates a target time period, and the trajectory control information indicates one or more target poses of the target object during the target time period.

[0080] In some embodiments, determining the historical voxel representation of a scene within a historical time period includes: using an encoder to perform 3D modeling of multiple images to determine historical scene features within a historical time period; and using a decoder to determine the historical voxel representation based on the historical scene features.

[0081] In some embodiments, generating a target voxel representation of a scene within a target time period includes: using an encoder to determine historical scene features of the scene at multiple scales within a historical time period based on the historical voxel representation; obtaining control features at multiple scales by feature extraction of control information; determining target scene features corresponding to the control information within the target time period based on the historical scene features at multiple scales and the control features at multiple scales; and using a decoder to determine the target voxel representation based on the target scene features.

[0082] In some embodiments, determining the target scene features corresponding to the control information for the target time period includes: for each of multiple scales, applying an attention mechanism to the historical scene features at that scale by using the control features at that scale as a condition to obtain the regulated scene features at that scale; and determining the target voxel features by fusing the regulated scene features determined for multiple scales respectively.

[0083] In some embodiments, generating a target voxel representation of a scene within a target time period includes: generating target scene features by denoising noisy data for the target time period based on historical voxel representations and control information; and determining the target voxel representation based on the target scene features using a decoder.

[0084] In some embodiments, process 400 is performed during the training of a trajectory generation model and a scene prediction model, and the trajectory generation model and the scene prediction model are trained by: constructing a loss function based on the difference between the predicted trajectory and the reference trajectory; and updating the trajectory generation model and the scene prediction model based on the loss function.

[0085] In some embodiments, generating a predicted trajectory of a target object includes: generating local target points of the target object based on the target object's state information, task information, and target voxel representation, wherein the state information indicates at least one of the target object's velocity, position, and orientation, the task information indicates constraint information related to generating the predicted trajectory, and the local target points indicate the local target position of the target object during its movement in the scene; and generating a predicted trajectory of the target object based on the target voxel representation and local target points using a trajectory generation model.

[0086] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 5 A schematic structural block diagram of an apparatus 500 for training a trajectory generation model according to some embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in a server 130. The various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0087] refer to Figure 5As shown, the device 500 includes an image acquisition module 510 configured to acquire multiple images of the scene where the movable target object is located during a historical time period; a target voxel representation generation module 520 configured to generate a target voxel representation of the scene during a target time period using a scene prediction model, based at least on the multiple images; and a predicted trajectory generation module 530 configured to generate a predicted trajectory for the target object during a target time period using a trajectory generation model, based at least on the target voxel representation.

[0088] In some embodiments, the target voxel representation generation module 520 is further configured to determine the historical voxel representation of the scene in a historical time period based on multiple images of the scene in a historical time period; and to generate the target voxel representation of the scene in a target time period using a scene prediction model, based at least on the historical voxel representation and control information related to the target time period.

[0089] In some embodiments, the control information includes at least one of time control information or trajectory control information, wherein the time control information indicates a target time period, and the trajectory control information indicates one or more target poses of the target object during the target time period.

[0090] In some embodiments, the target voxel representation generation module 520 is further configured to use an encoder to perform three-dimensional modeling of multiple images to determine historical scene features for a historical time period; and to use a decoder to determine historical voxel representations based on the historical scene features.

[0091] In some embodiments, the target voxel representation generation module 520 is further configured to: utilize an encoder to determine historical scene features of the scene at multiple scales in a historical time period based on the historical voxel representation; obtain control features at multiple scales by feature extraction of control information; determine target scene features corresponding to the control information in a target time period based on the historical scene features at multiple scales and the control features at multiple scales; and utilize a decoder to determine the target voxel representation based on the target scene features.

[0092] In some embodiments, the target voxel representation generation module 520 is further configured to, for each of a plurality of scales, apply an attention mechanism to historical scene features at that scale by using control features at that scale as conditions, to obtain regulated scene features at that scale; and to determine target voxel features by fusing regulated scene features determined for each of the plurality of scales.

[0093] In some embodiments, the target voxel representation generation module 520 is further configured to generate target scene features by denoising noise data for a target time period based on historical voxel representations and control information; and to determine the target voxel representation based on the target scene features using a decoder.

[0094] In some embodiments, the apparatus 500 is implemented during the training of the trajectory generation model and the scene prediction model. The apparatus 500 also includes an update module configured to construct a loss function based on the difference between the predicted trajectory and the reference trajectory; and to update the trajectory generation model and the scene prediction model based on the loss function.

[0095] In some embodiments, the predicted trajectory generation module 530 is further configured to generate local target points of the target object based on the target object's state information, task information, and target voxel representation, wherein the state information indicates at least one of the target object's velocity, position, and orientation, the task information indicates constraint information related to the generation of the predicted trajectory, and the local target points indicate the local target position of the target object during its movement in the scene; and to generate the predicted trajectory of the target object based on the target voxel representation and local target points using a trajectory generation model.

[0096] Figure 6 A block diagram is shown of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. The electronic device 600 may, for example, be used to implement... Figure 1 The control device 120 shown. It should be understood that, Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0097] refer to Figure 6 Electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0098] Electronic device 600 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 600.

[0099] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0100] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0101] Input device 660 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0102] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0103] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0104] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0105] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0106] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0107] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is determined to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for trajectory generation, comprising: Obtain multiple images of the scene where the movable target object is located within a historical time period; Using a scene prediction model, at least based on the plurality of images, a target voxel representation of the scene is generated for the target time period; as well as Using a trajectory generation model, at least based on the target voxel representation, a predicted trajectory for the target object during the target time period is generated.

2. The method according to claim 1, wherein generating a target voxel representation of the scene in the target time period based on the plurality of images comprises: Based on multiple images of the scene during the historical time period, determine the historical voxel representation of the scene during the historical time period; as well as Using the scene prediction model, at least based on the historical voxel representation and control information related to the target time period, a target voxel representation of the scene in the target time period is generated.

3. The method according to claim 2, wherein the control information includes at least one of time control information or trajectory control information, wherein the time control information indicates the target time period, and the trajectory control information indicates one or more target poses of the target object during the target time period.

4. The method according to claim 2, wherein determining the historical voxel representation of the scene in the historical time period comprises: Using an encoder, three-dimensional modeling is performed on the multiple images to determine the historical scene features of the historical time period; as well as The historical voxel representation is determined using a decoder based on the historical scene features.

5. The method according to claim 2, wherein generating the target voxel representation of the scene during the target time period comprises: Using an encoder, based on the historical voxel representation, the historical scene features of the scene at multiple scales within the historical time period are determined; By extracting features from the control information, control features at multiple scales are obtained; Based on the historical scene features at the multiple scales and the control features at the multiple scales, the target scene features corresponding to the control information for the target time period are determined. as well as The target voxel representation is determined using a decoder based on the target scene features.

6. The method according to claim 5, wherein determining the target scene features corresponding to the control information for the target time period includes: For each of the multiple scales, by using the control features at that scale as conditions, an attention mechanism is applied to the historical scene features at that scale to obtain the regulated scene features at that scale. as well as The target voxel features are determined by fusing the regulated scene features determined for the multiple scales respectively.

7. The method of claim 2, wherein generating the target voxel representation of the scene during the target time period comprises: The target scene features are generated by denoising the noisy data for the target time period based on the historical voxel representation and the control information. as well as The target voxel representation is determined using a decoder based on the target scene features.

8. The method of claim 1, wherein the method is performed during the training of the trajectory generation model and the scene prediction model, and the trajectory generation model and the scene prediction model are trained in the following manner: Based on the difference between the predicted trajectory and the reference trajectory, a loss function is constructed; and Based on the loss function, the trajectory generation model and the scene prediction model are updated.

9. The method according to claim 1, wherein generating the predicted trajectory of the target object comprises: Based on the target object's state information, task information, and target voxel representation, local target points of the target object are generated. The state information indicates at least one of the target object's velocity, position, and orientation. The task information indicates constraint information related to generating the predicted trajectory. The local target points indicate the local target position of the target object during its movement in the scene. Using the trajectory generation model, a predicted trajectory of the target object is generated based on the target voxel representation and the local target points.

10. An apparatus for trajectory generation, comprising: The image acquisition module is configured to acquire multiple images of the scene where the movable target object is located over a historical time period; The target voxel representation generation module is configured to generate a target voxel representation of the scene in a target time period based on at least the plurality of images using a scene prediction model. as well as The predicted trajectory generation module is configured to generate a predicted trajectory for the target object in the target time period using a trajectory generation model, based at least on the target voxel representation.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images