Method, apparatus, device and storage medium for trajectory generation

Abstract

Embodiments of the present disclosure relate to a method, device, equipment and storage medium for trajectory generation. The method comprises: determining image features of a scene based on a plurality of images of the scene in which a target object is movable; determining local target points of the target object, the local target points indicating local target positions of the target object in a process of traveling in the scene; and generating a predicted trajectory of the target object based on the image features and the local target points by using a trajectory generation model. In this way, the effective response and flexible control of the predicted trajectory to a task can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein generally relate to the field of computer technology, and more specifically, to methods, apparatus, devices, and storage media for trajectory generation. Background Technology

[0002] As the application of mobile intelligent agents in real-world environments continues to expand, trajectory planning technology is facing increasingly complex perceptual inputs and diverse task requirements. Especially in dynamic scenarios, effectively responding to control intentions from decision-making systems while ensuring trajectory rationality and safety has become a key challenge in trajectory generation. In recent years, end-to-end data-driven methods have received widespread attention due to their simple modeling process and good generalization ability. However, existing trajectory generation methods still face many challenges when dealing with complex dynamic environments. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for trajectory generation is provided. The method includes: determining image features of the scene based on multiple images of a scene in which a movable target object is located; determining local target points of the target object, the local target points indicating local target positions of the target object during its movement within the scene; and generating a predicted trajectory of the target object based on the image features and the local target points using a trajectory generation model.

[0004] In a second aspect of this disclosure, an apparatus for trajectory generation is provided. The apparatus includes: an image feature determination module configured to determine image features of a scene based on multiple images of a scene in which a movable target object is located; a local target point determination module configured to determine local target points of the target object, the local target points indicating local target positions of the target object during its movement in the scene; and a predicted trajectory generation module configured to generate a predicted trajectory of the target object based on the image features and the local target points using a trajectory generation model.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product includes computer-executable instructions that, when executed by a processor, implement the method according to a first aspect of this disclosure.

[0008] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0011] Figure 2 A schematic diagram of an example architecture for a trajectory generation model according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A flowchart of a process for trajectory generation according to some embodiments of the present disclosure is shown;

[0013] Figure 4 A schematic structural block diagram of an apparatus for trajectory generation according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. Detailed Implementation

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0017] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," "machine learning network," "neural network," or simply "network," and these terms are used interchangeably.

[0018] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0019] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0020] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0021] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0022] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0024] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown. References Figure 1One or more movable objects, such as a first robot 110-1, a second robot 110-2, and a third robot 110-3, are deployed in environment 100. First robot 110-1, second robot 110-2, and third robot 110-3 can also be collectively or individually referred to as robot 110. Each robot 110 may be equipped with control devices to control its operation. For example, ... Figure 1 As shown, control devices 120-1, 120-2, and 120-3 are respectively deployed at the first robot 110-1, the second robot 110-2, and the third robot 110-3, which may also be collectively referred to as control device 120 or individually. In some embodiments, only one robot may be deployed in the environment 100, and only the movement path of this one robot needs to be planned. In some embodiments, multiple robots may be deployed in the environment 100. In this case, path planning can be performed simultaneously for multiple robots in the environment 100, or path planning can be performed only for any one robot in the environment 100.

[0025] In embodiments of this disclosure, robot 110 can be used for various suitable purposes, such as a transport robot for delivering goods, and control device 120 can be used to plan the movement path of robot 110. Robot 110 can, for example, perform parcel delivery services or warehouse management services. As an example, robot 110 can operate in indoor environments, such as warehouse environments.

[0026] exist Figure 1 In environment 100, control device 120 can be any type of microcontroller, programmable logic controller, industrial computer, single-board computer, field-programmable gate array, digital signal processor, multi-core processor, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, control device 120 can control robot 110 based on control commands it generates.

[0027] In some embodiments, the operation of robot 110 can be controlled via server 130. Server 130 can receive relevant data for various tasks and remotely control the operation of robot 110. Control device 120 collects data from robot 110 and communicates with server 130 via network 140 to send the data to server 130. Server 130 determines control commands for the robot based on the acquired data and sends the control commands to control device 120. Control device 120 controls robot 110 based on the received control commands. Server 130 can be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0028] In some embodiments, robot 110 can connect to and communicate with server 130 via network 140. Robot 110 can send relevant data on various tasks performed by robot 110 to server 130 in real time.

[0029] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Furthermore, although in Figure 1 The illustration shows a robot, but embodiments of this disclosure can also be applied to other types of movable objects.

[0030] As briefly mentioned earlier, with the widespread application of autonomous mobile systems in fields such as industrial manufacturing, autonomous driving, and robotics, how to achieve safe and efficient path planning and trajectory prediction in complex environments has become one of the important research directions for mobile intelligent systems. The core objective of trajectory prediction is to generate a feasible and reasonable motion path for the moving body based on perceived environmental information and given task objectives.

[0031] Traditional trajectory planning methods typically employ a modular, phased approach. For example, an environment model is first built using a perception module, then a behavior decision module generates target instructions, and finally, a trajectory planning module calculates the specific path. While this segmented "perception-decision-planning" approach offers a degree of engineering control, information transmission bottlenecks exist between the multiple modules. The perception layer may lose some semantic or geometric details from the original image, affecting the accuracy of trajectory prediction. Furthermore, redundant processing and error accumulation between modules increase the overall system complexity and maintenance costs, and make it difficult to quickly respond to sudden changes in dynamic scenarios.

[0032] With the development of deep learning, end-to-end trajectory generation methods have gained widespread attention. These methods attempt to predict motion trajectories directly from perceptual inputs (such as images and state information) using a unified neural network structure, skipping explicit decision-making logic in the middle, aiming to achieve a more efficient decision-making loop. However, traditional end-to-end trajectory generation methods still face a core challenge: insufficient trajectory control capability. In practical applications, systems often need to execute specific actions based on high-level decision-making instructions, such as "drive to the side of the road," "avoid obstacles," and "follow the target ahead." These instructions may come from human users, machine learning models, or even cloud-based collaborative systems. However, conventional end-to-end trajectory generation methods generally lack a structured expression mechanism for such external task intentions, making it difficult to accurately align the trajectory generation process with specific control requirements, resulting in poor flexibility and weak controllability.

[0033] In view of this, according to embodiments of the present disclosure, an improved scheme for trajectory generation is proposed to at least partially solve one or more of the above-mentioned problems. In this scheme, firstly, image features of the scene are determined based on multiple images of the scene in which the movable target object is located. Then, local target points of the target object are determined, indicating the local target position of the target object during its movement within the scene. Further, a trajectory generation model is used to generate a predicted trajectory of the target object based on the image features and the local target points.

[0034] In the embodiments of this disclosure, local target points are introduced as intermediate control conditions to transform diverse trajectory control information into structurally defined positional targets. These local target points serve as explicit input conditions during trajectory generation, guiding the model to produce target-oriented trajectory outputs. This approach combines structural clarity with flexible scalability. Therefore, this method can support dynamic trajectory guidance from external control inputs and also achieve autonomous planning when no task instructions are available, thereby improving the controllability and task adaptability of the end-to-end trajectory generation model.

[0035] Some exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0036] Figure 2 A schematic diagram of an example architecture 200 for a trajectory generation model according to some embodiments of the present disclosure is shown. The following will primarily refer to... Figure 1 The example environment described herein is for illustrative purposes only and is not intended to limit the scope of the invention. Embodiments of this disclosure can also be applied to other types of environments. Furthermore, architecture 200 will be described relative to server 130, but this is merely exemplary. Architecture 200 may be implemented on a remote server of the target object (e.g., server 130), or on the target object's own control device (e.g., control device 120), or may be implemented collaboratively by the remote server and the control device.

[0037] like Figure 2As shown, server 130 can acquire multiple images 201 of the scene in which the movable target object is located. In embodiments of this disclosure, the target object may include various suitable types of movable objects, such as including but not limited to autonomous driving systems (e.g., autonomous vehicles), drones, robots, etc. Such objects are capable of autonomously moving toward a set target in their environment. For example, the target object may be one or more of the first robot 110-1, the second robot 110-2, and the third robot 110-3 described above (collectively or individually referred to as robot 110). Furthermore, for safe and efficient movement, it is necessary to perceive the environment in which the target object is located. Based on the perception results, the next action of the target object can be determined (such as determining the direction of travel, planning a reasonable path, etc.).

[0038] In some embodiments, the scene in which the target object is located can refer to the surrounding environment experienced by the target object within a certain time range, which may include static objects, dynamic objects, etc. Static objects may include, for example, roads, obstacles, buildings, shelves, and other objects that remain still over a relatively long period of time. Dynamic objects may include, for example, people, animals, other movable objects, and other objects that may move in real time.

[0039] During the movement of the target object, information about its surrounding environment can be acquired using image acquisition devices (e.g., cameras, vision sensors) to further construct 3D understanding or trajectory planning. Such image acquisition devices can be placed in the surrounding environment or on the target object. For example, server 130 can acquire multiple images 201 from one or more perspectives captured by a multi-camera system deployed on the target object. These images can be acquired from multiple angles such as forward view, side view, and surround view, covering environmental information surrounding the target object.

[0040] In some embodiments, to support the trajectory generation model's perception of the scene, it is necessary to extract image representations, also known as image features 202, from the acquired images that can express the semantics and spatial features of the scene. Server 130 can determine the image features 202 of the scene based on multiple images 201 acquired for the scene where the movable target object is located. Multiple images 201 are acquired based on the scene where the movable target object is located to obtain visual information related to the current environment. These images typically cover the area in front of, to the sides of, or around the target object, and can reflect the spatial structure of the environment, passageways, and other content.

[0041] In some embodiments, image features 202 can be determined based on multiple images 201 using a pre-trained image encoder 211. The server 110 can input multiple images 201 into the pre-trained image encoder 211. The encoder 211 can be a convolutional neural network, a transformer structure, a learning model incorporating multimodal perception, or any other image encoder structure that performs well in visual tasks. The task of the encoder 211 is to compress and represent the visual information (such as scene semantics, spatial geometry, etc.) contained in the images 201, extracting intermediate image features 202 with high-level abstraction capabilities.

[0042] In some embodiments, server 130 can determine local target points of the target object. Local target points indicate the local target position of the target object during its movement in the scene. In end-to-end trajectory generation tasks, models typically rely on raw input information such as images and states to directly predict the complete trajectory. While this approach possesses overall modeling capabilities, it often struggles to accurately understand and execute high-level task instructions when faced with diverse task objectives or clearly defined control intentions in real-world scenarios, resulting in a lack of controllability in trajectory behavior.

[0043] To address this issue, a local target point (or local waypoint) can be introduced as an intermediate control representation. Local target points explicitly indicate the short-term target position of the object during travel, serving as a spatial representation of the task instruction. By transforming various types of task requirements (e.g., "keep right") into explicit spatial target points (e.g., a location point to the right front), the model can obtain more direct and explicit control objectives, thereby improving the goal orientation and behavioral consistency of trajectory generation.

[0044] Continue to refer to Figure 2 The trajectory generation model 213 can generate a predicted trajectory 206 for a target object based on image features 202 and local target points 205. The trajectory generation model 213 can generate the predicted motion trajectory of the target object based on current environmental perception information and trajectory control information. For example, by using image features 202 and local target points 205 as joint inputs, it can generate the future movement path of the target object in the current environment, i.e., the predicted trajectory 206. Image features 202 provide spatial structure information of the environment (e.g., obstacle positions, passage space, path reachability, etc.). Local target points 205 provide explicit guidance on movement intention. Through these two types of information, the trajectory generation model 213 can take into account both local control intention and environmental perception constraints, outputting a predicted trajectory 206 that is consistent with the direction of the local target points and satisfies environmental constraints.

[0045] In embodiments of this disclosure, the trajectory generation model 212 can adopt any structural form suitable for trajectory modeling or trajectory generation tasks. For example, it can be a recurrent neural network, an encoder-decoder structure, a Transformer-based sequence modeling network, a diffusion model, a flow matching model, or a combination of the above structures.

[0046] Continue to refer to Figure 2 The local target point 205 of the target object can be generated by the target point generation model 212 based on at least one of the target object's state information 203 or task information 204 and image features 202. The state information indicates at least one of the target object's travel speed, current position, and current pose.

[0047] The state information 203 of the target object is used to describe its current motion state. The state information 203 may include, but is not limited to, the following: the spatial position of the target object in the current coordinate system, used to determine its starting point or the context of its current environment; the traveling speed of the target object, which helps the trajectory prediction model infer its reasonable direction of motion and rate changes; and the current posture of the target object (e.g., direction of travel or orientation angle), etc.

[0048] The target object's task information 204 describes the external objectives or constraints that the current trajectory prediction needs to meet. Task information 204 may include at least one of the following: initial constraint information related to the target object's movement in the scene and task description information related to the tasks the target object needs to perform in the scene. Initial constraint information refers to the basic task constraints related to the target object's movement set before the task begins. Initial constraint information may be expressed as a specification of movement mode or path preference.

[0049] Task description information refers to control task instructions received from external sources, used to dynamically adjust the behavior strategy of the target object. This type of information may include at least one of the following: task-specific instructions received from the user or task-specific instructions generated using a machine learning model.

[0050] The target point generation model 212 can determine local target points 205 of the target object during its movement based on any one or more of the aforementioned information, to express its movement trend or behavioral intention, and to provide control reference for subsequent trajectory generation. For example, at the beginning of a task, the target point generation model 212 can generate initial local target points based on preset initial task constraints to respond to the overall task objective. Alternatively or additionally, the target point generation model 212 can generate local target points based solely on the target object's state information in the absence of task information. Even without explicit task instructions, the target point generation model 212 can infer suitable short-term motion targets based on the target object's current position, posture, or speed. For example, if a passable path exists ahead, the system can autonomously generate a local target point ahead to guide the target object to continue moving forward.

[0051] Alternatively or additionally, the target point generation model 212 can receive external local target points as control input. For example, in some scenarios, local target points can be directly provided by an external system. For instance, a user can describe the desired goal using natural language commands, and the external system or an attached module can generate target point location information using machine learning models such as large language models. This information can be encoded into a standardized, general waypoint representation and fed as input to the trajectory generation model 213, replacing the results generated by the target point generation model, thereby achieving more flexible trajectory control.

[0052] In some embodiments, the aforementioned process of generating a predicted trajectory 206 of a target object based on image features 202 and local target points 205 is based on the architecture of trajectory generation model 213, and the local target points 205 are determined using target point generation model 212. This process can be performed during the inference phase or during the training phase of target point generation model 212 and trajectory generation model 213. Some embodiments are described below with reference to the training of target point generation model 212 and trajectory generation model 213.

[0053] In some embodiments, the target point generation model 212 can generate local target points 205 of the target object based on image features 202, the state information 203 of the target object, and task information 204, thereby providing structured intermediate control features for trajectory generation. Further, the trajectory generation model 213 can generate a predicted trajectory 206 of the target object based on image features 202 and local target points 205. The predicted trajectory 206 can be represented as a continuous sequence of location points to describe the movement path of the target object over a future period.

[0054] The process of generating the predicted trajectory 206 has been described above. This process can be implemented during the inference phase or the training phase. When implemented during the inference phase, the input information (e.g., multiple images 201, state information 203, and task information 204) can be information about the target object in actual operation. When implemented during the training phase, the input information can come from the training dataset. Some example implementations regarding model training are described below.

[0055] In some embodiments, a reference trajectory 207 may be introduced as a supervision signal to effectively train the target point generation model 212 and the trajectory generation model 213. The reference trajectory is the ideal trajectory that the target object should take under the current environmental conditions, given a certain state or task intent (e.g., data annotated by human experts, real system operation data, or data extracted from high-precision map navigation paths).

[0056] In some embodiments, server 130 may determine a first loss term based on the difference between the predicted trajectory 206 and the reference trajectory corresponding to the predicted trajectory 206. The reference trajectory corresponding to the predicted trajectory 206 may be a "trace ground truth" obtained under the same input conditions as the predicted trajectory 206 through methods such as human expert annotation, real system operation, and high-precision map extraction. The first loss term can be used to measure the difference between the model-generated predicted trajectory 206 and its corresponding reference trajectory. The calculation of the first loss term may include, for example, calculating the mean squared error between the predicted trajectory 206 and the reference trajectory.

[0057] In some embodiments, server 130 can determine a second loss term based on the difference between local target point 205 and reference target point. When using only trajectory error as the optimization objective, the model primarily learns how to reconstruct the path of the reference trajectory as closely as possible. However, in practical applications, simply reconstructing the trajectory is insufficient to support explicit, diverse, and even dynamically changing control commands. Therefore, explicit supervision of local target point 205, i.e., the second loss term, can be further introduced. This loss term can be measured using methods such as the mean squared error of the position difference.

[0058] In some embodiments, the reference target point can serve as a monitoring signal for the local target point 205, and it can be determined based on a reference trajectory. For example, the reference target point can be determined by sampling reference trajectory points corresponding to the local target point from the reference trajectory. Then, the reference target point is obtained by adding noise to the sampled reference trajectory points.

[0059] As an example, server 130 can sample a reference trajectory point that semantically or temporally corresponds to the current local target point 205 from the reference trajectory. This reference trajectory point typically represents the position that the target object should reach at a certain short-term future time, reflecting the local target trend in the trajectory. This sampling process can be defined based on fixed time intervals, trajectory length proportions, or task stages to ensure its representativeness and stability in the trajectory. Furthermore, to enhance the model's generalization ability and prevent overfitting, the sampled reference trajectory point can be subjected to noise perturbation to generate the final reference target point used for supervision.

[0060] In some embodiments, adding noise to the sampled reference trajectory points may include moving a random amount within a predetermined range from the position indicated by the reference trajectory points. For example, adding a random offset within a predetermined range to the position coordinates of the sampled trajectory points gives the generated reference target points a certain degree of randomness around the real trajectory. Alternatively or additionally, adding noise to the sampled reference trajectory points may include deflecting the orientation indicated by the reference trajectory points by a random angle within a predetermined range. For example, adding a random rotation angle within a predetermined range to the orientation angle of the sampled trajectory points to simulate changes in target orientation under uncertain scenarios. Reference target points generated in this way maintain overall consistency with the reference trajectory while possessing appropriate diversity, helping the model to more robustly learn the spatial representation of control intentions.

[0061] In some embodiments, server 130 can update trajectory generation model 213 and target point generation model 212 based on a first loss term and a second loss term. During training, server 130 can construct a loss function 214 based on a weighted combination of the first and second loss terms. The trajectory generation model 213 and target point generation model 212 can be jointly optimized based on loss function 214.

[0062] As an example, to achieve effective response to local target points during trajectory generation while simultaneously improving the generation quality of those local target points, a joint training strategy of trajectory generation model 213 and target point generation model 212 can be adopted. In each training iteration, the two models jointly participate in forward inference and gradient updates, and jointly optimize the overall system performance through their respective supervision signals. In this way, the trajectory prediction accuracy of trajectory generation model 213 can be improved while enhancing the quality of local target points generated by target point generation model 212. Furthermore, the trained trajectory generation model 213 supports a pluggable structure. Even if the target point generation model 212 is not included in the inference (generation) stage, and an external system is used to generate local target points, trajectory generation model 213 can still maintain good responsiveness to target points.

[0063] As can be more clearly understood from the description of the various embodiments of this disclosure, by introducing local target points as intermediate control signals, the responsiveness of the trajectory to task intent and the controllability of behavior are significantly enhanced while maintaining the advantages of the end-to-end trajectory generation architecture. Local target points, acting as a bridge between task information, state information, and environmental perception, provide a unified, explicit, and structured control representation, enabling the trajectory generation process to make autonomous decisions and flexibly follow external instructions. Based on this intermediate representation, controllable trajectory reasoning, collaborative modeling, and joint optimization can be achieved. While improving task adaptability, trajectory accuracy, and training stability, it also possesses good scalability and generalization capabilities, making it suitable for path planning and behavior control tasks in various dynamic environments.

[0064] Figure 3 A flowchart of a process 300 for trajectory generation according to some embodiments of the present disclosure is shown. Process 300 may be executed by a remote server of the target object (e.g., server 130), or by a control device of the target object itself (e.g., control device 120), or may be performed collaboratively by the remote server and the control device. Reference is made below. Figure 1 Describe the process 300.

[0065] In box 310, server 130 determines image features of the scene based on multiple images of the scene in which the movable target object is located.

[0066] In box 320, server 130 determines local target points of the target object, which indicate the local target position of the target object during its movement in the scene.

[0067] In box 330, server 130 uses a trajectory generation model to generate a predicted trajectory of the target object based on image features and local target points.

[0068] In some embodiments, determining local target points of a target object includes: generating local target points of the target object using a target point generation model based on at least one of the target object's state information or task information and image features.

[0069] In some embodiments, the status information indicates at least one of the target object's travel speed, current position, and current attitude.

[0070] In some embodiments, the task information includes one of the following: initial constraint information related to the movement of the target object in the scene, or task description information related to the task to be performed by the target object in the scene.

[0071] In some embodiments, the task description information includes at least one of the following: instructions for the task received from the user, or instructions for the task generated using a machine learning model.

[0072] In some embodiments, local target points are determined using a target point generation model. Process 400 is performed during the training of the trajectory generation model and the target point generation model, which are trained by: determining a first loss term based on the difference between a predicted trajectory and a reference trajectory corresponding to the predicted trajectory; determining a second loss term based on the difference between a local target point and a reference target point, the reference target point being determined based on the reference trajectory; and updating the trajectory generation model and the target point generation model based on the first and second loss terms.

[0073] In some embodiments, the reference target point is determined by sampling reference trajectory points corresponding to the local target point from the reference trajectory; and by adding noise to the sampled reference trajectory points to obtain the reference target point.

[0074] In some embodiments, adding noise to the sampled reference trajectory point includes at least one of the following: moving a random amount within a predetermined range from the position indicated by the reference trajectory point, or deflecting the orientation indicated by the reference trajectory point by a random angle within a predetermined range.

[0075] In some embodiments, the multiple images include images of a scene captured from multiple angles.

[0076] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an apparatus 400 for predictive trajectory generation according to some embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in server 130. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0077] refer to Figure 4 As shown, the device 400 includes an image feature determination module 410, configured to determine image features of the scene based on multiple images of the scene in which the movable target object is located; a local target point determination module 420, configured to determine local target points of the target object, the local target points indicating the local target position of the target object during its movement in the scene; and a predicted trajectory generation module 430, configured to generate a predicted trajectory of the target object based on the image features and the local target points using a trajectory generation model.

[0078] In some embodiments, the local target point determination module 420 is further configured to generate local target points of the target object using a target point generation model based on at least one of the target object's state information or task information and image features.

[0079] In some embodiments, the status information indicates at least one of the target object's travel speed, current position, and current attitude.

[0080] In some embodiments, the task information includes one of the following: initial constraint information related to the movement of the target object in the scene, or task description information related to the task to be performed by the target object in the scene.

[0081] In some embodiments, the task description information includes at least one of the following: instructions for the task received from the user, or instructions for the task generated using a machine learning model.

[0082] In some embodiments, local target points are determined using a target point generation model. The apparatus 400 further includes a training module configured to: determine a first loss term based on the difference between a predicted trajectory and a reference trajectory corresponding to the predicted trajectory; determine a second loss term based on the difference between local target points and reference target points, the reference target points being determined based on the reference trajectory; and update the trajectory generation model and the target point generation model based on the first and second loss terms.

[0083] In some embodiments, the reference target point is determined by sampling reference trajectory points corresponding to the local target point from the reference trajectory; and by adding noise to the sampled reference trajectory points to obtain the reference target point.

[0084] In some embodiments, adding noise to the sampled reference trajectory point includes at least one of the following: moving a random amount within a predetermined range from the position indicated by the reference trajectory point, or deflecting the orientation indicated by the reference trajectory point by a random angle within a predetermined range.

[0085] In some embodiments, the multiple images include images of a scene captured from multiple angles.

[0086] Figure 5 A block diagram is shown of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented. The electronic device 500 may, for example, be used to implement... Figure 1 The control device 120 shown. It should be understood that, Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein.

[0087] refer to Figure 5Electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0088] Electronic device 500 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data and accessible within electronic device 500.

[0089] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0090] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0091] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0092] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0093] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0095] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0097] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is determined to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for trajectory generation, comprising: Based on multiple images of the scene in which the movable target object is located, determine the image features of the scene; Determine local target points of the target object, wherein the local target points indicate the local target position of the target object during its movement in the scene; and Using a trajectory generation model, a predicted trajectory of the target object is generated based on the image features and the local target points.

2. The method according to claim 1, wherein determining the local target points of the target object comprises: Based on at least one of the state information or task information of the target object and the image features, local target points of the target object are generated using a target point generation model.

3. The method according to claim 2, wherein the state information indicates at least one of the target object's travel speed, current position, and current attitude.

4. The method according to claim 2, wherein the task information includes one of the following: Initial constraint information related to the movement of the target object in the scene, or Task description information related to the task that the target object is to perform in the scene.

5. The method according to claim 4, wherein the task description information includes at least one of the following: Instructions received from the user for the task, or Instructions for the task generated using a machine learning model.

6. The method of claim 1, wherein the local target point is determined using a target point generation model, the method being performed during the training of the trajectory generation model and the target point generation model, and the trajectory generation model and the target point generation model being trained in the following manner: Based on the difference between the predicted trajectory and the reference trajectory corresponding to the predicted trajectory, a first loss term is determined; Based on the difference between the local target point and the reference target point, a second loss term is determined, wherein the reference target point is determined based on the reference trajectory; and Based on the first loss term and the second loss term, the trajectory generation model and the target point generation model are updated.

7. The method of claim 6, wherein the reference target point is determined by: Sample reference trajectory points corresponding to the local target point from the reference trajectory; and The reference target point is obtained by adding noise to the sampled reference trajectory points.

8. The method of claim 7, wherein adding noise to the sampled reference trajectory points comprises at least one of the following: Move a random amount within a predetermined range from the position indicated by the reference trajectory point, or The orientation indicated by the reference trajectory point is deflected by a random angle within a predetermined range.

9. The method of claim 1, wherein the plurality of images comprises images of the scene acquired from multiple angles.

10. An apparatus for trajectory generation, comprising: The image feature determination module is configured to determine image features of the scene based on multiple images of the scene in which the movable target object is located; A local target point determination module is configured to determine local target points of the target object, wherein the local target points indicate the local target position of the target object during its movement in the scene; and The predicted trajectory generation module is configured to generate a predicted trajectory of the target object based on the image features and the local target points using a trajectory generation model.

11. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

12. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images