Method and apparatus for autonomous driving trajectory generation

Abstract

A method for generating an autonomous driving trajectory according to one embodiment includes the operation of generating a scene token and a first trajectory based on collected images, the operation of obtaining a first query command - the first query command includes a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory - and the operation of generating a third trajectory based on the scene token, the first trajectory, and the first query command using a large-scale language model, wherein the third trajectory is obtained by processing a second trajectory based on second information obtained in response to the second query, and the second trajectory is obtained based on first information obtained in response to the first query, and the first information and the second information can be obtained based on the scene token, the first trajectory, and the first query command.

Description

Technology Field

[0001] The following disclosure relates to a method and apparatus for generating an autonomous driving trajectory using a large-scale language model. Background Technology

[0002] Autonomous driving refers to the ability of a vehicle to drive itself without human intervention. Autonomous driving technology is already being applied in various fields, including personal vehicles, urban public electric buses, and taxis. With the continuous advancement of technology, autonomous driving technology is expected to be utilized and developed more extensively in the future.

[0003] Ensuring safety, explainability, and / or generalizability can be critical to the advancement of autonomous driving systems. While traditional autonomous driving systems adopt modular designs and built-in error detection mechanisms, they may suffer from reduced performance and limited generalization capabilities when handling complex situations. End-to-end autonomous driving systems can improve performance in complex scenarios by directly optimizing driving strategies through a data-driven approach. Brief explanation of the drawing

[0005] FIGS. 1a and FIGS. 1b are flowcharts illustrating a method for generating an autonomous driving trajectory according to one embodiment. FIG. 2 is a schematic block diagram for explaining an autonomous driving system according to one embodiment. FIG. 3 is an exemplary block diagram for illustrating a query command in an autonomous driving system according to one embodiment. FIG. 4a is a schematic diagram illustrating an example of a scene tokenizer according to one embodiment. FIG. 4b is a schematic flowchart illustrating the operation of a scene tokenizer according to one embodiment. FIGS. 5a to 5c are schematic diagrams illustrating the results obtained through an autonomous driving trajectory generation method according to one embodiment. FIG. 6 is an example diagram of the structure of an electronic device provided in an embodiment of the present invention. Specific details for implementing the invention

[0006] A method for generating an autonomous driving trajectory according to one embodiment includes the operation of generating a scene token and a first trajectory based on collected images, the operation of obtaining a first query command - the first query command includes a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory - and the operation of generating a third trajectory based on the scene token, the first trajectory, and the first query command using a large-scale language model, wherein the third trajectory is obtained by processing a second trajectory based on second information obtained in response to the second query, and the second trajectory is obtained based on first information obtained in response to the first query, and the first information and the second information may be obtained based on the scene token, the first trajectory, and the first query command.

[0007] The second information may include at least one of information regarding whether the direction of the second trajectory corresponds to the direction of the navigation command or information regarding whether the yaw angle between each pair of adjacent trajectory points in the second trajectory exceeds a first threshold value.

[0008] The operation of generating the third trajectory may include, based on the second information, if the direction of the second trajectory does not match the direction of the navigation command, adjusting the vertical coordinate value of a trajectory point within the second trajectory to make the direction of the trajectory correspond to the direction of the navigation command, and determining the third trajectory corresponding to the adjusted second trajectory.

[0009] Based on the second information above, if the yaw angle between at least one pair of adjacent trajectory points within the second trajectory exceeds a first threshold value, the horizontal coordinate value of the trajectory point within the second trajectory is gradually increased and the vertical coordinate value is adjusted so that the fluctuation range is smaller than the second threshold value, thereby determining the third trajectory corresponding to the adjusted second trajectory.

[0010] The operation of generating the third trajectory may include an operation of generating explanatory information regarding the processing process of the second trajectory based on the second information, and an operation of processing the second trajectory based on the explanatory information to obtain the third trajectory.

[0011] The operation of generating a third trajectory may include the operation of generating the first information and the second information based on the scene token, the first trajectory, and the first query command; the operation of generating a second trajectory regarding the first information based on the scene token, the first trajectory, the first query command, and the first information; and the operation of processing the second trajectory based on the second information to obtain a third trajectory.

[0012] The first information may include at least one of information related to a driving scene, information related to a key object in the driving scene, information related to a driving action to be performed, or information related to the prediction of a second trajectory.

[0013] The operation of generating a third trajectory may include the operation of embedding the first trajectory into the first query command to obtain a second query command, and the operation of generating a third trajectory based on the scene token and the second query command.

[0014] The operation of generating a scene token and a first trajectory based on the collected image may include the operation of extracting visual features from the image, the operation of obtaining a scene query representation—the scene query representation represents information related to the semantics of a driving scene—and a perception query representation—the perception query representation represents information related to at least one object in the driving scene—the operation of updating the scene query representation and the perception query representation based on the visual features, the operation of generating the scene token based on the updated perception query representation and the updated scene query representation, and the operation of generating the first trajectory based on the updated perception query representation.

[0015] The operation of updating the scene query expression and the detection query expression based on the above visual features may include the operation of acquiring a perception memory—the perception memory includes a detection query expression updated in the past and / or a scene query expression updated in the past—and the operation of updating the scene query expression and the detection query expression based on the above visual features and the perception memory.

[0016] An electronic device according to one embodiment comprises at least one processor including a processing circuit and a memory for storing instructions, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device generates a scene token and a first trajectory based on a collected image, obtains a first query command—the first query command includes a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory—and generates a third trajectory based on the scene token, the first trajectory, and the first query command using a large-scale language model, wherein the third trajectory is obtained by processing a second trajectory based on second information obtained in response to the second query, and the second trajectory is obtained based on first information obtained in response to the first query, and the first information and the second information are based on the scene token, the first trajectory, and the first query command It can be obtained.

[0017] The second information may include at least one of information regarding whether the direction of the second trajectory corresponds to the direction of the navigation command or information regarding whether the yaw angle between each pair of adjacent trajectory points in the second trajectory exceeds a first threshold value.

[0018] When the above instructions are executed individually or collectively by at least one processor, the electronic device may be configured to adjust the vertical coordinate values ​​of a trajectory point within the second trajectory based on the second information so that, if the direction of the second trajectory does not match the direction of the navigation command, the direction of the trajectory corresponds to the direction of the navigation command, and to determine the third trajectory corresponding to the adjusted second trajectory.

[0019] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may determine the third trajectory corresponding to the adjusted second trajectory by gradually increasing the horizontal coordinate value of the trajectory point within the second trajectory and adjusting the fluctuation range of the vertical coordinate value to be smaller than the second threshold value, based on the second information, when the yaw angle between at least one pair of adjacent trajectory points within the second trajectory exceeds the first threshold value.

[0020] When the above instructions are executed individually or collectively by at least one processor, the electronic device may generate explanatory information regarding the processing of the second trajectory based on the second information, and process the second trajectory based on the explanatory information to obtain the third trajectory.

[0021] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may generate the first information and the second information based on the scene token, the first trajectory, and the first query command, generate a second trajectory regarding the first information based on the scene token, the first trajectory, the first query command, and the first information, and process the second trajectory based on the second information to obtain a third trajectory.

[0022] The first information may include at least one of information related to a driving scene, information related to a key object in the driving scene, information related to a driving action to be performed, or information related to the prediction of a second trajectory.

[0023] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to embed the first trajectory into the first query command to obtain a second query command and generate a third trajectory based on the scene token and the second query command.

[0024] When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be configured to extract visual features from the image, obtain a scene query representation—the scene query representation represents information related to the semantics of the driving scene—and a perception query representation—the perception query representation represents information related to at least one object in the driving scene—update the scene query representation and the perception query representation based on the visual features, generate the scene token based on the updated perception query representation and the updated scene query representation, and generate the first trajectory based on the updated perception query representation.

[0025] In a recording medium storing instructions executable on a hardware device according to one embodiment, the instructions are characterized by being configured to generate a scene token and a first trajectory based on a collected image, obtain a first query command—the first query command includes a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory—and generate a third trajectory based on the scene token, the first trajectory, and the first query command using a large-scale language model, wherein the third trajectory is obtained by processing a second trajectory based on second information obtained in response to the second query, and the second trajectory is obtained based on first information obtained in response to the first query, and the first information and the second information may be obtained based on the scene token, the first trajectory, and the first query command.

[0027] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Accordingly, actual implementations are not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or substitutions included in the technical concept described by the embodiments.

[0028] Terms such as "first" or "second" may be used to describe various components, but these terms should be interpreted solely for the purpose of distinguishing one component from another. For example, the first component may be named the second component, and similarly, the second component may be named the first component.

[0029] When it is stated that a component is "connected" to another component, it should be understood that it may be directly connected to or coupled with that other component, or that there may be other components in between.

[0030] The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, terms such as "comprising" or "having" are intended to specify the existence of the described features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0031] In this document, each of the phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0032] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this specification.

[0033] The embodiments can be implemented in various forms of products, such as personal computers, laptop computers, tablet computers, smartphones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices. The embodiments will be described in detail below with reference to the attached drawings. In the description with reference to the attached drawings, identical components are given the same reference numeral regardless of the drawing number, and redundant descriptions thereof will be omitted.

[0035] At least some functions of the device or electronic device provided in the embodiments may be implemented through an AI model. For example, at least one module among several modules of the device or electronic device may be implemented through an AI model. Functions related to AI may be performed by non-volatile memory, volatile memory, and a processor.

[0036] The processor may include one or more processors. In this case, the one or more processors may be a general-purpose processor (e.g., a central processing unit (CPU), an application processor (AP), etc.) or a pure graphics processing unit (e.g., a graphics processing unit (GPU), a visual processing unit (VPU)), and / or an AI-dedicated processor (e.g., a neural processing unit (NPU)).

[0037] One or more processors control the processing of input data according to predefined behavior rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. Predefined behavior rules or AI models are provided through training or learning.

[0038] Here, provision by learning may mean obtaining an AI model having predefined behavior rules or desired features by applying a learning algorithm to multiple learning data. Such learning may be performed on the device or electronic device itself where the AI ​​according to the embodiment is performed, and / or may be implemented by a separate server / device / system.

[0039] An AI model may consist of multiple neural network layers. Each layer performs a neural network computation by calculating between the input data of that layer (e.g., the computation result of the previous layer and / or the input data of the AI ​​model) and multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, Convolutional Neural Networks (CNN), Deep Neural Networks (DNN), Recurrent Neural Networks (RNN), Restricted Boltzmann Machines (RBM), Deep Belief Networks (DBN), Bidirectional Recurrent Deep Neural Networks (BRDNN), Generative Correspondence Networks (GAN), and Deep Q Networks.

[0040] A learning algorithm is a method of training a specific target device (e.g., a robot) using multiple learning data to induce, allow, or control the target device to determine or predict. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0041] The method provided in the embodiments may be related to one or more of the technology fields such as voice, language, image, video, or data intelligence.

[0042] According to one embodiment, optionally, in the case of voice or language, a method performed on an electronic device for recognizing user voice and interpreting user intent may receive a voice signal as an analog signal through a voice acquisition device (e.g., a microphone) and convert a portion of the voice into computer-readable text using an Automatic Speech Recognition (ASR) model. The user's utterance intent may be obtained by interpreting the converted text using a Natural Language Understanding (NLU) model. The ASR model or the NLU model may be an AI model. The AI ​​model may be processed by an AI-dedicated processor designed with a hardware architecture specified for AI model processing. The AI ​​model may be obtained through training. Here, "obtained through training" means training a base AI model with multiple training data using a training algorithm to obtain predefined operational rules or an AI model of desired features (or objectives). Language understanding is a technology used to recognize, apply, or process human language / text, including natural language processing, machine translation, conversational systems, question answering, or speech recognition / synthesis.

[0043] According to one embodiment, optionally, in the field of images or videos, a method for identifying a trajectory among methods performed on an electronic device may use image data as input data for an AI model to obtain output data for identifying various features, knowledge, information, etc. within an image or image. The AI ​​model may be obtained through training. Here, "obtained through training" means training a basic AI model with various training data through a training algorithm to obtain a predefined operating rule or AI model of a desired feature (or purpose). The method of the present disclosure may relate to the field of visual understanding of artificial intelligence technology, which is a technology for recognizing and processing objects such as human vision, and may include, for example, object recognition, object tracking, image search, person recognition, scene recognition, 3D reconstruction / positioning, or image enhancement.

[0044] According to one embodiment, optionally, in the field of data intelligence processing, a method for inferring or predicting a trajectory by utilizing various knowledge using an AI model can be recommended or performed among methods performed on an electronic device. The processor of the electronic device can perform preprocessing on data to convert it into a form suitable for use as input to an AI model. An AI model can be obtained through training. Here, "obtained through training" means training a basic AI model with various training data through a training algorithm to obtain a predefined operational rule or AI model of a desired feature (or objective). Inference prediction is a technique that performs logical inference and prediction by determining information such as knowledge-based inference, optimization prediction, preference-based planning, or recommendation.

[0045] Hereinafter, the technical solution of the embodiment of the present disclosure and the technical effect generated by the technical solution of the present disclosure are explained through the description of various optional embodiments. It should be noted that the following embodiments may be referenced, borrowed, or combined with one another. Identical terms, similar features, and similar implementation steps, etc., are not described repeatedly in different embodiments.

[0047] FIGS. 1a and FIGS. 1b are flowcharts illustrating a method for generating an autonomous driving trajectory according to one embodiment.

[0048] For convenience of explanation, the operations (101 to 103 and 1031 to 1033) are described as being performed using the electronic device (600) illustrated in FIG. 6. However, these operations (101 to 103 and 1031 to 1033) may be used through any other suitable electronic device and within any suitable system. Additionally, the electronic device may be a vehicle including an autonomous driving system.

[0049] Furthermore, the operations of FIGS. 1a and 1b may be performed in the order and manner depicted, but the order of some operations may be changed or some operations may be omitted without departing from the spirit and scope of the depicted embodiments. A number of operations depicted in FIGS. 1a and 1b may be performed in parallel or simultaneously.

[0050] In operation 101, the electronic device (600) can generate a scene token and a first trajectory based on the collected image.

[0051] According to one embodiment, the collected images may be multi-view images. For example, a vehicle-mounted collection device may be installed on an autonomous vehicle, and may be, for example, a vehicle-mounted camera, a vehicle-mounted video camera, etc., but is not limited thereto. The number of vehicle-mounted collection devices may be one or more, thereby enabling the collection of images of the environment at different angles around the autonomous vehicle during driving. Those skilled in the art may set the location where the vehicle-mounted collection device is installed within the autonomous vehicle according to actual circumstances, and the embodiments of the present disclosure are not limited thereto.

[0052] According to one embodiment, optionally, the number of multi-view images may be one frame or multiple frames, and multiple frames of multi-view images may also be referred to as multi-view video.

[0053] According to one embodiment, optionally, the collected image may be an RGB (Red, Green, Blue) image, but is not limited thereto, and may be an image of another color mode.

[0054] According to one embodiment, the electronic device (600) can generate scene tokens and a first trajectory based on collected images. For example, the scene tokens may represent information within the driving scene, semantics of the driving scene, objects, relationships between objects, etc.

[0055] According to one embodiment, the scene token adopts a 3D scene token to represent information of a 3D scene and can be expressed by a predetermined S×C tensor, where S represents the number of scene tokens and C represents the number of feature channels. The first trajectory can also be referred to as an initial trajectory proposal and can be expressed by a predetermined T×2 tensor. T represents the number of trajectory points within a predetermined time in the predicted future, and 2 represents the 2D coordinates of each trajectory point. For example, T can be 6 and the predetermined time can be 3 seconds, which represents 6 trajectory points at 0.5-second intervals within the next 3 seconds.

[0056] According to one embodiment, the prediction of the first trajectory enables the model to directly analyze and subdivide specific trajectories, thereby alleviating challenges related to uncertain trajectory prediction and improving the overall decision-making process.

[0057] In operation 102, the electronic device (600) may obtain a first query command. The first query command may include a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory.

[0058] According to one embodiment, a query command may be descriptive text, a question, etc., regarding specific information within an image, and may be used to determine a relevant interpretation of the predicted trajectory. The query command may also be referred to as query text. The first query command may be a predetermined text, or it may be obtained by converting a predetermined text, and the described embodiments are not limited thereto.

[0059] According to one embodiment, a first query regarding the logic of trajectory generation can be used to determine knowledge related to trajectory generation to improve the explainability of trajectory generation. A second query regarding the safety of the trajectory can be used to determine knowledge related to safety to improve the safety of the trajectory.

[0060] In operation 103, the electronic device (600) can generate a third trajectory based on a scene token, a first trajectory, and a first query command using a large-scale language model. The third trajectory is obtained by processing a second trajectory based on second information in response to a second query, and the second trajectory is obtained based on first information in response to a first query, and the first information and the second information can be obtained based on a scene token, a first trajectory, and a first query command.

[0061] According to one embodiment, the large-scale language model may adopt an LLM (Large Language Model), a VLM (Visual Language Model), etc., but the described embodiments are not limited thereto. The LLM model can be applied to the field of autonomous driving to improve the reliability of trajectory prediction.

[0062] An electronic device (600) according to one embodiment may input a scene token, a first trajectory, and a first query command into a large-scale language model. The large-scale language model may output a generated third trajectory. The first information may be understood as knowledge related to the logic of trajectory generation extracted from the input during the processing of the large-scale language model, and the second trajectory is predicted based on the first information, which can improve trajectory prediction performance, for example, in complex scenes. The second trajectory may also be represented by a predetermined T×2 tensor. T represents the number of trajectory points within a predetermined time in the predicted future, and 2 may represent the two-dimensional coordinates of each trajectory point. The second information may be understood as knowledge related to the safety of the trajectory extracted from the input during the processing of the large-scale language model, and the third trajectory is obtained by processing the second trajectory based on the second information, which can contribute to the improvement of trajectory safety.

[0063] According to one embodiment, the first information and the second information may be integrated and utilized to improve the generalization ability of the autonomous driving system. The first information and / or the second information may also be output by the model, thereby improving the explainability of the model.

[0064] According to one embodiment, operation 103 may specifically include the following operations:

[0065] In operation 1031, the electronic device (600) can generate first information and second information using a large-scale language model based on a scene token, a first trajectory, and a first query command.

[0066] In operation 1032, the electronic device (600) can generate a second trajectory associated with the first information using a large-scale language model based on a scene token, a first trajectory, a first query command, and the first information.

[0067] In operation 1033, the electronic device (600) can obtain a third trajectory by processing the second trajectory using a large-scale language model based on the second information.

[0068] Accordingly, the autonomous driving trajectory generation method of the described embodiments can effectively combine knowledge related to the logic of trajectory generation and knowledge related to the safety of the trajectory, thereby simultaneously improving safety, explainability, and generalizability.

[0069] According to one embodiment, the second information may include at least one of i) information regarding whether the direction of the second trajectory matches the direction of the navigation command (hereinafter referred to as direction matching information) or ii) information regarding whether the yaw angle between adjacent trajectory points of each pair within the second trajectory exceeds a first threshold value (hereinafter referred to as yaw angle information).

[0070] According to one embodiment, direction matching information may primarily consider whether the trajectory violates a navigation command. A method for determining whether the trajectory violates a navigation command may first convert the trajectory into a direction action and then compare whether the direction action of the trajectory matches the navigation command. If it does not match, it may be determined that the trajectory violates the navigation command.

[0071] According to one embodiment, a third trajectory can be obtained by processing the second trajectory based on the second information. The processing method may be as follows: If the direction of the second trajectory does not match the direction of the navigation command, the electronic device (600) may adjust the vertical coordinate (y-axis) value of a trajectory point within the second trajectory to match the direction of the trajectory to the direction of the navigation command, and the adjusted second trajectory may be the third trajectory. An optimization method corresponding to the case where the trajectory violates the navigation command may be to adjust the y-value of the trajectory point to match the direction of the trajectory to the navigation command.

[0072] According to one embodiment, the yaw angle information may primarily consider whether the trajectory is not smooth. A method for determining whether the trajectory is not smooth may be to check whether the delta yaw between points within the trajectory exceeds a certain threshold (e.g., a first threshold). For example, if the trajectory contains six points, five yaw angles can be calculated between adjacent points of each pair, and it can be determined whether the five yaw angles exceed the first threshold. If all five yaw angles are less than or equal to the first threshold, the trajectory is determined to be smooth; if even a single yaw angle exceeds the first threshold, the trajectory is determined to be not smooth, which may result in sudden steering of the vehicle or deviation into an undriving zone. A person skilled in the art may set the first threshold according to actual conditions, for example, to 1.1, and the described embodiments are not limited thereto.

[0073] According to one embodiment, a third trajectory is obtained by processing the second trajectory based on the second information, and the processing method may be as follows: when the yaw angle between at least one pair of adjacent trajectory points within the second trajectory exceeds a first threshold, the electronic device (600) gradually increases the horizontal coordinate (x-axis) value of the trajectory point within the second trajectory and adjusts the fluctuation range of the vertical coordinate (y-axis) value so that it is smaller than the second threshold (which may be understood as preventing the vertical coordinate value from shaking, or directly instructing a large-scale language model to prevent the vertical coordinate value from shaking, and allowing the large-scale language model to process based on that understanding), so that the adjusted second trajectory can be used as the third trajectory.

[0074] According to one embodiment, in the case where the trajectory is not smooth, the corresponding optimization method may be as follows: The electronic device (600) can improve the smoothness of the trajectory by ensuring that the x-value of the trajectory gradually increases and that the y-value of the trajectory point does not waver (especially when the speed of the autonomous vehicle is slow).

[0075] According to one embodiment, if neither of the two situations described above occurs, the trajectory may be considered safe.

[0076] In the case where the trajectory is safe, the electronic device (600) may directly output the second trajectory to use as the third trajectory, or may output the second trajectory as the third trajectory based on internal processing of a large-scale language model (with some fine-tuning). That is, the second trajectory and the third trajectory may be the same or similar.

[0077] According to one embodiment, when the electronic device (600) processes a second trajectory based on second information to obtain a third trajectory, it may generate explanatory information regarding the processing process of the second trajectory based on the second information. Additionally, the electronic device (600) may process the second trajectory based on the explanatory information to obtain a third trajectory.

[0078] According to one embodiment, the step of processing the second trajectory based on the second information to obtain the third trajectory may also be called a reflection step, and the content of the reflection may be described as follows: the electronic device (600) may simulate and evaluate the safety of the given trajectory, describe an optimization method, and output the optimized trajectory.

[0079] According to one embodiment, the second information, explanatory information, etc., can all be expressed in a language / speech manner.

[0080] According to one embodiment, the described embodiments provide a reflection template, which can be expressed as follows: where the trajectory is <input each trajectory point>, the trajectory can be determined by <description of safety and specific situation>. Accordingly, an <optimization method> must be performed, and the planned trajectory can be <optimized each trajectory point>.

[0081] According to one embodiment, each input trajectory point corresponds to a second trajectory, and each optimized trajectory point may correspond to a third trajectory. Alternatively, for example, it may be expressed in another language as follows: If follow<input_traj> ,<safety_and_outcome> . Hence you should<how_to_optimize> , here is the planning trajectory<optimized_traj> .

[0082] For example, as shown in Table 1 below, an example of one reflection can be observed.

[0083] safety When the trajectory is [(+1.31, -0.01), (+2.77, -0.04), (+4.35, -0.04), (+6.00, -0.04), (+7.74, -0.05), (+9.63, -0.06)], the trajectory is confirmed to be safe and feasible. Therefore, it is sufficient to follow this basically or make minor adjustments, and the planned trajectory is [PT, (+1.70, 0.0), (+2.84, -0.02), (+4.06, -0.03), (+5.26, 0.0), (+6.52, +0.02), (+7.95, +0.08)]. Navigation command violation When the trajectory is [(+0.73, -0.01), (+1.73, +0.05), (+3.05, +0.18), (+4.56, +0.50), (+6.15, +1.15), (+7.86, 1.81)], the future movement of the trajectory is confirmed to be straight, which violates the left turn navigation command. Therefore, the direction must be adjusted to the left by increasing the y-axis value of the trajectory, and the planned trajectory is [PT, (+0.63, 0.0), (+1.67, +0.08), (+2.94, +0.34), (+4.36, +0.93), (+5.66, +1.89), (+6.93, +3.41)]. Trajectory unevenness When the trajectory is [(+1.19, -0.02), (+1.97, -0.03), (+2.42, -0.04), (+2.60, -0.05), (+2.61, -0.06), (+2.60, -0.06)], it is confirmed that the driving direction of the trajectory is discontinuous in time. Therefore, it must be ensured that the x-axis value of the trajectory does not decrease, and next, that the trajectory does not sway horizontally when the vehicle speed is slow. The planned trajectory is [PT, (+1.22, -0.02), (+2.08, -0.03), (+2.56, -0.04), (+2.77, -0.05), (+2.84, -0.05), (+2.88, -0.06)].

[0084] In an embodiment of the present disclosure, the first information may include at least one of i) information related to a driving scene, ii) information related to a key object within the driving scene, iii) information related to a driving action to be performed, or iv) information related to the prediction of a second trajectory. According to one embodiment, the information related to the driving scene may be understood as information or a response describing the current driving scene. The information related to the driving scene may be referred to as scene description information or a response. The information related to the driving scene includes road conditions (e.g., parking lot, intersection), traffic elements (e.g., pedestrian, vehicle, traffic light), time and weather conditions, and may include, for example, what the weather is like in the scene, whether pedestrians and vehicles are present in the scene, etc.

[0085] According to one embodiment, scene description information can be expressed in a language / speech manner.

[0086] According to one embodiment, the electronic device (600) can recognize and determine surrounding information using 3D point cloud data.

[0087] According to one embodiment, information related to a key object within a driving scene can be understood as a response related to a key object requiring attention within the driving scene. Information related to a key object within a driving scene can be referred to as attention or interest information or a response. Information related to a key object within a driving scene is used to identify key targets that may influence driving behavior, and may include, for example, the location of a vehicle or pedestrian requiring attention due to being in close proximity to the vehicle.

[0088] According to one embodiment, information of interest can be expressed in a language / speech manner.

[0089] According to one embodiment, information related to a driving action to be performed can be understood as a response related to an action to be performed by the vehicle in the future. Information related to a driving action to be performed can be referred to as decision information or a response. For example, direction actions may include going straight, turning left, turning right, stopping, etc., and speed actions may include acceleration, deceleration, a constant speed, stopping, etc.

[0090] According to one embodiment, the driving action can be selected from a predefined selection option.

[0091] According to one embodiment, the decision information may also include the reason for the decision, and, for example, the reason for the decision may be explained by predicting and analyzing the speed of the next 3 seconds based on the current scene and key target.

[0092] According to one embodiment, decision information can be expressed in a language / speech manner.

[0093] According to one embodiment, information related to the prediction of a second trajectory can be understood as a response related to the predicted trajectory. Information related to the driving action to be performed can be referred to as trajectory information or a response. This may predict a partial description of the trajectory and may also include point coordinate information of a specific trajectory. For example, a safe and feasible 3-second trajectory can be predicted based on the driving action and the first trajectory, and this may include six waypoints (or trajectory points). That is, point coordinate information of the predicted second trajectory may be included in the trajectory information.

[0094] Likewise, the query command corresponding to the trajectory information may also include point coordinate information of the first trajectory. Specifically, when the electronic device (600) generates the third trajectory, it may embed the first trajectory into the first query command to obtain the second query command, and generate the third trajectory based on the scene token and the second query command.

[0095] According to one embodiment, ii) information related to key objects within a driving scene, iii) information related to driving actions to be performed, and iv) information related to the prediction of a second trajectory can be trained and predicted based on labeling within a dataset.

[0097] FIG. 2 is a schematic block diagram for explaining an autonomous driving system according to one embodiment.

[0098] The description with reference to FIGS. 1a and FIGS. 1b may be applied in the same way to FIGS. 2, and duplicate content may be omitted.

[0099] One or more blocks and combinations of blocks in Fig. 2 may be implemented by a computer based on special-purpose hardware that performs specific functions, or by a combination of special-purpose hardware and computer instructions.

[0100] Based on at least one embodiment above, an embodiment of the present disclosure may be an example of autonomous driving trajectory planning. As illustrated in FIG. 2, an embodiment of the present disclosure may be an end-to-end autonomous driving system based on an LLM (220) (or VLM). The autonomous driving system may integrate inference, planning, and reflection. Specifically, through a scene tokenizer (210), all relevant scene information based on sensor data may be encoded to generate scene tokens and initial trajectory proposals (e.g., a first trajectory). Subsequently, a series of query commands Q including scene descriptions, key object recognition, action selection, trajectory planning, safety reflections, etc. may be introduced to represent prediction logic, specific trajectory information, safety evaluation, and knowledge regarding the optimized trajectory. Subsequently, corresponding knowledge may be extracted from the scene by learning the relationships between the query commands, trajectory proposals, and scene tokens using pre-trained parameters of a large-scale language model. Finally, the network can directly output inferences related to the trajectory, predicted trajectory, and safety assessments, and provide an optimized trajectory.

[0101] Referring to FIG. 2, the autonomous driving system receives sensor data through a scene tokenizer (210) and generates scene tokens and initial trajectory proposals that imply semantic information within the scene. The generated scene tokens and trajectory proposals can be input into a large-scale language model (LLM (220)).

[0102] LLM (220) can sequentially respond to a series of queries (Q) including the meaning of the driving scene, key objects, actions performed, trajectory planning and safety evaluation, and process the necessary information at each stage in the form of inference, planning, and reflection (verification).

[0103] For example, a query (Q) may be given in the form of “describe the structure of the current scene,” “identify objects requiring attention,” “determine future driving actions,” “is the trajectory safe,” and the LLM (220) may output a response (A) to this as a numerical trajectory along with a verbal description.

[0104] In the inference phase of the LLM (220), driving logic is understood from scene tokens, a predicted trajectory is generated in the planning phase, and safety verification and correction are performed in the reflection phase to derive the final planned trajectory.

[0105] Through this structure, the system can calculate an optimized trajectory step by step while maintaining the explainability of the driving situation. In addition, the LLM (220) can make consistent decisions that reflect the basis for judgment in the previous step by accumulating and referencing the response of each step.

[0107] FIG. 3 is an exemplary block diagram for illustrating a query command in an autonomous driving system according to one embodiment.

[0108] The description with reference to FIGS. 1a and 2 may be applied in the same way to FIG. 3, and duplicate content may be omitted.

[0109] According to one embodiment, the first query command may be a command query based on a Chain-of-Thought (COT). A COT may refer to breaking down a problem of relatively complex logic to form a complete thought process through a series of logically related thought processes. Thus, when performing reasoning or decision-making, logical steps within the thought process can be sequentially connected. In the field of artificial intelligence, a Chain of Thought (CoT) is a series of reasoning steps automatically generated by a model that can be used to describe the logical path to a final result, which can help improve the explainability of the model. In the described embodiment, the CoT may, for example, consist of five command queries (including explanation, interest, decision, trajectory, reflection, etc.) and corresponding responses. By applying an attention mechanism to these query qualities and scene tokens, a large-scale language model can extract information related to the command and generate a corresponding response (including a predicted trajectory and related interpretation).

[0110] Based on at least one embodiment above, the described embodiments may suggest other examples of autonomous driving trajectory planning. Specifically, a set of CoT query commands (including description, interest, decision, trajectory, reflection, etc.) including multiview RGB video and safety-oriented reflection steps may be input, and all relevant scene information based on multiview RGB video data may be encoded through a scene tokenizer (210) to generate scene tokens and trajectory proposals (e.g., first trajectory). The trajectory proposals may be embedded into trajectory query commands (e.g., acquired second query commands). Subsequently, by utilizing attention parameters of a pre-trained large-scale model, the relationship between safety-oriented CoT queries and scene tokens may be learned, and corresponding knowledge may be extracted from the scene according to a logical sequence. The LLM (220) network outputs a response to each query command, which includes inference related to the trajectory, a predicted trajectory (e.g., included in the trajectory response), and a safety assessment (e.g., included in the reflection response), and may output an optimized trajectory (e.g., included in the reflection response) based on the safety assessment results.

[0111] According to one embodiment, an electronic device (600) can input multiview RGB video into a scene tokenizer (210) to generate scene tokens and trajectory suggestions in which semantic information within the scene is encoded. For example, the electronic device (600) can sequentially input the generated scene tokens and trajectory suggestions into a large-scale language model (LLM (220)) to perform five query commands: description, interest, decision, trajectory, and reflection.

[0112] According to one embodiment, the electronic device (600) can use a description query command to understand the overall situation of the driving scene and extract semantic information such as the composition of the scene, relationships between objects, weather, and road shape. The electronic device (600) can use an interest query command to identify key objects requiring attention within the scene and determine the location and movement status of surrounding objects such as pedestrians or vehicles. The electronic device (600) can use a decision query command to determine future actions based on the current scene and key object information. For example, the electronic device (600) can select driving strategies such as deceleration, stopping, and turning.

[0113] According to one embodiment, the electronic device (600) can convert a predicted driving motion into a specific trajectory form using a trajectory query command and generate trajectory point coordinates over time. Additionally, the electronic device (600) can evaluate the safety of the generated trajectory using a reflection query command and generate an optimized final planned trajectory by considering whether there is a violation of navigation commands, yaw angle deviation, smoothness, etc.

[0114] According to one embodiment, the electronic device (600) can form a chain of thought (CoT) by sequentially connecting the results of each query command execution. The electronic device (600) can cumulatively refer to the responses generated at each stage of explanation, interest, decision, trajectory, and reflection to make a judgment at a subsequent stage based on the reasoning results of the previous stage. For example, the electronic device (600) can derive an interest response based on an explanation response, derive a decision response by referencing the interest response, and then sequentially generate a trajectory response and a reflection response.

[0115] According to one embodiment, the electronic device (600) can secure semantic understanding of the driving scene, rationality of decision-making, and consistency of trajectory generation through this accident chain structure. Additionally, the electronic device (600) can improve the explainability of the autonomous driving system by outputting each response result along with a linguistic description.

[0117] FIG. 4a is a schematic diagram illustrating an example of a scene tokenizer according to one embodiment.

[0118] FIG. 4b is a schematic flowchart illustrating the operation of a scene tokenizer according to one embodiment.

[0119] The description with reference to FIGS. 1a through 3 may be applied in the same way to FIGS. 4a and 4b, and overlapping content may be omitted.

[0120] Referring to FIG. 4a, a compact scene tokenizer (400) (an example of a scene tokenizer (210)) can be proposed.

[0121] A scene tokenizer (400) according to one embodiment can extract visual features by processing video data input from a multi-view video through a backbone (410). The backbone (410) can generate visual features related to objects, road boundaries, traffic signs, driving status of a vehicle, etc., within a scene by synthesizing images obtained from multiple views or converting them into a feature space.

[0122] An electronic device (600) according to one embodiment can spatially align visual features extracted from a backbone (410) using a transformation layer (420) and reconstruct them into information necessary for scene understanding. The transformation layer (420) can refine information such as the position, movement, direction, speed, and lane structure of an object based on the visual features and provide it as a cross-attention input for the next stage.

[0123] An electronic device (600) according to one embodiment can perform a cross-attention operation between a detection query expression and a scene query expression to combine context information of the entire scene and local information of the detected object. The cross-attention can correct changes in the state of the detected object to match the overall structure of the scene and mitigate information imbalances caused by interactions between objects or differences in field of view.

[0124] An electronic device (600) according to one embodiment can maintain a detection query expression generated in a previous frame using a detection memory storage and combine it with the detection query expression of the current frame to ensure temporal consistency. By utilizing this detection memory storage, the scene tokenizer (400) can perform continuous tracking of objects and state updates for consecutive image inputs.

[0125] An electronic device (600) according to one embodiment can finally integrate a detection query expression and a scene query expression by post-processing cross-attention results through a feedforward network (FFN). Through this process, the electronic device (600) can generate a scene token that reflects the relationship between the global structure and local objects within the scene.

[0126] An electronic device (600) according to one embodiment can perform various prediction tasks, such as trajectory proposal, self-vehicle position estimation, danger object detection, and scene summarization, using the scene tokens finally obtained. For example, the electronic device (600) can calculate the distance and relative speed of obstacles according to the direction of travel of the vehicle based on the scene tokens, or propose an appropriate driving trajectory according to the shape of the road.

[0127] More specifically, referring to FIG. 4a and FIG. 4b together, the processing of operation 101 may include the following operations (1011 to 1015).

[0128] In operation 1011, the electronic device (600) can extract visual features from an image.

[0129] According to one embodiment, a compact scene tokenizer (400) may include a backbone (410) network (backbone, a general computational module) and a transformation layer (420) (e.g., projection, a general computational module) for extracting visual features from an image.

[0130] In operation 1012, the electronic device (600) may obtain a scene query representation and a perception query representation. The perception query representation may represent information related to at least one object in a driving scene, and the scene query representation may represent information related to the semantics of the driving scene. The information related to the semantics of the driving scene may include the type of scene, relationships between objects, etc.

[0131] According to one embodiment, a perception query expression can be represented as a learnable parameter and can be expressed as a predetermined P-X-C tensor, where P represents the number of perceived objects and C represents the number of feature channels.

[0132] According to one embodiment, a scene query representation can also be represented as a learnable parameter and can be expressed as a predetermined S / X / C tensor, where S represents the number of scene queries and C represents the number of feature channels.

[0133] In operation 1013, the electronic device (600) can update the scene query representation and the detection query representation based on visual features.

[0134] According to one embodiment, the compact scene tokenizer (400) may further include a cross attention module (a general operation module) and a feed forward neural network (FFN, a general operation module), thereby performing an interaction between visual features, a scene query representation, and a detection query to extract various scene information through the query representation.

[0135] In operation 1014, the electronic device (600) can generate a scene token based on an updated detection query expression and an updated scene query expression.

[0136] According to one embodiment, the updated detection query representation may be converted into an auxiliary prediction, which may include 3D object location, object motion, vectorized map (e.g., elements such as boundary lines, dividing lines, crosswalks and centerlines), vehicle status, etc. However, the described embodiments are not limited thereto.

[0137] According to one embodiment, additional predictions can be further supervised through data labeling. Rich scene information, including object-centric dynamic and static elements, can be encoded into scene tokens in combination with updated scene query representations, which can help the LLM understand drivable areas and recognize the intent of various objects. Additionally, by explicitly encoding information related to the ego state, the LLM's awareness of the ego state and its role as a driving entity can be enhanced.

[0138] In operation 1015, the electronic device (600) can generate a first trajectory based on an updated detection query representation.

[0139] For example, the updated detection query expression can be converted into additional predictions and trajectory suggestions, whereby the electronic device (600) can obtain a trajectory suggestion (e.g., a first trajectory).

[0140] In the embodiments, updating the scene query expression and the detection query expression can achieve effects such as improving prediction accuracy by referring to historical data and identifying abnormal data.

[0141] An electronic device (600) according to one embodiment may obtain a perception memory from a perception memory bank when updating the scene query expression and the perception query expression based on the visual features. The perception memory may include a perception query expression updated in the past and / or a scene query expression updated in the past. The electronic device (600) may update the scene query expression and the perception query expression based on the visual features and the perception memory.

[0142] Likewise, the electronic device (600) can extract various scene information through the query representation by using a cross-attention module and a feed-forward neural network to perform interaction between visual features, a detection memory storage, a scene query representation, and a detection query representation.

[0143] As illustrated in FIGS. 4a and 4b, embodiments can be seen to show one example of the processing process of a compact scene tokenizer (400).

[0144] According to one embodiment, visual features are extracted from a multiview video (e.g., RGB video) using a backbone network (410) and a transformation layer (420). Then, utilizing a transformer structure (e.g., including two cross-attention network layers and one feed-forward network layer), various scene information is iteratively encoded into the scene query representation and the detection query representation through the interaction between the scene query representation, the detection query representation, the detection memory store, and the visual features. The updated detection query representation can be transformed into additional auxiliary predictions (e.g., 3D object location, object motion, vectorized map, vehicle state, etc.) and trajectory proposals. These additional predictions can be fused with the updated scene query representation to obtain scene tokens (e.g., 3D scene tokens).

[0145] Furthermore, the electronic device can input scene tokens into the aforementioned large-scale language model to process autonomous driving trajectory planning as follows:

[0146] 1) The electronic device (600) can generate a 3D scene token (scene token) and a trajectory suggestion (first trajectory) based on a multi-view camera image using a compact scene tokenizer (400).

[0147] 2) The electronic device (600) can use a large-scale language model to perform mutual operations between 3D scene tokens and CoT language query commands to obtain a finally predicted trajectory and a corresponding CoT language response.

[0148] Based on this, an end-to-end autonomous driving system according to one embodiment may execute a single frame or multiple frames of multi-view images as input, and the system may directly output a predicted trajectory and related descriptions (e.g., logic of trajectory generation, safety of the trajectory, etc.).

[0149] The large-scale language model-based end-to-end autonomous driving system provided in the embodiments is not limited to a specific type of knowledge and can integrate various types of knowledge, such as knowledge related to trajectory generation and knowledge related to trajectory safety. It can also improve the explainability and generalization capabilities of the system by utilizing the extensive knowledge, powerful reasoning, and generalization capabilities inherent in the large-scale language model. This allows for the simultaneous improvement of the safety, explainability, and generalization capabilities of the autonomous driving system without the need to integrate various knowledge by relying on a complex architecture.

[0150] The large-scale language model-based end-to-end autonomous driving system provided in the embodiments can be applied to trajectory prediction as follows.

[0151] 1) For example, the autonomous driving system may be applied to high-level driving actions such as acceleration, deceleration, straight driving, or turning. In the described embodiment, a reflection step may be provided to generate a final planned trajectory.

[0152] 2) For example, in a low-level planning process, the autonomous driving system can directly convert the control signals of the autonomous driving system.

[0153] 3) For example, an autonomous driving system can be applied to a low-level planned trajectory that includes descriptions, providing the necessary output format and enhanced explainability.

[0154] The embodiments ensure that the generated trajectory is consistent with the language-based description and can effectively utilize this description to improve trajectory performance (e.g., accuracy and safety).

[0155] In the embodiments, during the training phase of the autonomous driving system, the following strategy may be provided to obtain sufficient data for a large-scale language model to learn reflexive ability.

[0156] 1) During the training phase of an autonomous driving system, various methods can be used to generate trajectories requiring reflection, such as cross-validation of the training set or generating trajectories from the training set using a model with somewhat lower performance. Subsequently, in the REFLECTION part, one of the trajectories generated during training can be randomly sampled and used as input.

[0157] 2) In the TRAJECTORY section, there may be labeled trajectories during the supervision process. Therefore, to prevent leakage of the labeled trajectories, the TRAJECTORY section may be masked in the REFLECTION section. That is, while the REFLECTION section allows viewing of processes in other stages, relevant information regarding the TRAJECTORY stage may not be visible.

[0158] In the embodiments, during the test phase of the autonomous driving system, results from all stages of the CoT can be generated sequentially. In the REFLECTION stage, the prediction result of the TRAJECTORY stage is taken as input, and subsequently, the LLM can simulate and evaluate the safety of the predicted trajectory, explain the optimization method, and output the optimized trajectory.

[0160] FIGS. 5a to 5c are schematic diagrams illustrating the results obtained through an autonomous driving trajectory generation method according to one embodiment.

[0161] The description with reference to FIGS. 1a to 4b may be applied in the same way to FIGS. 5a to 5c, and overlapping content may be omitted.

[0162] The method disclosed in the embodiments has performed a large number of tests on a specific dataset, and it can be confirmed that the precision of the autonomous driving system provided in the embodiments exceeds that of the conventional system. Hereinafter, several effects can be observed along with the test results through FIGS. 5a and 5b. Among them, the trajectory (510) predicted by the conventional system is indicated as a hollow square, the final planned trajectory (e.g., third trajectory) (520) after reflection optimization provided in the embodiment of the present disclosure is indicated as a filled circle, and the labeled expert trajectory (530) can be indicated as a filled triangle.

[0163] Referring to FIG. 5a, the trajectory (510) predicted by the conventional system is directed to the left, and there may be a risk of collision with an oncoming vehicle. The final planned trajectory (520) after reflection optimization provided in the embodiment of the present disclosure is correct (straight), and it can be confirmed that this avoids the risk of collision.

[0164] Referring to FIG. 5b, the trajectory (510) predicted by the conventional system may be at risk of deviating from the drivable area. The final planned trajectory (520) after reflection optimization provided in the embodiment of the present disclosure is oriented correctly and may not deviate from the drivable area.

[0165] Referring to FIG. 5c, the trajectory (510) predicted by the conventional system is oriented incorrectly and there may be a risk of going out of the drivable area. The final planned trajectory (520) after reflection optimization provided in the embodiment of the present disclosure is oriented correctly, which can ensure not only avoiding the risk of collision with an oncoming vehicle but also not going out of the drivable area.

[0167] FIG. 6 is an example diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0168] The description with reference to FIGS. 1a through 5 may be applied in the same way to FIG. 6, and duplicate content may be omitted.

[0169] The technical solution provided in the embodiments can be applied not only to smart driving vehicles but also to servers such as independent physical servers, server clusters or distributed systems composed of multiple physical servers, and cloud servers.

[0170] The embodiments further provide an electronic device (600), said electronic device (600) including a processor, optionally including a transceiver and / or memory coupled with the processor, said processor may be configured to execute a step of the method provided by any one embodiment of the present disclosure.

[0171] As illustrated in FIG. 6, the electronic device (600) illustrated in FIG. 6 includes a processor (610) and a memory (630). The processor (610) and the memory (630) are connected to each other, for example, via a bus (620). Optionally, the electronic device (600) may further include a transceiver (640), and the transceiver (640) may be used for data exchange, such as transmitting data and / or receiving data between the electronic device and another electronic device. In actual applications, the transceiver (640) is not limited to one, and it should be noted that the structure of the electronic device (600) does not constitute a limitation to the embodiments of the present disclosure. Optionally, the electronic device may be a first network node, a second network node, or a third network node.

[0172] The processor (610) may be a CPU, a general-purpose processor, a DSP, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. This may implement or execute the various exemplary logic blocks, modules, and circuits described in this application. The processor (610) may also be a combination that realizes computing functions, including, for example, a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0173] The bus (620) may include a path for transmitting information between the above components. The bus (620) may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus (620) may be classified into an address bus, a data bus, a control bus, etc. For convenience of example, only one thick line is shown in FIG. 6, but there is not only one bus or only one type of bus.

[0174] The memory (630) may be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, and may be EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital multi-purpose discs, Blu-ray discs, etc.), disk storage media, other magnetic storage devices or any other media that can be used to carry or store computer programs and can be read by a computer, but is not limited thereto.

[0175] Memory (630) is used to store a computer program for executing an embodiment of the present disclosure and is controlled by a processor (610). The processor (610) is configured to execute the computer program stored in memory (630) to realize the steps illustrated in the above-described method embodiment.

[0177] The embodiments described above may be implemented as hardware components, software components, and / or combinations of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0178] Software may include computer programs, code, instructions, or a combination thereof, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave in order to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on computer-readable recording media.

[0179] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0180] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can apply various technical modifications and variations based on the above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0181] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A method for generating an autonomous driving trajectory, comprising: an operation of generating a scene token and a first trajectory based on collected images; an operation of obtaining a first query command - the first query command includes a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory -; and an operation of generating a third trajectory based on the scene token, the first trajectory, and the first query command using a large-scale language model, wherein the third trajectory is obtained by processing a second trajectory based on second information obtained in response to the second query, the second trajectory is obtained based on first information obtained in response to the first query, and the first information and the second information are obtained based on the scene token, the first trajectory, and the first query command. Claim 2 A method for generating an autonomous driving trajectory according to claim 1, wherein the second information comprises at least one of information regarding whether the direction of the second trajectory corresponds to the direction of a navigation command or information regarding whether the yaw angle between adjacent pairs of trajectory points in the second trajectory exceeds a first threshold value. Claim 3 A method for generating an autonomous driving trajectory according to claim 2, wherein the operation of generating the third trajectory includes, based on the second information, if the direction of the second trajectory does not match the direction of the navigation command, adjusting the vertical coordinate value of a trajectory point within the second trajectory to make the direction of the trajectory correspond to the direction of the navigation command, and determining the third trajectory corresponding to the adjusted second trajectory. Claim 4 A method for generating an autonomous driving trajectory according to claim 2, comprising the operation of determining the third trajectory corresponding to the adjusted second trajectory by gradually increasing the horizontal coordinate value of a trajectory point within the second trajectory and adjusting the fluctuation range of the vertical coordinate value so that it is smaller than the second threshold value when the yaw angle between at least one pair of adjacent trajectory points within the second trajectory exceeds a first threshold value based on the second information. Claim 5 A method for generating an autonomous driving trajectory according to claim 1, wherein the operation of generating the third trajectory comprises: an operation of generating explanatory information regarding the processing process of the second trajectory based on the second information; and an operation of processing the second trajectory based on the explanatory information to obtain the third trajectory. Claim 6 A method for generating an autonomous driving trajectory according to claim 1, wherein the operation of generating a third trajectory comprises: an operation of generating the first information and the second information based on the scene token, the first trajectory, and the first query command; an operation of generating a second trajectory regarding the first information based on the scene token, the first trajectory, the first query command, and the first information; and an operation of processing the second trajectory based on the second information to obtain a third trajectory. Claim 7 A method for generating an autonomous driving trajectory according to claim 1, wherein the first information comprises at least one of information related to a driving scene; information related to a key object in the driving scene; information related to a driving action to be performed; or information related to the prediction of a second trajectory. Claim 8 A method for generating an autonomous driving trajectory according to claim 1, wherein the operation of generating a third trajectory comprises: the operation of embedding the first trajectory into the first query command to obtain a second query command; and the operation of generating a third trajectory based on the scene token and the second query command. Claim 9 A method for generating an autonomous driving trajectory according to claim 1, wherein the operation of generating a scene token and a first trajectory based on the collected image comprises: an operation of extracting visual features from the image; an operation of obtaining a scene query representation—wherein the scene query representation represents information related to the semantics of a driving scene—and a perception query representation—wherein the perception query representation represents information related to at least one object in the driving scene—; an operation of updating the scene query representation and the perception query representation based on the visual features; an operation of generating the scene token based on the updated perception query representation and the updated scene query representation; and an operation of generating the first trajectory based on the updated perception query representation. Claim 10 A method for generating an autonomous driving trajectory according to claim 9, wherein the operation of updating the scene query expression and the detection query expression based on the visual features comprises: the operation of acquiring a perception memory—the perception memory includes a detection query expression updated in the past and / or a scene query expression updated in the past—and the operation of updating the scene query expression and the detection query expression based on the visual features and the perception memory. Claim 11 In an electronic device, at least one processor including a processing circuit; An electronic device comprising a memory for storing instructions, wherein when the instructions are executed individually or collectively by at least one processor, the electronic device generates a scene token and a first trajectory based on a collected image, obtains a first query command—the first query command includes a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory—and generates a third trajectory based on the scene token, the first trajectory, and the first query command using a large-scale language model, wherein the third trajectory is obtained by processing a second trajectory based on second information obtained in response to the second query, wherein the second trajectory is obtained based on first information obtained in response to the first query, and wherein the first information and the second information are obtained based on the scene token, the first trajectory, and the first query command. Claim 12 An electronic device according to claim 11, wherein the second information comprises at least one of information regarding whether the direction of the second trajectory corresponds to the direction of a navigation command or information regarding whether the yaw angle between adjacent pairs of trajectory points of each pair of the second trajectory exceeds a first threshold value. Claim 13 An electronic device according to claim 12, wherein when the instructions are executed individually or collectively by at least one processor, the electronic device causes, based on the second information, if the direction of the second trajectory does not coincide with the direction of the navigation command, to adjust the vertical coordinate value of a trajectory point within the second trajectory so that the direction of the trajectory corresponds to the direction of the navigation command, and to determine the third trajectory corresponding to the adjusted second trajectory. Claim 14 An electronic device according to claim 12, wherein when the instructions are executed individually or collectively by at least one processor, the electronic device causes, based on the second information, when the yaw angle between at least one pair of adjacent trajectory points within the second trajectory exceeds a first threshold value, the horizontal coordinate value of a trajectory point within the second trajectory is gradually increased and the vertical coordinate value is adjusted so that the range of variation is smaller than the second threshold value, thereby determining the third trajectory corresponding to the adjusted second trajectory. Claim 15 An electronic device according to claim 11, wherein when the instructions are executed individually or collectively by at least one processor, the electronic device generates explanatory information regarding the processing of the second trajectory based on the second information, and processes the second trajectory based on the explanatory information to obtain the third trajectory. Claim 16 An electronic device according to claim 11, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device generates the first information and the second information based on the scene token, the first trajectory and the first query command, generates a second trajectory regarding the first information based on the scene token, the first trajectory, the first query command and the first information, and processes the second trajectory based on the second information to obtain a third trajectory. Claim 17 An electronic device according to claim 11, wherein the first information comprises at least one of information related to a driving scene; information related to a key object in the driving scene; information related to a driving action to be performed; or information related to the prediction of a second trajectory. Claim 18 An electronic device according to claim 11, wherein when the instructions are executed individually or collectively by the at least one processor, the electronic device enables the electronic device to embed the first trajectory into the first query command to obtain a second query command and generate a third trajectory based on the scene token and the second query command. Claim 19 An electronic device according to claim 11, wherein when the instructions are executed individually or collectively by at least one processor, the electronic device extracts visual features from the image, obtains a scene query representation—the scene query representation represents information related to the semantics of a driving scene—and a perception query representation—the perception query representation represents information related to at least one object in the driving scene—updates the scene query representation and the perception query representation based on the visual features, generates the scene token based on the updated perception query representation and the updated scene query representation, and generates the first trajectory based on the updated perception query representation. Claim 20 A recording medium storing instructions executable on a hardware device, wherein the instructions are configured to: generate a scene token and a first trajectory based on a collected image; obtain a first query command—the first query command includes a first query regarding the logic of trajectory generation and a second query regarding the safety of the trajectory—and generate a third trajectory based on the scene token, the first trajectory, and the first query command using a large-scale language model; wherein the third trajectory is obtained by processing a second trajectory based on second information obtained in response to the second query; wherein the second trajectory is obtained based on first information obtained in response to the first query; and wherein the first information and the second information are obtained based on the scene token, the first trajectory, and the first query command.

Images