ELECTRONIC DEVICE FOR INTEGRATION AND PREDICTION OF FUTURE TRAJECTORIES OF ANY NUMBER OF SURROUNDING VEHICLES AND METHOD FOR OPERATION OF THIS DEVICE

Abstract

An operating procedure of an electronic device, comprising: the recognition of historical trajectories of one or more surrounding objects; Predictions of future trajectories of surrounding objects in an integrated manner based on the identified historical trajectories; and Planning a trajectory for the electronic device based on the predicted future trajectories of the surrounding objects, including the prediction of future trajectories: Configuring a graph model based on the identified historical trajectories; and Prediction of future trajectories based on the graph model, where the graph model comprises a multitude of nodes and a multitude of edges connecting the nodes, where the nodes represent the surrounding objects or the electronic device, where the edges indicate interactions between the surrounding objects and between the electronic device and each of the surrounding objects, or the directional dependence of the interactions, where edges connecting nodes that display the surrounding objects contain bidirectionality, and wherein an edge connecting any of the nodes indicating the surrounding object and a node indicating the electronic device contains a unidirectionality directed from the electronic device to the surrounding object.

Description

CROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority under 35 USC §119 to Korean Patent Application No. 10-2020-0022060, filed February 24, 2020, which is incorporated herein by reference in its entirety. BACKGROUND OF THE INVENTION 1. Technical field

[0002] Various embodiments relate to an electronic device for the integrated prediction of future trajectories of any number of surrounding vehicles and its operation. 2. Description of the state of the art

[0003] Recently, autonomous driving technology has been further developed. The development of autonomous driving technology in various driving situations, such as on a highway, a downtown street, and an intersection, is underway. Accordingly, the development of technology capable of dealing with any number of surrounding vehicles in any driving situation is becoming increasingly important. Accordingly, research is being conducted on predicting the driving trajectory of a vehicle with respect to multiple surrounding vehicles. However, in essence, an autonomous vehicle encounters a changeable road environment, and the traffic in each road environment is variably changed in real time. Accordingly, it is difficult to ensure consistent performance and computational load when predicting the driving trajectories of multiple vehicles with respect to the real-time varying traffic.

[0004] A concept for recognizing historical trajectories of one or more surrounding objects and predicting future trajectories of the surrounding objects based on the recognized historical trajectories is known, for example, from KR 10 1 951 595 B1. In contrast, DE 10 2018 215 668 A1 discloses a concept for planning a future driving trajectory based on future trajectories of surrounding objects. To consider all objects, including their behavior and interactions, the use of nodes and edges can be provided in neural networks according to KR 10 2 068 279 B1. SUMMARY OF THE INVENTION

[0005] Various embodiments provide an electronic device capable of ensuring consistent performance and computational load in predicting a vehicle's route based on real-time varying traffic and operating accordingly.

[0006] Various embodiments provide an electronic device capable of predicting future trajectories of any number of surrounding vehicles in an integrated manner and with a corresponding method of operation.

[0007] According to various embodiments, a method of operation of an electronic device may include detecting trajectories of one or more surrounding objects, integrated prediction of future trajectories of the surrounding objects based on the detected trajectories, and planning a travel trajectory of the electronic device based on the predicted future trajectories of the surrounding objects.

[0008] According to various embodiments, an electronic device may include at least one of a sensor module or a camera module and a processor coupled to at least one of the sensor module or the camera module and configured to collect information about environmental situations through at least one of the sensor module or the camera module. The processor may be configured to detect trajectories of one or more surrounding objects, predict future trajectories of the surrounding objects in an integrated manner based on the trajectories, and plan a travel trajectory of the electronic device based on the predicted future trajectories of the surrounding objects. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram illustrating an electronic device according to various embodiments. Fig. 2 and Fig. 3 are diagrams for describing the operating characteristics of the electronic device according to various embodiments. Fig. 4 is a diagram illustrating an operation of the electronic device according to various embodiments. Fig. 5 is a diagram illustrating an operation of integrating and predicting future trajectories in Fig. 4 illustrated. Fig. 6 is a diagram describing an operation for configuring a Fig. 5 shown graph model. Fig. Figure 7 is a diagram illustrating an operation for predicting future trajectories based on a graph model in Fig. 5 illustrated. Fig. 8 and Fig. 9 are diagrams describing the process of predicting future trajectories based on a graph model in Fig. 5. Fig. 10 is a diagram for describing the operating effects of the electronic device according to various embodiments. DETAILED DESCRIPTION

[0009] Various embodiments of this document are described below with reference to the accompanying drawings.

[0010] Fig. 1 is a diagram illustrating an electronic device 100 according to various embodiments. Fig. 2 and Fig. 3 are diagrams for describing the operating characteristics of the electronic device 100 according to various embodiments.

[0011] With reference to Fig. 1, according to various embodiments, the electronic device 100 may include at least one communication module 110, a camera module 120, a sensor module 130, an input module 140, an output module 150, a driver module 160, a memory 170, or a processor 180. In one embodiment, one of the elements of the electronic device 100 may be omitted, or one or more other elements may be added to the electronic device 100. In one embodiment, the electronic device 100 may be an autonomous vehicle. According to another embodiment, the electronic device 100 may be installed in a vehicle and implement an autonomous vehicle.

[0012] The communication module 110 can support communication between the electronic device 100 and an external device (not shown). In this case, the communication module 110 can include at least one wireless communication module or one wired communication module. Depending on an embodiment, the wireless communication module can support at least one long-range communication method or one short-range communication method. The short-range communication method can include, for example, Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA). In the wireless communication method, the communication can be performed using the long-range communication method over a network. The network can include at least one of the following computer networks: a cellular network, the Internet, a local area network (LAN), or a wide area network (WAN).According to another embodiment, the wireless communication module may support communication with a global navigation satellite system (GNSS). For example, the GNSS may include a global positioning system (GPS).

[0013] The camera module 120 can capture an external image of the electronic device 100. In this case, the camera module 120 can be installed at a predetermined location on the electronic device 100 and capture an external image. Furthermore, the camera module 120 can generate image data for an external image of the electronic device 100. For example, the camera module 120 can include at least one lens, at least one image sensor, an image signal processor, or a flash.

[0014] The sensor module 130 can detect a state of the electronic device 100 or an external environment of the electronic device 100. Furthermore, the sensor module 130 can generate sensor data for the state of the electronic device 100 or the external environment of the electronic device 100. For example, the sensor module 130 can include at least one of an acceleration sensor, a gyroscope sensor, an image sensor, a radar sensor, a lidar sensor, or an ultrasonic sensor.

[0015] The input module 140 may receive a command or data from outside the electronic device 100 to be used for at least one of the elements of the electronic device 100. For example, the input module 140 may include at least one of a microphone, mouse, or keyboard. In one embodiment, the input module may include at least one of a touch circuit configured to detect a touch or a sensor circuit configured to measure the intensity of a force generated by a touch.

[0016] The output module 150 can provide information to the outside of the electronic device 100. In this case, the output module 150 can include at least one display module or one audio module. The display module can output information visually. For example, the display module can include at least one display device, a hologram device, or a projector. In one embodiment, the display module can be assembled with at least one of the touch circuits or sensor circuits of the input module 140 and implemented as a touch screen. The audio module can output information in audio form. For example, the audio module can include at least one speaker or one receiver.

[0017] The drive module 160 can be used to operate the electronic device 100. If the electronic device 100 is an autonomous vehicle, the drive module 160 can include various parts, depending on the embodiment. According to another embodiment, if the electronic device 100 is installed in a vehicle to implement an autonomous vehicle, the drive module 160 can be connected to various parts of the vehicle. Accordingly, the drive module 160 can operate while controlling at least one of the parts. For example, the parts can include at least one motor module, an acceleration module, a braking module, a steering module, or a navigation module.

[0018] Memory 170 may store at least one of the programs or data used by at least one of the elements of electronic device 100. For example, memory 170 may include at least one of volatile and non-volatile memory.

[0019] The processor 180 can control at least one of the elements of the electronic device 100 by executing a program of the memory 170 and can perform data processing or operations. The processor 180 can collect information about the environmental situations of the electronic device 100. In this case, the processor 180 can collect the information about the environmental situations of the electronic device 100 based on at least one of the image data obtained by the camera module 120 or the sensing data obtained by the sensor module 130. The processor 180 can predict the future trajectory of the surrounding objects to plan the travel trajectory of the electronic device 100 based on the environmental situation of the electronic device 100. Accordingly, the processor 180 can control an operation of the electronic device 100 based on the planned travel trajectory of the electronic device 100.For this purpose, the processor 180 may control the drive module 160 based on the planned travel trajectory of the electronic device 100. The processor 180 may recognize the historical trajectories of surrounding objects based on information about environmental situations of the electronic device 100. In this case, the surrounding objects may be vehicles around the electronic device 100. For example, all surrounding objects may be moving at corresponding speeds, and at least one of the surrounding objects may be stopped. In this case, the processor 180 may recognize state information about the electronic device 100 and recognize state information about each of the surrounding objects. Furthermore, as shown in FIG. Fig. 2, configure a moving coordinate system based on a constant speed model using state information about the electronic device 100 and display the relative positions of the surrounding objects on the moving coordinate system. In Fig. 2, the electronic device 100 can be represented as a moving object. In this case, the coordinate carrier of the moving coordinate system can move in accordance with the speed of the electronic device 100. Accordingly, the processor 180 can recognize historical trajectories of the surrounding objects based on the positions of the surrounding objects on the moving coordinate system.

[0020] As in Fig. 3, the processor 180 may predict future trajectories of the surrounding objects in an integrated manner based on the detected historical trajectories of the surrounding objects. In this case, the processor 180 may predict the future trajectories by integrating and estimating the interactions between the surrounding objects and the electronic device 100. In this case, the processor 180 may estimate the interactions in an integrated manner based on the properties of the interactions. For example, the properties of the interactions may be simultaneously multicentric, diffusive, and time-varying. Simultaneous multicentricity may indicate that multiple interactions are occurring simultaneously.Diffusivity may indicate that an interaction between the electronic device 100 and any one of the surrounding objects, or between any two of the surrounding objects, may gradually diffuse to affect the entire electronic device 100 and the surrounding objects. Temporal variability may indicate that an interaction changes over time. To this end, the processor 180 may configure a graph model for an environmental situation of the electronic device 100 by implementing graph modeling, as shown in FIG. Fig. 3(b), based on trajectories of the surrounding objects, as in Fig. 3(a). Furthermore, the processor 180 may configure a so-called “graph convolutional neural network” based on the graph model, as shown in Fig. 3(b), and can predict future trajectories of the surrounding objects, as shown in Fig. 3(c), the convolutional neural network based on a long-term short-term memory (LSTM) can predict the future trajectories of the surrounding objects from the graph model and the neural graph. Accordingly, the processor 180 can plan the driving trajectory of the electronic device 100 based on the predicted future trajectories of the surrounding objects. Fig. 4 is a diagram illustrating an operation of the electronic device 100 according to various embodiments.

[0021] With reference to Fig. 4, in operation 410, the electronic device 100 may detect an environmental situation of the electronic device 100. The processor 180 may receive image data via the camera module 120. The processor 180 may receive sensing data via the sensor module 130. Accordingly, the processor 180 may collect information about the environmental situation of the electronic device 100 based on at least one of the image or sensing data.

[0022] In this case, the processor 180 may detect state information about the electronic device 100. In this case, the surrounding objects may be vehicles around the electronic device 100. For example, all surrounding objects may be moving at the corresponding speed, and at least one of the surrounding objects may be stopped. Furthermore, the processor 180, as in Fig. 2, configure a moving coordinate system based on a constant velocity model using the state information on the electronic device 100. In Fig. 2, the electronic device 100 can be represented as a moving object. In this case, the coordinate carrier of the moving coordinate system can move in accordance with the speed of the electronic device 100. Furthermore, the processor 180 can detect state information about each of the surrounding objects based on the information about the environmental situation of the electronic device 100.

[0023] At operation 420, the electronic device 100 may predict future trajectories of the electronic device 100 and the surrounding objects in an integrated manner. In this case, the processor 180 may represent the relative positions of the surrounding objects on the moving coordinate system based on a constant velocity model, as shown in Fig. 2, based on the state information about the surrounding objects. Accordingly, the processor 180 can recognize historical trajectories of the surrounding objects based on the positions of the surrounding objects on the moving coordinate system based on a constant-velocity model. Furthermore, the processor 180 can predict future trajectories of the electronic device 100 and the surrounding objects in an integrated manner based on the trajectories of the surrounding objects. In this case, the processor 180 can estimate the interactions between the surrounding objects and the electronic device 100 in an integrated manner and predict the future trajectories based on the interactions.In this case, the processor 180 may estimate the interactions in an integrated manner based on the properties of the interactions and predict the future trajectories based on the interactions. For example, the properties of the interactions may be simultaneously multicentric, diffusive, and time-varying. Simultaneous multicentricity may indicate that multiple interactions occur simultaneously. Diffusivity may indicate that an interaction between the electronic device 100 and any one of the surrounding objects, or between any two of the surrounding objects, may gradually diffuse to affect the entire electronic device 100 and the surrounding objects. Temporal variability may indicate that an interaction changes over time. This may be discussed later with reference to FIG. Fig. 5 and Fig. 6 are described in more detail.

[0024] Fig. Figure 5 is a diagram showing the integrated operation of predicting the future trajectories in Fig. 4 illustrated. Fig. 6 is a diagram describing an operation for configuring a Fig. 5 shown graph model.

[0025] With reference to Fig. 5, in operation 510, the electronic device 100 may configure an environmental situation of the electronic device 100 as a graph model. The processor 180 may configure the graph model for the environmental situation of the electronic device 100 by performing graph modeling based on state information about the electronic device 100 and the surrounding objects. In this case, the processor 180 may configure the graph model using a moving coordinate system based on a constant speed model. For example, the processor 180 may, as in Fig. shown as a graph model, configure the electronic device 100 and the surrounding objects as well as the interactions between the electronic device 100 and the surrounding objects.

[0026] The graph model can be, for example, as in Fig. 6(b) and may include a plurality of nodes and a plurality of edges connecting the plurality of nodes. The nodes may each indicate the electronic device 100 and the surrounding objects. In this case, the nodes may represent at least a position, a speed, or a heading angle, e.g., based on state information about the electronic device 100 or the surrounding objects. In Fig. 6(b), the nodes comprise a first node (e.g., node-ego; node 0) and second nodes (e.g., node 1, node 2, node 3, node 4, and node 5). The first node (e.g., node-me; node 0) may denote the electronic device 100. The second nodes (e.g., node 1, node 2, node 3, node 4, and node 5) may each indicate the surrounding objects. The edges may each indicate the directionality of the interactions between the electronic device 100 and the surrounding objects. The edges may represent at least a relative location or a relative velocity, e.g., based on relative state information about the electronic device 100 or the surrounding objects. In this case, an edge connecting the first node (e.g., node-me; node 0) and any of the second nodes (e.g., node 1, node 2, node 3, node 4, and node 5) may have unidirectionality from the first node (e.g.,Node-I; Node 0) to any of the second nodes (e.g., Node 1, Node 2, Node 3, Node 4, and Node 5). The reason for this is that the electronic device 100 can be controlled autonomously. An edge connecting any two of the second nodes (e.g., Node 1, Node 2, Node 3, Node 4, and Node 5) can contain bidirectionality.

[0027] In operation 520, the electronic device 100 may predict future trajectories of the electronic device 100 and the surrounding objects in an integrated manner based on the graph model. The processor 180 may predict the future trajectories of the electronic device 100 and the surrounding objects based on the interactions between the surrounding objects and the electronic device 100. In this case, the processor 180 may configure a graph convolutional neural network based on the graph model and predict the future trajectories from the graph convolutional neural network based on a long-short-term memory (LSTM) network. This may be described later with reference to Fig. 7, Fig. 8 and Fig. 9 are described in more detail.

[0028] Fig. Figure 7 is a diagram showing the functionality of predicting future trajectories based on the graph model in Fig. 5 illustrates. Fig. 8 and Fig. 9 are diagrams describing the functionality of the prediction of future trajectories based on the graph model in Fig. 5.

[0029] With reference to Fig. 7, the electronic device 100 may, in operation 710, assign importance to each of the surrounding objects using a scalable attention mechanism. The scalable attention mechanism may be as in Fig. 8(a). The electronic device 100 can process an indefinite number of surrounding objects that vary in real time in an integrated manner through the scalable attention mechanism. In this case, the processor 180 can relatively evaluate two nodes connected by each edge in the graph model based on state information about the electronic device 100 and the surrounding objects. Furthermore, the processor 180 can assign importance to each of the nodes based on the evaluation results.

[0030] At operation 720, electronic device 100 may calculate an adjacency matrix based on the importance of each of the surrounding objects. Processor 180 may calculate an adjacency matrix that takes into account the importance of each of the surrounding objects.

[0031] In operation 730, the electronic device 100 may configure a graph convolution neural network using the adjacency matrix. The processor 180 may configure the graph convolution neural network based on the graph model and the adjacency matrix. In this case, the processor 180 may, as shown in Fig. 8(b), configure the neural graph convolutional neural network by performing a graph convolution operation.

[0032] In operation 740, the electronic device 100 may predict the future trajectories of surrounding objects based on the long-term memory network (LSTM). The processor 180 may predict the future trajectories from the graph model and the LSTM-based graph convolutional neural network. In this case, the LSTM may be configured with an encoding and a decoding structure. As shown in Fig. 9(a), the processor 180 can detect a motion characteristic of each electronic device 100 and the surrounding objects, as well as interaction characteristics between the electronic device 100 and the surrounding objects, based on the state information about the electronic device 100 and the surrounding objects through the coding structure of the LSTM. Furthermore, the processor 180 can plan hidden state information and memory cell state information based on the time-changing motion characteristics and the interaction characteristics through the coding structure of the LSTM. Furthermore, as shown in Fig. 9(b), predict the future trajectories based on the state information about the electronic device 100 and the surrounding objects together with the hidden state information and the memory cell state information by the decoding structure of the LSTM.

[0033] After that, the electronic device can be 100 to Fig. 5 and perform operation 530. Referring to Fig. 5, at operation 530, the electronic device 100 may plan the trajectory of the electronic device 100 based on the predicted future trajectories of the surrounding objects. The processor 180 may plan an optimal travel trajectory that may be consistent with the predicted future trajectories of the surrounding objects.

[0034] After that, the electronic device can be 100 to Fig. 4 and perform operation 430. Looking back at Fig. 4, at operation 430, the electronic device 100 may control the electronic device 100. The processor 180 may control the drive module 160 based on the travel trajectory of the electronic device 100. In this case, the drive module 160 may operate while controlling at least one of the various parts. For example, the various parts may include at least one of the following: a motor module, an acceleration module, a braking module, a steering module, or a navigation module. Accordingly, an autonomous vehicle corresponding to the electronic device 100 or a vehicle including the electronic device 100 may travel on the roadway.

[0035] According to various embodiments, the electronic device 100 can predict future trajectories of an indefinite number of surrounding objects in an integrated manner based on the historical trajectories of the surrounding objects. In this case, the electronic device 100 can configure a graph model with respect to the surrounding objects and predict the future trajectories in an integrated manner based on the graph model and the LSTM. Accordingly, the electronic device 100 can predict the future trajectories of an indefinite number of surrounding objects in an integrated manner. Accordingly, the electronic device 100 can predict the future trajectory of a vehicle in accordance with traffic that varies in real time.That is, the electronic device 100 can plan the travel trajectory of the electronic device 100 based on the historical trajectory and the predicted future trajectories of the objects, including the electronic device 100 and the surrounding objects. Furthermore, because the electronic device 100 plans the travel trajectory based on future predicted trajectories in an integrated manner, the electronic device 100 can ensure consistent performance and a high computational load when predicting the travel trajectory.

[0036] Fig. 10 is a diagram for describing the operating effects of the electronic device 100 according to various embodiments. As shown in Fig. As shown in Figure 10(a), the computational load according to various embodiments is smaller than the computational load according to existing technologies and is also consistent regardless of the number of surrounding objects. Furthermore, as shown in Fig. As shown in Figure 10(b), the prediction error according to various embodiments is smaller than the prediction error according to existing technologies and is also consistent regardless of the number of surrounding objects. Accordingly, the performance according to various embodiments is superior to the performance of existing technologies and is also consistent regardless of the number of surrounding objects.

[0037] An operating method of the electronic device 100 according to various embodiments may include detecting historical trajectories of one or more surrounding objects, integrated prediction of future trajectories of the surrounding objects based on the detected historical trajectories, and planning the travel trajectory of the electronic device 100 based on the predicted future trajectories of the surrounding objects.

[0038] According to various embodiments, predicting the future trajectories may include configuring a graph model based on the detected historical trajectories and predicting the future trajectories based on the graph model.

[0039] According to various embodiments, the graph model comprises a plurality of nodes and a plurality of edges connecting the plurality of nodes.

[0040] According to various embodiments, the nodes may each display the surrounding objects and the electronic device 100.

[0041] According to various embodiments, the edges may indicate interactions between the surrounding objects and between the electronic device 100 and each of the surrounding objects or their directionality.

[0042] According to various embodiments, the edges connecting nodes indicating the surrounding objects may include bidirectionality.

[0043] According to various embodiments, an edge connecting one of the nodes indicating the surrounding objects to a node indicating the electronic device 100 may include a unidirectionality directed from the electronic device 100 to the surrounding object.

[0044] According to various embodiments, the prediction of the future trajectories may further comprise the relative evaluation of two nodes connected by an edge based on state information about the electronic device 100 and state information about each of the surrounding objects, wherein the two nodes are the electronic device 100 and one of the surrounding objects or two of the surrounding objects, and the assignment of importance to each of the nodes and the configuration of a graph convolutional neural network using an adjacency matrix whose importance has been taken into account.

[0045] According to various embodiments, the prediction of the future trajectories may also include the prediction of the future trajectories from the graph convolution neural network based on the LSTM.

[0046] According to various embodiments, detecting the trajectories may include configuring a moving coordinate system based on a constant velocity model using the state information on the electronic device 100, representing the relative positions of the surrounding objects on the moving coordinate system, and detecting the historical trajectories based on the time-series positions.

[0047] According to various embodiments, the prediction of the future trajectories may also include the prediction of the future trajectories on the moving coordinate system.

[0048] According to various embodiments, the operation of the electronic device 100 may also include controlling the electronic device 100 based on the planned roadway.

[0049] According to various embodiments, the electronic device 100 may be an autonomous vehicle, and surrounding objects may be vehicles around the electronic device 100.

[0050] According to various embodiments, the electronic device 100 may include at least one sensor module 130 or a camera module 120 and a processor 180 connected to at least one sensor module 130 or a camera module 120 and configured to collect information about environmental situations via at least one sensor module 130 or a camera module 120.

[0051] According to various embodiments, the processor 180 may be configured to detect historical trajectories of one or more surrounding objects, predict future trajectories of the surrounding objects in an integrated manner based on the detected historical trajectories, and plan the driving trajectory of the electronic device 100 based on the predicted future trajectories of the surrounding objects.

[0052] According to various embodiments, the processor 180 may be configured to configure a graph model based on the detected historical trajectories and predict the future trajectories based on the graph model.

[0053] According to various embodiments, the graph model may include a plurality of nodes and a plurality of edges connecting the plurality of nodes.

[0054] According to various embodiments, the nodes may each display the surrounding objects and the electronic device 100.

[0055] According to various embodiments, the edges may indicate interactions between the surrounding objects and between the electronic device 100 and each of the surrounding objects or their directionality.

[0056] According to various embodiments, the edges connecting nodes indicating the surrounding objects may include bidirectionality.

[0057] According to various embodiments, an edge connecting one of the nodes indicating the surrounding objects to a node indicating the electronic device 100 may include a unidirectionality directed from the electronic device 100 to the surrounding object.

[0058] According to various embodiments, the processor 180 may be configured to relatively evaluate two nodes connected by an edge based on state information about the electronic device 100 and state information about each of the surrounding objects, where the two nodes are the electronic device 100 and one of the surrounding objects or two of the surrounding objects, assign importance to each of the nodes, and configure a graph convolutional neural network using an adjacency matrix whose importance has been taken into account.

[0059] According to various embodiments, the processor 180 may be configured to predict the future trajectories from the LSTM-based graph convolution neural network.

[0060] According to various embodiments, the processor 180 may be configured to configure a moving coordinate system based on a constant velocity model using the state information on the electronic device 100, represent the relative positions of the surrounding objects on the moving coordinate system, and recognize the historical trajectories based on the time-series positions.

[0061] According to various embodiments, the processor 180 may be configured to predict the future trajectories on the moving coordinate system.

[0062] According to various embodiments, the processor 180 may be configured to control the electronic device 100 based on the planned travel trajectory.

[0063] According to various embodiments, the electronic device 100 may be an autonomous vehicle, and the surrounding objects may be vehicles around the electronic device 100.

[0064] Various embodiments of this document may be implemented as software, including one or more instructions stored in a storage medium (e.g., memory 170) that can be read by a machine (e.g., electronic device 100). For example, the processor (e.g., processor 180) of the machine may invoke at least one of the instructions stored on the storage medium and execute the instruction. This allows the machine to operate to perform at least one function based on the at least one invoked instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium.In this case, the term "non-transitory" simply means that the storage medium is a physical device and does not contain a signal (e.g., an electromagnetic wave). The term does not distinguish between a case in which data is stored semi-permanently on the storage medium and a case in which data is stored temporarily on the storage medium.

[0065] According to various embodiments, the non-transitory computer-readable storage medium may store one or more programs for execution that recognize the historical trajectories of one or more surrounding objects, predict future trajectories of the surrounding objects in an integrated manner based on the trajectories, and plan the travel trajectory of the electronic device 100 based on the predicted future trajectories of the surrounding objects.

[0066] The embodiments of this document and the terms used in the embodiments are not intended to limit the technology described in this document to a particular embodiment, but are to be construed to include various modifications, equivalents, and / or alternatives of a corresponding embodiment. In the description of the drawings, similar reference numerals may be used in similar elements. A singular term may include a plural term unless the context clearly defines otherwise. In this document, a term such as "A or B," "at least one of A and / or B," "A, B, or C," or "at least one of A, B, and / or C" may include all possible combinations of the listed elements together.Expressions such as "a first," "a second," "the first," and "the second" can modify corresponding elements regardless of their order or meaning and are used to distinguish only one element from the other and do not delimit the corresponding elements. When one (e.g., first) element is described as being "(functionally or communicatively) connected to" or "coupled to" another (e.g., second) element, one element may be directly connected to the other element or connected to the other element through another element (e.g., third element).

[0067] As used in this document, the "module" refers to a unit configured with hardware, software, or firmware and can be used interchangeably with a term such as logic, a logical block, a part, or a circuit. The module can be an integrated part, a minimal unit for performing one or more functions, or a part thereof. For example, the module can be configured with an application-specific integrated circuit (ASIC).

[0068] According to various embodiments, each of the described elements (e.g., module or program) may include a single unit or a plurality of units. According to various embodiments, one or more of the above-mentioned elements or operations may be omitted, or one or more other elements or operations may be added. Alternatively or additionally, multiple elements (e.g., modules or programs) may be integrated into one element. In such a case, the integrated elements may perform one or more functions of each of a plurality of elements that are identical or similar to those performed by a corresponding element of the plurality of elements prior to integration of the elements.According to various embodiments, other elements performed by a module, operation, or other program may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be performed in a different order or omitted, or one or more other operations may be added.

[0069] According to various embodiments, the electronic device can predict future trajectories of surrounding objects in an integrated manner based on the trajectories of the surrounding objects. In this case, the electronic device can predict future trajectories of an indefinite number of surrounding objects in an integrated manner. Accordingly, the electronic device can predict the driving trajectory of a vehicle based on traffic that varies in real time. Furthermore, the electronic device can ensure consistent performance and computational load when predicting various situations during driving and the driving trajectories of an indefinite number of surrounding objects, since the electronic device plans the driving trajectories based on the integrated and predicted future trajectories.

Claims

[1] A method of operating an electronic device, comprising: the recognition of historical trajectories of one or more surrounding objects; Predicting future trajectories of surrounding objects in an integrated manner based on the detected historical trajectories; and Planning a travel trajectory of the electronic device based on the predicted future trajectories of the surrounding objects, where the prediction of future trajectories includes: configuring a graph model based on the detected historical trajectories; and Prediction of future trajectories based on the graph model, wherein the graph model comprises a plurality of nodes and a plurality of edges connecting the nodes, where the nodes indicate the surrounding objects or the electronic device, where the edges indicate interactions between the surrounding objects and between the electronic device and each of the surrounding objects, or the directionality of the interactions, where edges connecting nodes indicating the surrounding objects contain bidirectionality, and wherein an edge connecting any of the nodes indicating the surrounding object and a node indicating the electronic device contains a unidirectionality directed from the electronic device to the surrounding object. [2] The operating method of claim 1, wherein predicting the future trajectories based on the graph model further comprises: relatively evaluating two nodes connected by an edge based on state information about the electronic device and state information about each of the surrounding objects and assigning importance to each of the nodes, wherein the two nodes are the electronic device and one of the surrounding objects or two of the surrounding objects; and the configuration of a graph convolution neural network using an adjacency matrix whose importance was taken into account. [3] The operating method according to claim 2, wherein predicting the future trajectories comprises predicting the future trajectories from the graph convolutional neural network based on a long-short-term memory (LSTM) network. [4] The operating method of claim 1, wherein detecting the historical trajectories comprises: Configuring a moving coordinate system based on a constant velocity model using state information about the electronic device; Representing the relative positions of the surrounding objects on the moving coordinate system; and the recognition of historical trajectories based on time-series positions. [5] The method of operation according to claim 4, wherein predicting the future trajectories comprises predicting the future trajectories on the moving coordinate system. [6] The operating method of claim 1, wherein the operating method of claim 1 further comprises controlling the electronic device based on the planned travel trajectory. [7] The operating method according to claim 1, wherein the electronic device is an autonomous vehicle, and where the surrounding objects are vehicles around the electronic device. [8] Electronic device comprising: at least one of a sensor module or a camera module; and a processor coupled to at least one of the sensor module or the camera module and configured to collect information about environmental situations through at least one of the sensor module or the camera module, wherein the processor is configured to recognize historical trajectories of one or more surrounding objects, predict future trajectories of the surrounding objects in an integrated manner based on the detected historical trajectories, and to plan a travel trajectory of the electronic device based on the predicted future trajectories of the surrounding objects, wherein the processor is configured to configure a graph model based on the detected historical trajectories, and predict future trajectories based on the graph model, and wherein the graph model comprises a plurality of nodes and a plurality of edges connecting the nodes, where the nodes indicate the surrounding objects or the electronic device, and where the edges indicate interactions between the surrounding objects and between the electronic device and each of the surrounding objects, or the directionality of the interactions, and where edges connecting nodes indicating the surrounding objects contain bidirectionality, and wherein an edge connecting any of the nodes indicating the surrounding object and a node indicating the electronic device contains a unidirectionality directed from the electronic device to the surrounding object. [9] The electronic device of claim 8, wherein the processor is configured to: to relatively evaluate two nodes connected by an edge based on state information about the electronic device and state information about each of the surrounding objects and to assign importance to each of the nodes, wherein the two nodes are the electronic device and one of the surrounding objects or two of the surrounding objects, and Configure a graph convolutional neural network using an adjacency matrix whose importance has been taken into account. [10] The electronic device of claim 9, wherein the processor is configured to predict the future trajectories from the graph convolutional neural network based on a long-short-term memory (LSTM) network. [11] The electronic device of claim 8, wherein the processor is configured to to configure a moving coordinate system based on a constant velocity model using state information about the electronic device, represent the relative positions of the surrounding objects on the moving coordinate system, and to recognize the historical trajectories based on the time-series positions. [12] The electronic device of claim 11, wherein the processor is configured to predict the future trajectories on the moving coordinate system. [13] The electronic device of claim 8, wherein the processor is configured to control the electronic device based on the planned travel trajectory. [14] The electronic device of claim 8, wherein the electronic device is an autonomous vehicle, and where the surrounding objects are vehicles around the electronic device.

Images