Method and apparatus for controlling driving of vehicle and vehicle

Abstract

To provide a method and an apparatus for controlling driving of a vehicle and the vehicle.SOLUTION: A method for controlling driving of a vehicle according to one embodiment includes receiving multi-view images including image frames at consecutive time points corresponding to a driving environment of the vehicle, extracting BEV features and map queries respectively corresponding to the consecutive time points for each of the image frames, generating a vectorized map by predicting and vectorizing map elements included in the image frames based on first memory tokens stored in a memory corresponding to queries of previous image frames of the image frames, the BEV features, and the map queries, and controlling the driving of the vehicle based on the vectorized map.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] The following embodiments relate to a method and an apparatus for controlling vehicle travel, and a vehicle. [Technical Field]

[0002] Advances in neural networks have enabled electronic devices in various fields to analyze input data using neural network-based models to extract and / or generate useful information. For example, constructing high-quality maps by predicting road geometric information (e.g., lanes, road signs) using neural networks is an important task for safe autonomous driving. Static map elements that make up high-quality maps are important information for autonomous vehicles, including applications such as lane keeping, path planning, and trajectory prediction. However, static map elements may be repeatedly occluded by various dynamic objects on the road. Summary of the Invention [Problem to be solved by the invention]

[0003] SUMMARY OF THE INVENTION It is an object of the present invention to provide a method and device for controlling the running of a vehicle, and a vehicle. [Means for solving the problem]

[0004] According to one embodiment, a method for controlling vehicle driving includes receiving a multi-view video including video frames at successive time points corresponding to a vehicle driving environment; extracting BEV features and a map query corresponding to each of the successive time points for each of the video frames; generating a vectorized map by predicting and vectorizing map elements included in the video frames based on a first memory token, the BEV features, and the map query stored in a memory corresponding to a query of a video frame previous to the video frame; and controlling the vehicle driving based on the vectorized map.

[0005] The step of extracting the BEV features and map query may include the steps of extracting perspective (PV) image features corresponding to the image frame using a backbone network, converting the perspective (PV) image features into the BEV features, extracting the frame-level map query used to construct the vectorized map based on the BEV features and a query corresponding to the image frame, and outputting the BEV features and the map query.

[0006] The step of generating the vectorized map may include the steps of: interpreting the first memory token; generating at least one of a map token including the map elements that constitute the vectorized map and a clip token including vectorized features corresponding to the image frame based on the map query, the BEV features, and the first memory token; and generating the vectorized map based on the map token.

[0007] The step of generating at least one of the map token and the clip token may include the steps of generating the clip token including a clue to the map element in a feature space corresponding to the video frame from the map query and the first memory token, updating the BEV feature using the clip token so that the BEV feature includes the hidden map element, and generating the map token using the updated BEV feature and the map query.

[0008] The size of the map query may be determined based on at least one of the size of the clip token, the number of the map elements, and the number of points per map element.

[0009] The step of updating the BEV features may include the steps of extracting a query from the BEV features, extracting a key and a value from the clip token, and updating the BEV features using the query, the key, and the value via a cross-attention network and a feedforward network.

[0010] Generating the map token may include generating the map token from the map query and the updated BEV features using a deformable attention network, a separated self-attention network, and a feedforward network.

[0011] The step of generating the map token may include extracting the query from the map query using the deformable attention network and generating the map token by obtaining values ​​from the updated BEV features.

[0012] The step of generating the vectorized map includes a step of predicting the map elements included in the video frame using a pre-trained neural network and generating the vectorized map vectorized for each instance, and the pre-trained neural network may include at least one of a second-first neural network that reads the first memory token in the memory or records a second memory token in the memory, and a second-second neural network that generates the vectorized map corresponding to a current frame among the video frames based on the map query, the BEV features, and the first memory token.

[0013] The step of generating the vectorization map may include the steps of recording the map tokens in the memory by the 2-1 neural network, and generating the vectorization map by passing a map token corresponding to the current frame among the map tokens through a prediction head.

[0014] The method for controlling vehicle driving may further include the steps of generating the second memory token by recording the map token and clip token in the memory using the second-1 neural network, and outputting the second memory token.

[0015] The 2-1 neural network can preserve time information corresponding to the previous video frame by interpreting the first memory token corresponding to the previous video frame and propagating it as an input to the 2-2 neural network.

[0016] The 2-1 neural network can associate information between clips through the propagation of clip tokens generated by the 2-2 neural network, thereby establishing intra-clip connections between the map elements.

[0017] The 2-1 neural network can generate the second memory token containing global map information through embedding of a learnable frame based on the map token and the clip token generated by the 2-2 neural network and store the second memory token in the memory.

[0018] The second-first neural network may generate the second memory token by combining a clip token, the map token, and the first memory token.

[0019] The 2-2 neural network can generate the vectorized map by outputting a map token corresponding to a current frame having a fixed time window corresponding to the length of the video frame based on the first memory token, the BEV features, and the map query.

[0020] The map elements may include at least one of a crosswalk, a road, a lane, a road boundary, a building, a boundary stone, and a traffic light included in the driving environment.

[0021] In one embodiment, an apparatus for controlling vehicle driving includes a communication interface that receives multi-view video including video frames at successive time points corresponding to a vehicle driving environment; a first neural network that extracts BEV features and map queries corresponding to each of the successive time points for each video frame; a second neural network that predicts and vectorizes map elements included in the video frame based on a first memory token, the BEV features, and the map query stored in a memory corresponding to a query of a video frame previous to the video frame, to generate a vectorized map; and a processor that controls the vehicle driving based on the vectorized map.

[0022] In one embodiment, a vehicle includes a sensor that captures multi-view images including image frames at successive points in time corresponding to the vehicle's driving environment; a neural network that extracts BEV features and map queries corresponding to each of the successive points in time for each image frame, and generates a vectorized map by predicting map elements included in the image frame based on a first memory token stored in a memory corresponding to a query of a previous image frame of the image frame, the BEV features, and the map query; and a processor that generates a control signal for driving the vehicle based on the vectorized map. [Effects of the Invention]

[0023] According to the present invention, a method and apparatus for controlling vehicle travel and a vehicle can be provided. [Brief explanation of the drawings]

[0024] [Figure 1] 4 is a flowchart illustrating a method for controlling vehicle travel according to an embodiment. [Figure 2] 1 is a flowchart illustrating a method for extracting BEV features and queries according to one embodiment. [Figure 3] 1 is a flowchart illustrating a method for generating a vectorized map according to one embodiment. [Figure 4]1 is a flowchart illustrating a method for generating at least one of a map token and a clip token according to one embodiment. [Figure 5] 1 is a diagram illustrating a framework of a device for controlling vehicle traveling according to an embodiment. [Figure 6] FIG. 10 is a diagram illustrating a clip token according to an embodiment. [Figure 7] FIG. 10 is a detailed implementation diagram of a 2-2 neural network according to an embodiment. [Figure 8] 4 is a flowchart illustrating a method for controlling vehicle travel according to an embodiment. [Figure 9] 1 is a diagram illustrating a framework of a device for controlling vehicle traveling according to an embodiment. [Figure 10A] 1 is a diagram illustrating the structure and operation of a device for controlling vehicle traveling according to an embodiment; [Figure 10B] 1 is a diagram illustrating an embodiment in which a control device according to an embodiment is mounted on a vehicle and recognizes lanes while the vehicle is traveling. [Figure 11] FIG. 10 is a diagram illustrating the operation of a feature exchange module according to an embodiment. [Figure 12] FIG. 10 is a diagram illustrating the operation of a map element generation module according to an embodiment. [Figure 13] FIG. 1 is a diagram illustrating a method for designing a loss for neural network training according to an embodiment. [Figure 14] 1 is a block diagram of a device for controlling vehicle traveling according to an embodiment; [Figure 15] 1 is a block diagram of a vehicle according to an embodiment; DETAILED DESCRIPTION OF THE INVENTION

[0025] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified in various forms. Therefore, the embodiments are not limited to the specific disclosed forms, and the scope of the present specification includes modifications, equivalents, or alternatives within the technical spirit.

[0026] Although terms such as "first" or "second" may be used to describe multiple components, such terms should be construed only to distinguish one component from the other components. For example, a first component may be designated as a second component, and similarly, a second component may be designated as a first component.

[0027] When a component is referred to as being "coupled" or "connected" to another component, it should be understood that although it is directly coupled or connected to the other component, there may be other components in between.

[0028] The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, phrases such as "at least one of A or B" and "at least one of A, B, or C" may each include any one or all possible combinations of the items listed with the phrase. In this specification, the terms "comprise" or "have" and the like indicate the presence of a feature, numeral, step, operation, element, part, or combination thereof described in the specification, and should be understood as not precluding the possibility of the presence or addition of one or more other features, numerals, steps, operations, elements, parts, or combinations thereof.

[0029] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which the present invention belongs. Commonly used predefined terms should be interpreted as having a meaning that matches the meaning they have in the context of the relevant art, and should not be interpreted as having an ideal or overly formal meaning unless expressly defined in this specification.

[0030] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals will be used to designate the same elements regardless of the reference numerals, and redundant description thereof will be omitted.

[0031] The embodiments described below may be utilized in an augmented reality navigation system, such as a smart vehicle, to generate visual information to assist in the operation of an autonomous vehicle. The embodiments may also be utilized in camera-based systems, such as front-facing cameras, multi-cameras, and surround-view monitor (SVM) systems for autonomous driving or advanced driver assistance systems (ADAS), and in systems that utilize lidar and / or radar for autonomous driving or ADAS. Additionally, the embodiments may be used to interpret visual information in devices including intelligent systems, such as head-up displays (HUDs) installed for in-vehicle driving assistance or fully autonomous driving, to assist in safe and comfortable driving. The embodiments may be applied to, for example, autonomous automobiles, intelligent vehicles, smartphones, navigation systems, and mobile devices.

[0032] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals will be used to designate the same elements regardless of the reference numerals, and redundant description thereof will be omitted.

[0033] 1 is a flowchart illustrating a method for controlling vehicle travel according to one embodiment. In the following embodiment, the steps are performed in order, but not necessarily in order. For example, the order of the steps may be changed, or at least two steps may be performed in parallel.

[0034] Referring to FIG. 1, a device for controlling vehicle travel according to one embodiment (hereinafter referred to as a "control device") can control vehicle travel based on a vectorized map through steps 110 to 140.

[0035] In step 110, the control device receives a multi-view image including image frames at successive points in time corresponding to the vehicle's driving environment. The "vehicle" refers to an intelligent vehicle equipped with an advanced driver assistance system (ADAS) and / or an automatic driving system (AD) that recognizes and / or determines the driving situation using sensors, image processing devices, communication devices, etc., to control the vehicle's operation or notify the driver. The intelligent driver assistance system and / or the autonomous driving system can recognize fixed objects on the road, road markings including lanes, and / or road signs, mainly from the driving image captured by the camera.

[0036] The vehicle may map the estimated target distance to a pre-constructed map based on the image frame acquired from the camera and correct the vehicle's position, while correcting the lanes of the road recognized by the vehicle and related driving information. In addition, the vehicle may be provided with various driving information including the recognized lanes related to the road through navigation.

[0037] Hereinafter, in this specification, the term "vehicle" is used to encompass all types of transportation means that have a drive engine and are used to move people or goods, such as a car, bus, motorcycle, or truck. A vehicle may also be referred to as an "ego vehicle."

[0038] The multi-view image may be a set of synchronized video frames at consecutive time points. The consecutive time points refer to, for example, three consecutive time points, such as time point t-2, time point t-1, and time point t. If time point t-1 corresponds to the current time point, time point t-2 corresponds to the previous time point, and time point t corresponds to the next time point. The video frames may be, for example, video frames of a driving video including a vehicle, a lane, a curb, a sidewalk, the surrounding environment, and / or a fixed object. The video frames may be captured using a camera (e.g., a single camera or multiple cameras) mounted on the front of a vehicle (e.g., the host vehicle), but are not necessarily limited to this. It is assumed here that calibration information of the camera is known in advance. The camera may include, for example, a mono camera, a vision sensor, an image sensor, or a device performing a similar function. Alternatively, the video frames may be captured by a camera included in the control device or a device other than the control device. The image frame includes not only fixed objects that do not move and are fixed in a certain position, such as roads, road signs, and road lanes, but also moving objects such as other vehicles and pedestrians traveling around the vehicle. A "road" refers to a path on which multiple vehicles, including the vehicle, travel, and includes various types of roads, such as expressways, national highways, local roads, express national highways, and motorways. A road includes one or more lanes. A road on which a vehicle is traveling is called a "traveling road." A "lane" refers to a road space divided by lanes marked on the road surface. A lane is divided by lanes on the left and right sides of the lane. Road signs include, but are not limited to, at least one of a speed sign, a distance sign, and a guidepost.

[0039] In step 120, the control device extracts BEV (Bird's Eye View) features and map queries corresponding to successive time points for each video frame. The BEV features and map queries may be extracted, for example, by a first neural network 510 shown in FIGS. 5 and 7 below. The method by which the control device extracts the BEV features and queries will be described in more detail with reference to FIGS. 2 and 5 below.

[0040] In step 130, the control device generates a vectorized map by predicting and vectorizing map elements included in the video frame based on the first memory token, BEV features, and map query stored in the memory in response to a query of the previous video frame. The vectorized map is generated by recognizing road information around the vehicle and generating a local map in vector form, such as centerlines, lanes, crosswalks, and road boundary information. The vectorized map may include, for example, map element classes, map element coordinate information, and map element directions. The map elements may include, for example, at least one of crosswalks, roads, lanes, road boundaries, buildings, boundary stones, road signs, and traffic lights included in a high-resolution map corresponding to the driving environment or a vectorized map, but are not necessarily limited thereto. The map elements may have unique indexes corresponding to successive time points. The vectorized map may be in the form of a point cloud, such as vectorized map 570 shown in FIG. 5, but is not necessarily limited thereto.

[0041] The control device may predict map elements included in a video frame using, for example, a pre-trained neural network and generate a vectorized map by vectorizing each instance. Here, the pre-trained neural network may include, for example, at least one of the second-first neural networks 540-1, 540-3, and the second-second neural network 540-2 shown in FIGS. 5 and 7. Here, an "instance" corresponds to an individual component of a high-resolution map or a vectorized map. Examples of instances include map elements such as roads, lanes, traffic lights, curbs, and crosswalks, but are not limited to these. In addition, the instances may further include various objects included in a video of a vehicle traveling.

[0042] As will be described in more detail below, the 2-1 neural network 540-1 reads the first memory token from memory using a token summarizer. The 2-1 neural network 540-3 writes the second memory token to memory using a token summarizer. Here, the token summarizer can select and summarize valid and important tokens from among the map token 550, clip token 560, and the first memory token. The 2-2 neural network 540-2 generates a vectorized map corresponding to the current frame of the video frames based on the map query, BEV features, and the first memory token. The control device can record the map token in memory using the 2-1 neural network 540-3, and generate a vectorized map by passing the map token corresponding to the current frame from the map tokens recorded in memory through a prediction head.

[0043] As will be described in more detail below, the second-first neural networks 540-1 and 540-3 store time information corresponding to the previous video frame by reading the first memory token corresponding to the previous video frame and propagating it as input to the second-second neural network 540-2. The second-first neural network 540-3 can associate inter-clip information through the propagation of the clip tokens generated by the second-second neural network 540-2 and establish intra-clip associations between map elements. The second-second neural network 540-2 can also generate a vectorized map by outputting a map token corresponding to the current frame having a certain time window corresponding to the length of the video frame based on the first memory token, BEV features, and map query.

[0044] In step 140, the control device controls the vehicle's driving based on the vectorized map. The control device can control the vehicle's driving by generating various control parameters for the autonomous driving of the vehicle, such as vehicle path setting, lane detection, vehicle steering, vehicle driving, and vehicle driving assistance, based on the vectorized map.

[0045] 2 is a flowchart illustrating a method for extracting BEV features and queries according to an embodiment. Referring to FIG. 2, a control device according to an embodiment extracts BEV features and queries through steps 210 to 240.

[0046] In step 210, the control device extracts perspective (PV) image features corresponding to the image frame using, for example, a backbone network 511 shown in Fig. 5. The backbone network 511 corresponds to an encoder that encodes the image features.

[0047] In step 220, the control device converts the perspective view (PV) image features extracted in step 210 into BEV features. The control device may convert the perspective view (PV) image features into BEV features, for example, using a PV to BEV converter 513 shown in FIG. 5. The PV to BEV converter 513 may convert the perspective view (PV) image features into feature vectors in BEV (Bird's Eye View) space, in other words, BEV features, using, for example, an inverse perspective mapping (IPM) method, but is not necessarily limited to this. The inverse perspective mapping method can remove the perspective effect from an input image (image frame) and / or a segmentation image having a perspective effect and convert position information on the image plane into position information in a world coordinate system.

[0048] In step 230, the control device extracts a frame-level map query to be used to construct a vectorized map based on the BEV features converted in step 220 and a query (Q) corresponding to the video frame. The control device can extract the frame-level map query using, for example, a map decoder 515 shown in FIG. 5. The map decoder 515 may be, for example, a prediction head that decodes feature vectors and outputs instances. The size of the map query is determined based on at least one of, for example, but not necessarily limited to, the size of a clip token, the number of map elements, and the number of points per map element.

[0049] In step 240 , the controller outputs the BEV features transformed in step 220 and the map query extracted from step 230 .

[0050] 3 is a flowchart showing a method for generating a vectorized map according to an embodiment. Referring to FIG. 3, a control device according to an embodiment can generate a vectorized map through steps 310 to 330.

[0051] In step 310, the control device reads a first memory token. The first memory token is information corresponding to a previous video frame. The first memory token is a token stored in memory in response to a query for a previous video frame. The first memory token may be a clip token and a map token stored in memory in response to a query for a previous video frame, or may be a token that is a modified version of these (clip token and map token).

[0052] In step 320, the control device generates at least one of a map token including map elements constituting a vectorized map and a clip token including vectorized features corresponding to a video frame based on the map query and BEV features output in step 240 and the first memory token read in step 310. Here, a "clip token" includes cues to temporal map elements in feature space. The method by which the control device generates at least one of a map token and a clip token will be described in more detail below with reference to FIG. 4.

[0053] In step 330, the controller generates a vectorized map based on the map tokens generated in step 320. The controller can generate the vectorized map by decoding the map tokens with various learning matrices.

[0054] 4 is a flowchart illustrating a method for generating at least one of a map token and a clip token according to an embodiment. Referring to FIG. 4, a control device according to an embodiment can generate at least one of a map token and a clip token through steps 410 to 430.

[0055] In step 410, the controller generates a clip token from the map query and the first memory token, the clip token including a clue to a map element in the feature space corresponding to the video frame. The controller may generate the clip token, for example, by clip token generator 542 shown in Figures 5 and 7. Clip tokens are described in more detail below with reference to Figure 6.

[0056] In step 420, the control device updates the BEV features using the clip tokens generated in step 410 to include map elements where the BEV features are hidden. Because maps are static information, they have the same information or values ​​at the same locations throughout the map, even at different times. The control device can update the BEV features to include map elements where the BEV features are hidden by utilizing map information acquired at other times using the above features at the current time. Here, hidden map elements may also be referred to as "occluded map elements," as they are map elements that are hidden by dynamic objects such as surrounding vehicles. In this way, the control device can generate a vectorized map with high accuracy by utilizing image information from multiple times.

[0057] The control device can update the BEV features using the clip token generated in step 410, for example, by the BEV Updater 544 shown in Figures 5 and 7. The control device, for example, extracts a query (Q) from the BEV features and a key (K) and a value (V) from the clip token. The control device can update the BEV features through a cross-attention network and a feedforward network using the query, key, and value.

[0058] In step 430, the control device generates a map token using the BEV features updated in step 420 and the map query extracted from step 230. The control device may generate the map token using, for example, but not limited to, the Map Generator 546 shown in Figures 5 and 7. The control device can generate the map token from the map query and the updated BEV features using a deformable attention network, a self-attention network, and a feedforward network. The control device can generate the map token by extracting a query from the map query using the deformable attention network and obtaining a value from the updated BEV features.

[0059] 5 is a diagram illustrating the framework of a device for controlling vehicle driving according to an embodiment. Referring to FIG. 5, the framework of a control device 500 is shown, which outputs a clip-level vectorized high-resolution map (HD map) generated from clip-level multi-view video 501 according to an embodiment.

[0060] The control device 500 can generate a high-quality clip-level vectorized map, or vectorized map 570, from clip-level multi-view images 501. The vectorized map 570 is constructed by predicting instance-wise vectorized representations of static map elements (e.g., crosswalks, lane markings, road boundaries). Static map elements are important information for autonomous vehicle navigation, such as lane keeping, path planning, and trajectory prediction.

[0061] Hereinafter, the term "clip" as used in this specification refers to a set of features at multiple (e.g., three or five) consecutive points in time. Also, a "token" corresponds to a vectorized feature for one clip and / or map element.

[0062] For example, when a set of synchronized multi-view images (e.g., clip input) is input, the control device 500 can sequentially construct a clip-level vectorized high-definition (HD) map. The control device 500 can first extract frame-level BEV features 525 and map queries 520 using a first neural network 510. The control device 500 can extract map queries 520 and BEV features 525 independently for each image frame using the first neural network 510. The BEV features 525 are features that match the image view viewed by the user and can be expressed in various forms, such as pixel value, color, texture, shape, etc. The BEV features 525 play an important role in understanding and analyzing the content of the image. The BEV features 525 may be referred to as "rasterized features" because they are pixel-based image features. The first neural network 510 can be, for example, a frame-level MapNet and can use the MapTRv2 architecture, but is not necessarily limited thereto.

[0063] The operation of the first neural network 510 will now be described in more detail.

[0064] The first neural network 510 extracts a clip set of BEV features 525 and a map query 520 from the synchronized multi-view video 501. The first neural network 510 includes, for example, a backbone network 511, a PV to BEV converter 513, and a map decoder 515.

[0065] The backbone network 511 can extract perspective (PV) image features corresponding to the video frame. The backbone network 511 can be, for example, but is not limited to, a ResNet50. The first neural network 510 can convert the perspective (PV) image features extracted by the backbone network 511 into BEV features 525 via a PV-to-BEV converter 513.

[0066] The PV to BEV converter 513 converts the image features of a perspective view (PV) corresponding to the multi-view image 501 into BEV features 525. The PV to BEV converter 513 may, for example, use an inverse perspective mapping (IPM) method to convert the image features of the perspective view (PV) into feature vectors in BEV (Bird's Eye View) space, in other words, BEV features 525, but is not necessarily limited to this.

[0067] The PV to BEV converter 513 can extract integrated 3D BEV features 525 encompassing the vehicle's surroundings using various perspective (PV) to bird's eye view (BEV) conversion methods. The PV to BEV converter 513 may use, for example, LSS (Lift, Splat, Shoot)-based BEV feature pooling. The BEV features 525 can be used to query a map in the map decoder 515.

[0068] The map decoder 515 can output a frame-level map query 520 that can be directly used to construct a high-quality vectorized map 570 via the BEV features 525. The map decoder 515 can decode and predict the map query 520 corresponding to the map element from the BEV features 525 via a task-specific head.

[0069] The first neural network 510 outputs BEV features 525, which are the result of the PV-to-BEV converter 513, and a map query 520, which is the result of the map decoder 515, respectively.

[0070] The controller 500 then applies the previously extracted information (e.g., map query 520 and BEV features 525) as input to a second neural network 540 to generate clip tokens 560. Clip tokens 560 contain clip-level temporal map information and can directly interact with the compact BEV features 525.

[0071] Here, the control device 500 can set a long-term temporal relationship with the current video frame using a first memory token 530 created corresponding to a previous clip and stored in the memory 527. Here, the first memory token 530 can correspond to a clip query.

[0072] The control device 500 can generate clip tokens 560 including clues to temporal map elements in a compact feature space by the second neural network 540 using the first memory tokens 530 and map queries 520 read from the memory 527. Here, the clip tokens 560 correspond to clip-level map information. The clip tokens 560 will be described in more detail with reference to FIG. 6 below. The control device 500 can update the BEV features 525 using the clip tokens 560.

[0073] The control device 500 updates the BEV feature 525 using the clip token 560 so that the BEV feature 525 includes the hidden map element. The control device 500 updates the BEV feature 525 using the clip token 560 so that the hidden map element becomes visible. The size of the clip token 560 may be, for example, 50 (number of vectors) x 256 (dimensions per feature).

[0074] The control device 500 can update the BEV features 525 using the BEV updater 544 of the second neural network 540 and generate a high-quality clip-level vectorized map 570 through the updated BEV features 555. The control device 500 can construct the high-quality clip-level vectorized map 570 by extracting map tokens 550 using the updated BEV features 555.

[0075] After inferring clip-level vectorization map 570, controller 500 generates a new second memory token using second-first neural network 540-3 using clip token 560, map token 550, and the current memory token (e.g., first memory token 530). Controller 500 can generate the second memory token using second-first neural network 540-3 by recording map token 550 and clip token 560 in memory 527.

[0076] For example, the size of the map token 550 may be, for example, 3(T) x 50 (number of vectors) x 256 (dimensions per feature). The result of applying a learning matrix to the map token 550 is 3(T) x 50 (vector map) x [2(x, y) x 20 + number of categories]. The size of the updated BEV features 555 may be, for example, 3(T) x 100 (width) x 200 (height) x 256 (dimensions per feature), where T represents the time window (length of the clip).

[0077] For example, if the first memory token has a size of 96 (number of vectors) × 256 (dimensions per feature), then 3(T) × 50 map queries 520 and 50 clip queries 530, totaling 96 first memory tokens and 200 queries (96 + 200), are reduced to 50 during the memory interpretation process.

[0078] The memory read process may be performed in the form of, for example, "1,2,3→4,5,6→7,8,9→...", "1,2,3→2,3,4→3,4,5→...", or "1,1,1→1,1,2→1,2,3→2,3,4→3,4,5→...".

[0079] The control device 500 outputs a second memory token that can be used to provide temporal cues for clip-level inference corresponding to a subsequent video frame that follows the current video frame.

[0080] The second neural network 540 includes, for example, a second-first neural network 540-1, a second-second neural network 540-2, and a second-first neural network 540-3, where the second-first neural network 540-1 and the second-first neural network 540-3 correspond to the same neural network.

[0081] The second-first neural networks 540-1 and 540-3 read the first memory token 530 stored in the memory 527 or record the second memory token in the memory 527. We refer to the case where the second-first neural network 540-1 is used to read the first memory token 530 in the memory 527, and we also refer to the case where the second-first neural network 540-3 is used to record the second memory token in the memory 527.

[0082] The second-first neural networks 540-1 and 540-3 propagate the tokens of the previous clip input as the next clip input and store the dense time information of the previous frame. The second-first neural network 540-3 propagates the map token 550 brought by the second-second neural network 540-2.

number

number

[0083] The second-first neural networks 540-1, 540-3 use, for example, a token summarizer to interpret the first memory token 530 stored in the memory 527, or to record the second memory token in the memory 527.

[0084] The second-second neural network 540-2 may generate a vectorized map corresponding to a current frame (e.g., t frame). The second-second neural network 540-2 generates a vectorized map 570 corresponding to the current one of the video frames based on the map query 520, the BEV features 525, and the first memory tokens 530. The second-second neural network 540-2 may include, for example, a clip token generator 542, a BEV updater 544, and a map generator 546. The structure and operation of the second-second neural network 540-2 are described in more detail below with reference to FIG. 7.

[0085] The second-first neural network 540-3 can establish intra-clip connections between map elements by sending inter-clip tokens to the BEV features 525 of the next video frame. The second-first neural network 540-3 generates a second memory token by combining a clip token 560, a map token 550, and a first memory token 530. The second-first neural network 540-3 can generate a second memory token containing global map information through learnable frame embedding based on the map token 550 and clip token 560 generated by the second-second neural network 540-2 and store the second memory token in memory 527. The second-first neural network 540-3 can manage the map information for the second memory token stored in memory 527 using TTM.

[0086] The second-first neural network 540-3 can generate second memory tokens using tokens (eg, map token 550 and clip token 560) generated in the second-second neural network 540-2 to construct global relationships.

[0087] The interpretation process in the second neural network 540 will be described as follows. The second neural network 540 interprets the first memory token 530 and the map query 520 to generate a compact clip token 560 containing global map information. Here, the second neural network 540 interprets the first memory token 530 and the map query 520 using a token summarizer that efficiently selects information tokens as input following the TTM. The second neural network 540 performs positional embedding using a position-based memory addressing scheme. Positional embedding will be described in more detail below.

[0088] Since memory is not available at the first clip level pipeline of the second neural network 540, we first create the first memory token 530 through learnable clip embedding.

[0089] The recording process in the second neural network 540 will now be described.

[0090] The second neural network 540 can use the token summarizer to perform a recording operation to the memory.

number

number

number

number

[0091] The newly generated second memory token through the recording operation is used in the read operation for the first layer of the second neural network 540-2 in the subsequent clip-by-clip processing. The control device 500 performs the recording process using the same position-based memory addressing scheme as used in the read operation. The recording operation is applied to the last layer of the second neural network 540-2, generating a new second memory token, thereby creating a clip token 560.

number

number

[0092] The controller 500 can efficiently execute the clip-level pipeline by avoiding overlapping calculations due to temporal strides while the second neural network 540 builds the global map relationships.

[0093] The second neural network 540 can efficiently perform online inference by collecting clip-level temporal map information and learning discriminatory associations.

[0094] In one embodiment, by introducing a clip-level pipeline, a second neural network 540 can be inferred online that can efficiently construct a global map by mapping BEV features 525 within a clip set and propagating the map information to subsequent timestamps.

[0095] The second neural network 540 performs clip-level inference NT / S times for a sequence of NT frames by performing efficient inference based on a clip-level pipeline with a temporal stride of S. The structure and operation of the second neural network 540-2 are described in more detail below with reference to FIG.

[0096] The second neural network 540 is a framework designed to expose hidden map information that cannot be captured by frame-level BEV features 525 alone. To avoid excessive computation, the controller 500 can perform inference on a clip-by-clip basis using a time window (length of the clip) T and a stride S.

[0097] Since the first neural network 510 is based on the frame-level MapTRv2 architecture, the first neural network 510 may be trained using, for example, loss functions used in MapTR and MapTRv2, and may further be trained using one2one loss and auxiliary dense prediction losses.

[0098] For example, classification loss, point2point loss, and edge direction loss may be used to train the second neural network 540.

[0099] 6 is a diagram illustrating clip tokens according to an embodiment. Referring to FIG. 6, a diagram 600 is shown illustrating clip tokens 560 corresponding to BEV features at three consecutive time points (e.g., time points t-2, t-1, and t) according to an embodiment.

[0100] As mentioned above, a set of video features at successive points in time is called a “clip,” and a vectorized feature for one clip is called a “token.” In other words, a clip token 560 corresponds to a vectorized feature for video at successive points in time, and includes cues for temporal map elements in the feature space.

[0101] The controller generates a compact clip token 560 constructed from the temporal map information in the clip input, and updates the BEV features corresponding to each successive point in time using the clip token 560 to represent the hidden map elements seen in a particular frame.

[0102] The controller indicates hidden map elements through the interaction between clip tokens 560 with higher-level map information and BEV features. The controller can associate information between clips through the propagation of clip tokens 560 to establish intra-clip associations between map elements. The controller can update and visualize hidden (or occluded) BEV features, for example, through one-dimensional PCA (Principal Component Analysis) projection based on clip tokens 560, where the BEV features are extracted from a second neural network.

[0103] 7 is a detailed implementation diagram of a 2-2 neural network according to one embodiment. Referring to FIG. 7, a diagram 700 is shown for explaining the detailed structure and operation of a 2-2 neural network according to one embodiment. The 2-2 neural network 540-2 may also be referred to as an "intra-clip unveiler."

[0104] The second-second neural network 540-2 may be composed of a series of L layers. The second-second neural network 540-2 may include, for example, a clip token generator 542, a BEV updater 544, and a map generator 546. The clip token generator 542, the BEV updater 544, and the map generator 546 each include various attention layers and feedforward layers. Each attention layer and feedforward layer may be implemented with, for example, standard layer normalization, dropout, and residual connections.

[0105] The second neural network 540-2 is a neural network for the frame-level map query 520. map , BEV characteristics 525 FBEV, and memory readout 710 U Read , can be used to output map tokens 550, updated BEV features 555, and clip tokens 560.

[0106] The clip token generator 542 generates a compact clip token 560 using the first memory token 530 read from memory and the map query 520. The clip token generator 542 extracts a query (Q) from the first memory token 530 and a key (K) and a value (V) from the map query 520.

[0107] The query, key, and value extracted by the clip token generator 542 can be used to generate clip tokens 560 via positional embedding through the cross-attention layer, self-attention layer, and feedforward layer of the clip token generator 542.

[0108] The clip token generator 542 generates a frame-level map query 520

number

number

[0109] The BEV updater 544 receives the CLIP token 560 generated by the CLIP token generator 542.

number

number

[0110] The BEV updater 544 updates the BEV feature 525 via the clip token 560 corresponding to the hidden region of the original BEV feature 525, and generates the updated BEV feature 555.

number

number

[0111] The map generator 546 generates the updated BEV features 555 generated by the BEV updater 544.

number

number

[0112] The map generator 546 generates map queries 520 using a configurable attention layer.

number

number

number

number

[0113] The standard transformer structure is permutation-invariant, but can use position information supplemented with temporal information to predict map elements at the clip level.

[0114] The controller 500 can use a fixed 3D sinusoidal location embedding for the BEV features. The controller 500 may use the learnable location embedding used in the frame-level first neural network along with a newly defined learnable temporal location embedding for the map token. The controller 500 defines a new learnable location embedding for the clip token. Similarly, the controller 500 defines learnable location embeddings for the memory tokens (e.g., the first memory token 530 and the second memory token) used to interpret and record the second-first neural networks 540-1, 540-3.

[0115] The memory of the second neural network 540-3 contains a map token 550.

number

number

[0116] FIG. 8 is a flowchart showing a method for controlling vehicle traveling according to an embodiment, and FIG. 9 is a diagram showing the framework of a device for controlling vehicle traveling according to an embodiment.

[0117] 8 and 9, the control device according to one embodiment controls the running of the vehicle based on the vector map through steps 810 to 850.

[0118] In step 810, the control device receives sensor information 910 including synchronized multi-view images (e.g., 910-1, 910-2, 910-3) including image frames at successive times (e.g., time1, time2, time3) corresponding to the vehicle's driving environment. The sensor information may be detected by a variety of single or multiple sensors, such as a camera sensor, a vision sensor, a lidar sensor, a multi-camera sensor, and a radar sensor.

[0119] In step 820, the control device extracts features corresponding to successive time points from the sensor information 910 received in step 810. The control device may extract the features by, for example, but not limited to, the first neural network 510 or the feature extraction module 930 described above with reference to FIG.

[0120] In step 830, the control device generates map elements corresponding to the driving environment by propagating the features extracted in step 820 for each time point into a pre-trained neural network 950. By inputting the features extracted in step 820 into the pre-trained neural network 950, the control device can obtain map information corresponding to the occluded area at each time point.

[0121] More specifically, the control device can grasp map information at different times, including each point in time, by combining (or exchanging) the features extracted in step 820. The control device can grasp map information at different times by combining the features extracted using a pre-trained neural network 950. The control device updates the features extracted in step 820 using the map information at different times. Here, the pre-trained neural network 950 may be, but is not limited to, one of a Transformer and a 3D convolutional neural network (CNN), which can reference information for several hours. The pre-trained neural network 950 may be, for example, the second neural network 540 described above with reference to FIG. 5.

[0122] The control device can define prompts to be used to generate map elements for map information for each time point. The control device can acquire map information using the prompts defined for each time point and the combined (or interacted) features. The control device can simultaneously generate map elements for all time points using the prompts defined for each time point and the updated features. The control device can simultaneously generate map elements for all time points from prompts based on the updated features using, for example, a deep neural network-based transformer decoder. Here, among the prompts defined for each time point, prompts with the same index can generate the same map element.

[0123] The control device can generate map elements by defining the same morphological characteristics of the map elements at successive time points using the acquired map information. The map elements have unique indexes corresponding to each time point. Map elements generated with the same index at each time point can generate the same map elements as map elements generated at different times.

[0124] In step 840, the control device generates a vectorized map for each time point using the map elements generated in step 830.

[0125] In step 850, the control device controls the vehicle's travel based on the vector map generated in step 840.

[0126] 10A is a diagram illustrating the structure and operation of a device for controlling vehicle travel according to an embodiment. Referring to FIG. 10A, a diagram 1000 is shown illustrating the structure and operation of a control device 1000 according to an embodiment.

[0127] The control device 1000 enables the neural network 1030 to learn the same morphological characteristics of the map elements that make up the vectorized map at successive time points based on features extracted by the feature extraction module 930 from the single and / or multiple sensor information 910 input at successive time points, thereby enabling the neural network 1030 to generate a high-performance vectorized map 1050 at each time point.

[0128] The neural network 1030 includes a feature exchange module 1033 for feature exchange between single and / or multiple sensor information 910 at successive time points, and a map element generation module 1036 for predicting and / or generating map elements that constitute a vectorized map at successive time points. The control device 1000 analyzes the temporal consistency of the map elements at successive time points, and makes the neural network 1030 learn the same morphological characteristics of the map elements.

[0129] Acquisition of single and / or multiple sensor information 910 and feature extraction by feature extraction module 930 may be performed by feature extractor 1010 .

[0130] The feature extraction unit 1010 extracts features from single or multiple sensor information input at successive times.

[0131] The feature exchange module 1033 exchanges the features extracted at each time point to obtain information about map elements that exist at different times.

[0132] The map element generation module 1036 can define prompts for generating map elements that make up a vectorized map for each time period, and generate map elements based on characteristics grasped through interactions between map elements at different times.

[0133] The control device 1000 can predict the vectorized map 1050 corresponding to each time point by the map elements. The control device 1000 can train the neural network 1030 based on the comparison result between the vectorized map 1050 and the correct GT.

[0134] The control device 1000 can generate a vectorized map for each time point by combining or interchanging features between each time point using the same sensor information input at successive times. The control device 1000 analyzes the temporal consistency between map elements at successive times, allowing the neural network 1030 to learn the same morphological characteristics.

[0135] More specifically, the feature extractor 1010 receives single and / or multiple sensor information 910 (e.g., vehicle surroundings information (Enc)) from a single or multiple sensors at successive points in time, and extracts features F at each point in time through a feature extraction module (Enc) 930 as shown in Equation (1) below. t can be extracted.

[0136]

number

[0137] Various types of sensors capable of recognizing map information, such as a camera, lidar, or radar, may be used as a single or multiple sensors. The feature extraction module 930 may be, for example, a deep neural network-based encoder. The feature extraction module 930 is a neural network or encoder of various types capable of extracting features from sensor information, such as a convolutional neural network or a transformer.

[0138] The control device 1000 can generate an improved feature Ft^ for each time point by exchanging the feature Ft at each time point as shown in Equation (2) below through the feature exchange module (Comm) 1033.

[0139]

number

[0140] The control device 1000 can complement map information blocked by an obstacle at the current time point with features at other times through feature exchange in images at each time point. The feature exchange module (Comm) 1033 can use various neural network modules capable of spatiotemporal data processing, such as 3D convolutional neural networks (CNNs) based on deep neural networks and 3D transformers. The operation of the feature exchange module (Comm) 1033 will be described in more detail with reference to FIG. 11 below.

[0141] The map element generation module 1036 defines N prompts Qn,t for generating map elements for each time point (t). Here, the prompts Qn,t can generate N corresponding map elements Mn,t for each time point by referring to the features Ft^ improved through the map element generation module (Dec) 1036 as shown in Equation (3).

[0142]

number

[0143] Here, yt corresponds to the prediction result of the map element generation module 1036.

[0144] The map element generation module 1036 simultaneously generates map elements for all consecutive time points by referring to features at all consecutive time points, thereby analyzing the temporal consistency between map elements and enabling the neural network 1030 to learn the same morphological characteristics. The map element generation module 1036 may be, for example, a deep neural network-based transformer decoder, but is not limited to this. The prompt defines feature information for each time point, and prompts with the same index at each time point generate the same map element. The map element generation module 1036 can generate map elements by referring to features exchanged via the feature exchange module 1033. Here, map elements with the same index at each time point can generate the same map element at different times. The map element generation module 1036 collects the generated map elements to finally generate a vectorized map 1050 corresponding to each time point. The operation of the map element generation module 1036 will be described in more detail with reference to FIG. 12 below.

[0145] The control device 1000 compares and analyzes the generated vectorized maps 1050 at each time point to calculate a temporal consistency loss (TCL) as shown in the following equation (4).

[0146]

number

[0147] Here, T indicates T consecutive time points, and yt indicates the prediction result at time point t.

[0148] The control device applies a loss function L that makes the prediction results of two time points similar using a matching relationship based on the prediction results for the vectorized map (e.g., a comparison result between the generated vectorized map and the correct GT). Here, various loss functions, such as L1 loss, may be used as the loss function. The control device can calculate a final loss value by repeating the above process for all time points (e.g., tN to t).

[0149] The control device can train the feature extraction module 930, the feature exchange module 1033, and the map element generation module 1036 according to the comparison result between the generated vectorized map and the correct GT. Here, there are two types of losses used in training the feature extraction module 930, the feature exchange module 1033, and the map element generation module 1036.

[0150] One of the losses can be the loss between a commonly known predicted value (e.g., a map element) and the correct GT, which can be induced to generate a vectorized map that looks the same as the correct answer. Here, the map element may have a unique element index for each time point. The control device matches the correct GT with the map element so that elements with the same index at different times generate the same map element. For example, if the map element corresponding to the nth index at time 1 (time 1) generates the lane immediately to the left of the vehicle, the control device can match the map element corresponding to the nth index at time 2 (time 2) with the correct answer so that it also generates the lane immediately to the left of the vehicle. In this way, if the same index generates the same map element, the control device connects the map elements at different times, enabling tracking of map elements at consecutive times.

[0151] The other loss is a loss that induces the same map elements to be generated in the same form in vectorized maps generated at different times.A variety of information can be used during learning, and by using vehicle movement information, vectorized maps generated at different times can be warped to different times.

[0152] The control device can eliminate flickering or distortion that may occur in vectorized maps generated at successive times by using a loss that ensures that vectorized maps warped at different times and the vectorized map at the current time produce the same result.

[0153] The control device 1000 can calculate two types of losses, supervised learning and unsupervised learning, using the instance matching method and apply them to the training of the neural network 1030. The training method of the neural network 1030 will be described in more detail with reference to FIG. 13 below.

[0154] FIG. 10B is a diagram for explaining an embodiment in which a control device according to an embodiment is mounted on a vehicle and recognizes lanes while the vehicle is traveling.

[0155] When the control device 1000 is installed in a vehicle and generates the vectorized map 1050 online, the control device 1000 can generate the vectorized map 1050 for each time point as shown in FIG. 10B.

[0156] For example, since there are no features stored at previous time points at time 1 (time1), the control device 1000 can extract the features at time 1 (time1) and use them as input to the feature exchange module 1033, while simultaneously storing the features at time 1 (time1) in memory.

[0157] At time 2 (time2), the control device 1000 does not newly extract features for time 1 (time1), but uses the features for time 1 (time1) stored in memory at the previous time as input to the feature exchange module 1033. The control device 1000 may store features for time 2 (time2) in memory.

[0158] Furthermore, at time 3 (time3), the control device 1000 does not extract new features from time 1 (time1) and time 2 (time2), but uses features stored in memory at previous times (e.g., features from time 1 (time1) and features from time 2 (time2)) and uses them as input to the feature exchange module 1033.

[0159] The control device generates vectorized maps corresponding to successive time points (e.g., time1, time2, time3) as described above, and then controls vehicle driving using the vectorized map (time3) corresponding to time point 3 (time3), which is the current time point. Here, the vehicle driving control may include, for example, lane recognition, driving route setting, and driving parameter setting, which assist vehicle driving, or control parameter setting, which controls vehicle driving, but is not necessarily limited thereto.

[0160] 11 is a diagram illustrating the operation of a feature exchange module according to one embodiment. Referring to FIG. 11, a diagram 1100 is shown illustrating an example of the structure of the feature exchange module 1033 according to one embodiment.

[0161] The feature exchange module 1033 includes, for example,

number

[0162] 12 is a diagram illustrating the operation of a map element generation module according to one embodiment. Referring to FIG. 12, a diagram 1200 is shown illustrating an example of the structure of the map element generation module 1036 according to one embodiment.

[0163] The map element generation module 1036 receives the enhanced features at all times from the feature exchange module 1033.

number

number

[0164] The map element generation module 1036 can generate map element features that refer to the enhanced features by sequentially applying the values, keys, and queries extracted for each time point as inputs to L transformer decoder blocks 1210, 1230, and 1250. The map element generation module 1036 passes the map element features that refer to the enhanced features through a 2D vectorized map regression layer 1270 to generate N map elements corresponding to each time point. Here, the N map elements may have a size of, for example, T×N×2.

[0165] 13 is a diagram illustrating a method for designing losses for neural network training according to an embodiment. Referring to FIG. 13, a diagram 1300 is shown illustrating a training process of a neural network 1030 in which a control device according to an embodiment calculates two types of losses, supervised learning and unsupervised learning, using an instance matching method.

[0166] The control device can eliminate flickering or distortion that occurs in vectorized maps generated at successive times by using a loss that ensures that vectorized maps warped at different times and the vectorized map at the current time result in the same result.

[0167] The instance matching loss may be calculated in the following order:

[0168] The control device outputs the predicted result yt corresponding to the time t 1350 at the time t-1 1330.

number

[0169] The control device performs a process to obtain a result of the warping between the warped image and the predicted image.

number

[0170] The control device may calculate an instance matching loss between the warped image and the original image. Here, the matching algorithm may be, for example, a Hungarian algorithm, but is not limited thereto.

[0171] Here, according to supervised learning, the control device calculates the matching relationship with the correct answer (GT; g), and then

number

[0172] In contrast, with unsupervised learning, the controller

number

[0173] 14 is a block diagram of a device for controlling vehicle travel according to an embodiment. Referring to FIG. 14, a control device 1400 according to an embodiment includes a communication interface 1410, a memory 1430 including a first neural network 1433 and a second neural network 1436, and a processor 1450. The communication interface 1410, the memory 1430, and the processor 1450 may be connected via a communication bus 1405.

[0174] The communication interface 1410 receives a multi-view video including video frames at successive points in time corresponding to the vehicle's driving environment.

[0175] The memory 1430 stores pre-trained neural network parameters. The processor 1450 may generate a vectorized map using a neural network model to which the parameters stored in the memory 1430 are applied. The memory 1430 may also store various data and programs. The memory 1430 may include a volatile memory or a non-volatile memory. The memory 1430 may include a large-capacity storage medium such as a hard disk to store various data.

[0176] The first neural network 1433 extracts BEV features and map queries corresponding to successive time points for each video frame. The first neural network 1433 may be, for example, the first neural network 510 described above with reference to FIG. 5, but is not necessarily limited thereto.

[0177] The second neural network 1436 generates a vectorized map by predicting and vectorizing map elements included in the video frame based on the first memory token, BEV features, and map query stored in the memory 1430 in response to the query of the previous video frame of the video frame. The second neural network 1436 may be, for example, the second neural network 540 described above with reference to FIG. 5, but is not necessarily limited thereto.

[0178] The processor 1450 controls the vehicle's driving based on the vectorized map generated by the second neural network 1436. The processor 1450 can control the vehicle's driving by generating various control parameters for steering the vehicle and assisting the vehicle's driving based on the vectorized map.

[0179] The processor 1450 may also perform at least one of the methods described above with reference to FIGS. 1 to 13 or an algorithm corresponding to at least one of the methods. The processor 1450 may be a data processing device implemented in hardware having circuits with physical structures for performing desired operations. For example, the desired operations include code or instructions included in a program. The processor 1450 may be configured, for example, as a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU). For example, the control device 1400 may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0180] The processor 1450 executes programs and controls the control device 1400. The program codes executed by the processor 1450 may be stored in the memory 1430.

[0181] 15 is a block diagram of a vehicle according to one embodiment. Referring to FIG. 15, a vehicle 1500 according to one embodiment includes a sensor 1510, a neural network 1530, and a processor 1550. The sensor 1510, the neural network 1530, and the processor 1550 may be connected via a communication bus 1505.

[0182] The sensor 1510 captures a multi-view image including successive image frames corresponding to the vehicle's driving environment. The sensor 1510 may include, but is not limited to, a camera sensor, a vision sensor, a lidar sensor, a multi-camera sensor, and a radar sensor, for example.

[0183] The neural network 1530 extracts BEV features and map queries corresponding to successive time points for each video frame. The neural network 1530 generates a vectorized map by predicting map elements included in the video frame based on the first memory token, BEV features, and map queries stored in memory corresponding to queries of the previous video frame. The neural network 1530 may be, for example, a neural network in which the first neural network 510 and the second neural network 540 shown in FIG. 5 are combined into one, or a neural network in which the first neural network 510 and the second neural network 540 are separated.

[0184] Processor 1550 generates control signals for vehicle operation based on the vectorized map generated by neural network 1530.

[0185] Furthermore, processor 1550 may perform at least one of the methods described above with reference to FIGS. 1 to 14 or an algorithm corresponding to at least one of the methods. Processor 1550 may be a data processing device implemented as hardware having circuits with physical structures for executing desired operations. For example, the desired operations include code or instructions included in a program. Processor 1550 may be configured as, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU). For example, vehicle 1500 may include a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA).

[0186] The processor 1550 executes programs and controls the vehicle 1500. The program code executed by the processor 1550 may be stored in a memory.

[0187] The above-described embodiments may be implemented using hardware components, software components, and / or a combination of hardware and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or other adaptive supersampling device capable of executing and responding to commands. The processing device may execute an operating system (OS) and software applications that run on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used, but those skilled in the art will recognize that the processing device may include multiple processing elements and / or multiple types of processing elements. For example, the processing device may include multiple processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

[0188] Software may include computer programs, code, instructions, or any combination thereof, capable of configuring a processing device or instructing the processing device, either individually or collectively, as desired. The software and / or data may be permanently embodied in any type of machine, component, physical device, virtual device, computer storage medium, or device, or transmitted signal wave, to be interpreted by the processing device or to provide instructions or data to the processing device. The software may be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0189] The method according to the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable recording medium. The recording medium may include program instructions, data files, data structures, and the like, alone or in combination. The recording medium and program instructions may be specially designed and constructed for the purposes of the present invention, or may be well-known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language code, such as that generated by a compiler, but also high-level language code that is executed by a computer using an interpreter, for example.

[0190] The hardware adaptive supersampling devices described above may be configured to operate as one or more software models to perform the operations shown in this invention, and vice versa.

[0191] Although the above-described embodiments have been described using limited drawings, those skilled in the art may apply various technical modifications and variations based on the above description. For example, the described techniques may be performed in a different order than described, and / or the components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different manner than described, and suitable results may still be achieved even if they are replaced or substituted with other components or equivalents. Therefore, other implementations, other embodiments, and equivalents of the claims are also within the scope of the following claims. [Explanation of symbols]

[0192] 501 clip-level multi-view footage 510 First Neural Network 520 Map Query 525 BEV Features 527 memory 530 Clip Query 540 Second Neural Network 570 Vectorized Maps

Claims

1. receiving a multi-view video including video frames at successive points in time corresponding to a driving environment of the vehicle; extracting BEV features and map queries corresponding to each of the consecutive time points for each of the video frames; generating a vectorized map by predicting and vectorizing map elements included in the video frame based on a first memory token stored in a memory in response to a query of a video frame previous to the video frame, the BEV features, and the map query; controlling the vehicle's travel based on the vectorized map; A method for controlling vehicle travel, comprising:

2. The step of extracting BEV features and map queries includes: extracting perspective (PV) image features corresponding to the video frames using a backbone network; converting the perspective view (PV) video features into the BEV features; extracting the frame-level map query based on the BEV features and a query corresponding to the video frame, which is used to construct the vectorized map; outputting the BEV features and the map query; 2. The method of controlling vehicle travel of claim 1, comprising:

3. The step of generating the vectorized map includes: reading the first memory token; generating at least one of a map token including the map elements constituting the vectorized map and a clip token including vectorized features corresponding to the video frame based on the map query, the BEV features, and the first memory token; generating the vectorized map based on the map tokens; 2. The method of controlling vehicle travel of claim 1, comprising:

4. The step of generating at least one of the map token and the clip token comprises: generating the clip token from the map query and the first memory token, the clip token including a clue to the map element in a feature space corresponding to the video frame; updating the BEV feature with the clip token so that the BEV feature includes the hidden map element; generating the map token using the updated BEV features and the map query; 4. The method of controlling vehicle travel according to claim 3, comprising:

5. 4. The method of claim 3, wherein the size of the map query is determined based on at least one of the size of the clip token, the number of map elements, and the number of points per map element.

6. The step of updating the BEV characteristics includes: extracting a query from the BEV features; extracting a key and a value from the CLIP token; updating the BEV features via a cross-attention network and a feedforward network using the query, the key, and the value; 5. The method of claim 4, comprising:

7. 5. The method for controlling vehicle navigation of claim 4, wherein generating the map token includes generating the map token from the map query and the updated BEV features using a deformable attention network, a separated self-attention network, and a feedforward network.

8. 8. The method for controlling vehicle driving of claim 7, wherein generating the map token includes extracting the query from the map query using the deformable attention network and generating the map token by obtaining values ​​from the updated BEV features.

9. The step of generating the vectorized map includes a step of predicting the map elements included in the video frame using a pre-trained neural network, and generating the vectorized map vectorized for each instance; The pre-trained neural network comprises: a second-first neural network for reading the first memory token from the memory or recording a second memory token in the memory; a second-second neural network that generates the vectorized map corresponding to a current frame of the video frames based on the map query, the BEV features, and the first memory token; 4. The method of claim 3, further comprising at least one of:

10. The step of generating the vectorized map includes: recording, by said second-first neural network, said map token in said memory; generating the vectorization map by passing a map token corresponding to a current frame among the map tokens through a prediction head; 10. The method of controlling vehicle travel of claim 9, comprising:

11. generating said second memory token by recording said map token and clip token in said memory using said second-first neural network; outputting the second memory token; The method of controlling vehicle travel of claim 9, further comprising:

12. 10. The method of claim 9, wherein the second-first neural network stores time information corresponding to the previous video frame by interpreting a first memory token corresponding to the previous video frame and propagating it as an input to the second-second neural network.

13. The method for controlling vehicle driving according to claim 9, wherein the second-first neural network associates information between clips through propagation of clip tokens generated in the second-second neural network to set intra-clip connections between the map elements.

14. The method for controlling vehicle driving described in claim 9, wherein the second-1 neural network generates the second memory token containing global map information through learnable frame embedding based on the map token and the clip token generated by the second-2 neural network and stores the second memory token in the memory.

15. 10. The method of claim 9, wherein the second-first neural network generates the second memory token by combining a clip token, the map token, and the first memory token.

16. 10. The method for controlling vehicle driving of claim 9, wherein the second-second neural network generates the vectorized map by outputting a map token corresponding to a current frame having a fixed time window corresponding to the length of the video frame based on the first memory token, the BEV features, and the map query.

17. The method for controlling vehicle travel according to claim 1 , wherein the map elements include at least one of a crosswalk, a road, a lane, a road boundary, a building, a boundary stone, and a traffic light included in the travel environment.

18. A computer program stored on a computer readable storage medium for performing the method according to any one of claims 1 to 16 in combination with hardware.

19. a communication interface for receiving a multi-view video including video frames at successive points in time corresponding to a driving environment of the vehicle; a first neural network for extracting BEV features and map queries corresponding to the successive time points for each of the video frames; a second neural network that predicts and vectorizes map elements included in the video frame based on a first memory token stored in a memory in response to a query of a video frame previous to the video frame, the BEV features, and the map query, to generate a vectorized map; a processor for controlling the vehicle's travel based on the vectorized map; A device for controlling the running of a vehicle, including:

20. a sensor for capturing a multi-view video including video frames at successive points in time corresponding to a vehicle's driving environment; a neural network that extracts BEV features and map queries corresponding to the successive time points for each of the video frames, and generates a vectorized map by predicting map elements included in the video frame based on a first memory token, the BEV features, and the map query stored in a memory corresponding to a query of a video frame preceding the video frame; a processor that generates control signals for driving the vehicle based on the vectorized map; Including, vehicles.

Images