Vehicle driving control method and device

Abstract

The invention discloses a method and device for controlling driving of a vehicle and the vehicle. The method for controlling driving of a vehicle includes: receiving a multi-view image including image frames at consecutive points in time corresponding to a driving environment of the vehicle; extracting bird's-eye view (BEV) features and map queries respectively corresponding to the continuous time points for each image frame; generating a vectorized map by predicting and vectorizing map elements included in the image frames based on the BEV features, the map query, and a first memory flag stored in the memory corresponding to a query of a previously processed one of the image frames; and controlling driving of the vehicle based on the vectorized map.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of Korean Patent Application No. 10-2024-0063980, filed May 16, 2024, and Korean Patent Application No. 10-2024-0090377, filed July 9, 2024, in the Korean Intellectual Property Office, the disclosure of both of which is incorporated herein by reference in its entirety for all purposes. TECHNICAL FIELD

[0003] The following embodiments relate to a method and apparatus for vehicle driving control. BACKGROUND

[0004] With the development of neural networks, electronic devices in various fields can analyze input data using a neural network-based model and extract and / or generate effective information. For example, predicting road geometry (e.g., lanes, road markings, etc.) using a neural network and constructing a high-quality map are becoming a key task for safe autonomous driving. Static map elements included in a high-quality map are important information for autonomous vehicle applications such as lane keeping, path planning, and trajectory prediction. However, static map elements can be repeatedly occluded by various dynamic objects on the road in underlying sensing data. SUMMARY

[0005] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0006] In one general aspect, a method of controlling driving of a vehicle includes receiving a multi-view image including image frames of a driving environment of the vehicle at consecutive time points, extracting, for each image frame, bird's eye view (BEV) features respectively corresponding to the consecutive time points, extracting, for each image frame, map queries respectively corresponding to the consecutive time points, generating a vectorized map by predicting and vectorizing map elements represented in the image frames, the generating being based on the BEV features, the map queries, and first memory tags corresponding to queries of previously processed image frames in the image frames stored in a memory, and controlling the driving of the vehicle based on the vectorized map.

[0007] The extraction of the BEV features and the map queries can include extracting image features of a perspective view (PV) corresponding to the image frame using a backbone network, converting the image features of the PV to BEV features, extracting frame-level map queries for constructing the vectorized map based on the BEV features and queries corresponding to the image frame, and outputting the BEV features and the map queries.

[0008] The generation of the vectorized map can include reading a first memory token, generating map tokens including map elements included in the vectorized map and clip tokens including vectorized features corresponding to the image frame based on the map queries, the BEV features, and the first memory token, and generating the vectorized map based on the map tokens.

[0009] The generation of the map tokens and the clip tokens can include generating the clip tokens including clues for map elements in a feature space corresponding to the image frame from the map queries and the first memory token, updating the BEV features using the clip tokens such that the BEV features include hidden map elements, and generating the map tokens using the updated BEV features and the map queries.

[0010] The size of the map queries can be determined based on a size of the clip tokens, a number of the map elements, or a number of points for each of the map elements.

[0011] The updating of the BEV features can include extracting queries from the BEV features, extracting keys and values from the clip tokens, and updating the BEV features using the queries, the keys, and the values via a cross-attention network and a feed-forward network.

[0012] The generation of the map tokens can include generating the map tokens from the map queries and the updated BEV features using a deformable attention network, a decoupled self-attention network, and a feed-forward network.

[0013] The generation of the map tokens can include generating the map tokens by extracting queries from the map queries using a deformable attention network and obtaining values from the updated BEV features.

[0014] The generation of a vectorized map may include: generating a vectorized map by predicting map elements represented in an image frame using a pre-trained neural network and vectorizing the map elements for each instance, wherein the pre-trained neural network may include at least one of the following: a second-first neural network configured to read a first memory tag from memory or write a second memory tag to memory; and a second-second neural network configured to generate map tags including the map elements included in the vectorized map and clipping tags including vectorized features corresponding to the image frame, based on a map query, BEV features, and the first memory tag, and to generate a vectorized map corresponding to the current frame in the image frames based on the map tags.

[0015] The generation of a vectorized map may include: writing map markers into memory via a second-first neural network; and generating a vectorized map as the map markers corresponding to the current frame pass through the prediction head.

[0016] The method may further include: generating a second memory tag by writing map tags and clipping tags into memory using a second-first neural network; and outputting the second memory tag.

[0017] The second-first neural network can be configured to retain time information corresponding to the previously processed image frames by reading a first memory tag corresponding to the previously processed image frame and propagating the first memory tag as input to the second neural network.

[0018] The second-first neural network can be configured to establish intra-clip associations between map elements by associating inter-clip information through the propagation of clip markers generated in the second-second neural network.

[0019] The second-1 neural network can be configured to generate a second memory tag that includes global map information based on the map tags and clipping tags generated in the second-2 neural network by embedding learnable frames, and store the second memory tag in memory.

[0020] The second-first neural network can be configured to generate a second memory tag by combining clipping tags, map tags, and the first memory tag.

[0021] The second-second neural network can be configured to generate a vectorized map by outputting map markers corresponding to the current frame, based on a first memory tag, BEV features, and a map query. The current frame has a predetermined time window corresponding to the length of the image frame.

[0022] Map elements include pedestrian crossings, roads, lanes, lane edges, buildings, curbs, or traffic lights in the driving environment.

[0023] In another general aspect, an apparatus for controlling the driving of a vehicle includes: a communication interface configured to receive multi-view images, the multi-view images including image frames of the vehicle's driving environment at consecutive time points; a first neural network configured to extract BEV features corresponding to consecutive time points for each image frame, and configured to extract map queries corresponding to consecutive time points for each image frame; a second neural network configured to generate a vectorized map by predicting and vectorizing map elements represented in the image frames, the generation being based on BEV features, map queries, and first memory tags stored in memory corresponding to queries of previously processed image frames; and a processor configured to control the driving of the vehicle based on the vectorized map.

[0024] In another general aspect, a vehicle includes: a sensor configured to capture multi-view images, the multi-view images including image frames at consecutive time points corresponding to the driving environment of the vehicle; a neural network configured to extract BEV features and map queries corresponding to the consecutive time points for each image frame, and to generate a vectorized map by predicting and vectorizing map elements represented in the image frames based on the BEV features, the map queries, and first memory tags stored in memory corresponding to queries of previously processed image frames in the image frames; and a processor configured to generate control signals for driving the vehicle based on the vectorized map.

[0025] Other features and aspects will become clear from the following detailed description, drawings and claims. Attached Figure Description

[0026] Figure 1 An example method for controlling vehicle driving according to one or more embodiments is shown.

[0027] Figure 2 An example method for extracting bird's-eye view (BEV) features and queries according to one or more embodiments is shown.

[0028] Figure 3 An example method for generating vectorized maps is shown.

[0029] Figure 4 Examples of generating map markers and / or clip markers according to one or more embodiments are shown.

[0030] Figure 5 An example framework of a device for controlling vehicle driving according to one or more embodiments is shown.

[0031] Figure 6 An example of clip markers according to one or more embodiments is shown.

[0032] Figure 7 An example implementation of the second-2 neural network according to one or more embodiments is shown.

[0033] Figure 8 An example method for controlling vehicle driving according to one or more embodiments is shown.

[0034] Figure 9 An example framework of a device for controlling vehicle driving according to one or more embodiments is shown.

[0035] Figure 10A Examples of the structure and operation of a device for controlling vehicle driving according to one or more embodiments are shown.

[0036] Figure 10B An example is shown where a control device according to one or more embodiments is installed on a vehicle and identifies lanes during driving.

[0037] Figure 11 An example of the operation of a feature exchange module according to one or more embodiments is shown.

[0038] Figure 12 An example of the operation of a map element generation module according to one or more embodiments is shown.

[0039] Figure 13 An example method for designing a loss for training a neural network according to one or more embodiments is shown.

[0040] Figure 14 An example device for controlling vehicle driving is shown according to one or more embodiments.

[0041] Figure 15 An example vehicle according to one or more embodiments is shown.

[0042] Throughout the accompanying drawings and detailed description, unless otherwise described or provided, it will be understood that the same or similar reference numerals refer to the same or similar elements, features, and structures. The drawings may not be drawn to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be enlarged. Detailed Implementation

[0043] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent after understanding the disclosure of this application. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be significantly altered after understanding the disclosure of this application, except for operations that must be performed in a certain order. Furthermore, for clarity and conciseness, descriptions of features known after understanding the disclosure of this application may be omitted.

[0044] The features described herein may be implemented in various forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways in which the methods, apparatuses, and / or systems described herein will be apparent upon understanding the disclosure of this application.

[0045] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. The articles “a,” “an,” and “the” are also intended to include plural forms unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any one of the listed items, and any combination of any two or more. As a non-limiting example, the terms “comprising” or “containing,” “including” or “comprises,” and “having” or “containing” indicate the presence of the stated feature, quantity, operation, component, element, and / or combination thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0046] Throughout this specification, when a component or element is described as being "connected to," "coupled to," or "joined to" another component or element, it may be directly "connected to," "coupled to," or "joined to" that other component or element, or there may reasonably be one or more other components or elements present in between. When a component or element is described as being "directly connected to," "directly coupled to," or "directly joined to" another component or element, there may be no other elements present in between. Similarly, for example, "between" and "directly between," and "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0047] Although terms such as “first,” “second,” and “third,” or A, B, (a), (b), etc., may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited by these terms. For example, each of these terms is not used to define the nature, order, or sequence of the corresponding component, part, region, layer, or section, but only to distinguish the corresponding component, part, region, layer, or section from other components, parts, regions, layers, or sections. Therefore, without departing from the teaching of the examples, the first component, part, region, layer, or section mentioned in the examples described herein may also be referred to as the second component, part, region, layer, or section.

[0048] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, and as commonly understood based on an understanding of the disclosure of this application. Terms such as those defined in common dictionaries shall be interpreted as having the same meaning as in the context of the related technologies and the disclosure of this application, and shall not be interpreted as having an ideal or overly formal meaning, unless expressly defined herein. In this document, the term “may” in relation to an example or embodiment (e.g., what an example or embodiment may include or implement) indicates the existence of at least one example or embodiment that includes or implements such a feature, and not all examples are limited thereto.

[0049] The examples described below can be used to generate visual / image information in augmented reality navigation systems for intelligent vehicles to assist steering in autonomous or driver-assisted vehicles. These examples can be used in systems that utilize cameras (e.g., front-facing cameras, multi-camera, and surround-view monitor (SVM) systems) for autonomous driving or advanced driver assistance systems (ADAS); and systems that utilize lidar and / or radar for autonomous driving or ADAS. Additionally, these examples can be used to assist safe and comfortable driving by interpreting visual information using devices including intelligent systems (e.g., head-up displays (HUDs) installed in vehicles for driver assistance or fully autonomous driving). These examples can be applied to autonomous vehicles, intelligent vehicles, smartphones, navigation systems, mobile devices, etc. The foregoing systems and applications are non-limiting examples.

[0050] Figure 1 An example of controlling vehicle driving according to one or more embodiments is shown.

[0051] refer to Figure 1 The device used to control the driving of the vehicle (hereinafter referred to as the "control device") can control the driving of the vehicle by operating 110 to 140 based on a vectorized map.

[0052] In operation 110, the control device can receive multi-view images, which include image frames of the vehicle being driven in a driving environment at consecutive points in time. The vehicle can be, for example, an intelligent vehicle equipped with an ADAS and / or AD system that uses sensors, image processing equipment, communication equipment, etc., to identify and / or determine conditions of the driving environment during driving, in order to control the operation of the vehicle and / or notify the driver of these conditions. The ADAS and / or AD system can identify stationary objects on the driving road, including lane markings, road signs, etc., based on driving images captured by cameras.

[0053] The vehicle can correct its position or trajectory by mapping (i) target distances estimated by ADAS based on image frames obtained from cameras to (ii) pre-built map information, and this correction can be performed simultaneously by correcting the representation of the lanes (hereinafter “lanes”) of the driving road identified by the vehicle, as well as lane-related driving information. Additionally, the vehicle can receive various driving information, including the identified lanes associated with the driving road, through a (local and / or remote) navigation system.

[0054] As a non-limiting example, a vehicle can be any type of means of transportation used to move people or objects using a drive engine (e.g., car, bus, motorcycle, or truck). A vehicle can also be referred to as an "ego vehicle."

[0055] Multi-view images can be synchronized image frames at consecutive time points. Three consecutive time points can be time point t-2, time point t-1, and time point t. While time point t-1 corresponds to the current time point, time point t-2 can correspond to a previous time point, and time point t can correspond to a subsequent time point. Image frames can be, for example, image frames of driving scenes including vehicles, lanes, curbs, sidewalks, surrounding environment, stationary objects, etc. Image frames can be acquired using, but are not limited to, a capture device (e.g., a single camera or multiple cameras) mounted at the front of a vehicle (e.g., an autonomous vehicle). In this case, it can be assumed that the calibration information of the capture device is known. For example, the capture device can include a single camera, a vision sensor, an image sensor, or a device for performing similar functions. Alternatively, image frames can be images captured by a capture device included in a control unit, or by a device other than a control unit. Image frames can include not only stationary objects such as roads, road signs, and lanes, but also moving objects such as pedestrians or other vehicles driving around the vehicle. Roads can be paths traveled by vehicles, such as highways, national roads, local roads, expressways, or dedicated motor vehicle lanes. A road may include one or more lanes. A road on which vehicles drive is sometimes referred to as a "driving road." A lane is a road space that is distinguished from one another by lane markings on the road surface. Lanes can be distinguished by lane lines on the left and / or right sides. Road signs may be, for example, speed signs, distance signs, or milestones, but are not limited to these.

[0056] In operation 120, the control device can extract bird's-eye view (BEV) features and map queries corresponding to consecutive time points for each image frame. This can be achieved, for example, as follows: Figure 5 and Figure 7 The first neural network 510 shown is used to extract BEV features and map queries. (Reference) Figure 2 and Figure 5 Describe the method by which the control device can extract BEV features and perform map queries.

[0057] In operation 130, the control device can generate a vectorized map by predicting and vectorizing map elements included in the image frame (described later), wherein the prediction is based on (i) a first memory tag (described below) stored in memory corresponding to a query of a previous image frame (within the image frame), (ii) BEV features, and (iii) a map query. The vectorized map can be a local map of vector data, representing, for example, centerlines, lanes, crosswalks, and road boundary information, and generated in vector form by identifying road information around the vehicle. The vectorized map may include, for example, information indicating the category of map elements, coordinate information of map elements, and orientation of map elements. Map elements may include, but are not limited to, crosswalks, roads, lanes, lane boundaries, buildings, curbs, road signs, traffic lights, etc., included in a high-resolution map or a vectorized map corresponding to the driving environment. Map elements may have unique indices corresponding to consecutive points in time. The vectorized map may be output in the form of a point cloud, for example... Figure 5 The vectorized map 570 shown is an example, but not limited to it.

[0058] The control device can generate a vectorized map by (i) predicting map elements included in an image frame (using a pre-trained neural network); and (ii) vectorizing each predicted map element instance. In this case, the pre-trained neural network may include... Figure 5 and Figure 7 The diagram shows at least one of the second-1 neural networks 540-1 and 540-3 and the second-2 neural network 540-2 (“2-1” refers to the first part of the second neural network, and “2-2” refers to the second part of the second neural network). Here, an instance (e.g., an object) may correspond to a single component of a high-resolution map or a vectorized map. Instances may include, but are not limited to, map elements such as roads, lanes, traffic lights, curbs, pedestrian crossings, etc. Additionally, instances may also include various objects included in a driving image of a vehicle.

[0059] As described in more detail below, the second-first neural network 540-1 can read / receive first memory tags from memory using a tag aggregator. Furthermore, the second-first neural network 540-3 can write second memory tags into memory using a tag aggregator. Here, the tag aggregator can select and aggregate valid and important tags from map tags 550, clipping tags 560, and the first memory tags. The second-second neural network 540-2 can generate a vectorized map corresponding to the current frame (within the image frames) based on map queries, BEV features, and the first memory tags. The control device can write map tags into memory using the second-first neural network 540-3, and generate a vectorized map by passing the map tags corresponding to the current frame from the map tags written into memory through a prediction head.

[0060] As described in more detail below, neural networks 540-1 and 540-3, in their second-first stage, can retain temporal information corresponding to previous image frames by reading a first memory tag corresponding to the previous image frame and propagating the first memory tag as input to neural network 540-2, in their second-second stage. Neural network 540-3, in its second-first stage, can establish intra-clip associations between map elements by associating inter-clip information through the propagation of clip tags generated in neural network 540-2, in addition to using the first memory tag, BEV features, and map tags corresponding to the current frame as output by the map query, wherein the current frame has a predetermined time window corresponding to the length of the image frame.

[0061] In operation 140, the control unit can control the driving of the vehicle based on a vectorized map. The control unit can control the driving of the vehicle by generating various control parameters for autonomous driving of the vehicle (e.g., vehicle path setting, lane detection, vehicle steering, vehicle driving, and vehicle driving assistance) based on the vectorized map.

[0062] Figure 2 Example methods for extracting BEV features and map queries according to one or more embodiments are shown. References Figure 2 The control device can extract BEV features and perform map queries through operations 210 to 240.

[0063] In operation 210, the control device can use, for example... Figure 5 The backbone network 511 shown is used to extract image features of the perspective (PV) corresponding to the image frame. The backbone network 511 can be used as an encoder to encode the image features.

[0064] In operation 220, the control device can convert the image features of the PV extracted in operation 210 into BEV features. The control device can use, for example...Figure 5 The PV-to-BEV converter 513 shown converts image features of PV into BEV features. The PV-to-BEV converter 513 can use, for example, inverse perspective mapping (IPM) techniques to convert the image features of PV into feature vectors (i.e., BEV features) in BEV space, but is not limited to this. IPM techniques may involve removing perspective effects from input images (image frames) and / or segmented images that have perspective effects, and converting positional information on the image plane (e.g., similar to projection in three dimensions) into positional information in the world coordinate system.

[0065] In operation 230, the control device can extract a frame-level map query (used to construct a vectorized map), and this extraction can be based on BEV features (BEV features obtained from the transformation in operation 220) and on query Q corresponding to the image frame. The control device can use, for example... Figure 5 The map decoder 515 shown is used to extract frame-level map queries. The map decoder 515 may, for example, decode feature vectors and output a prediction head of instances. The size of the map query may be determined based on, for example, the size of clipping markers, the number of predicted map elements, and / or the number of points per map element, but is not limited thereto.

[0066] In operation 240, the control device can output the BEV features from the transformation in operation 220, and can output the map query extracted in operation 230.

[0067] Figure 3 An example method for generating a vectorized map according to one or more embodiments is shown. References Figure 3 The control device can generate a vectorized map by operating 310 to 330.

[0068] In operation 310, the control device may read a first memory token. The first memory token may be information corresponding to an image frame at a previous time point. The first memory token may be a token stored in memory corresponding to a query of a previous image frame (one or more image frames at a previous time point). The first memory token may be a clip token and a map token stored in memory (which correspond to queries of image frames at previous time points), or it may be a processed form of these tokens (clip tokens and map tokens).

[0069] In operation 320, the control device may generate at least one of the following based on the map query output in operation 240, (iii) BEV features, and (iv) the first memory tag read in operation 310: (i) map tags including map elements included in the vectorized map; and (ii) clip tags including vectorized features corresponding to image frames. Here, clip tags may include cues for temporal map elements in the feature space. (See reference...) Figure 4 Describe a method for a control device to generate at least one of map markers and clipping markers.

[0070] In operation 330, the control device can generate a vectorized map based on the map markers generated in operation 320. The control device can generate the vectorized map by decoding the map markers using various learning matrices.

[0071] Figure 4 Example methods for generating map markers and / or clip markers according to one or more embodiments are shown. References Figure 4 The control device can generate map markers and / or clip markers by operating 410 to 430.

[0072] In operation 410, the control device can generate clipping markers based on a map query and first memory markers, the clipping markers including clues for map elements in the feature space corresponding to the image frame. The control device can, for example, via... Figure 5 and Figure 7 The clip marker generator 542 shown is used to generate clip markers. (See reference...) Figure 6 To describe clipping tags.

[0073] In operation 420, the control unit can update the BEV features using the clipping markers generated in operation 410, such that the BEV features include hidden (occluded) map elements. Maps typically represent static information, and therefore can have the same information or values ​​at the same global / physical location even when the time point changes. The control unit can use the aforementioned features to update the BEV features at the current time point by utilizing map information obtained at other time points, such that the BEV features include hidden map elements. Here, because hidden map elements are map elements occluded by dynamic objects such as nearby vehicles, hidden map elements can also be referred to as "occluded map elements." Accordingly, the control unit can generate a vectorized map by utilizing image information from multiple time points to generate a single vectorized map with high accuracy.

[0074] The control device can, for example, be via Figure 5 and Figure 7The BEV updater 544 shown uses clipping tags generated in operation 410 to update the BEV features. For example, the control device can extract a query Q from the BEV features and a key K and a value V from the clipping tags. The control device can use the query, key, and value to update the BEV features via a cross-attention network and a feedforward network.

[0075] In operation 430, the control device can generate map markers using the BEV features updated in operation 420 and the map queries extracted in operation 230. The control device can use, for example... Figure 5 and Figure 7 The map generator 546 shown is used to generate map markers, but the example is not limited to this. The control device can use deformable attention networks, self-attention networks, and feedforward networks to generate map markers based on map queries and updated BEV features. The control device can use deformable attention networks to extract queries from map queries and generate map markers by obtaining values ​​from updated BEV features.

[0076] Figure 5 An example of a frame for controlling vehicle driving is shown according to one or more embodiments. Figure 5 The frame of a control device 500 is shown, which outputs a clip-level vectorized high-definition (HD) map generated from a clip-level multi-view image 501.

[0077] The control device 500 can generate a clip-level vectorized HD map, i.e., a vectorized map 570, from a clip-level multi-view image 501. Constructing the vectorized map 570 can be a task of predicting instantiated vectorized representations of static map elements (e.g., pedestrian crossings, lane dividers, and road boundaries). Static map elements can be, for example, information important / critical for autonomous vehicle driving (e.g., lane keeping, path planning, and trajectory prediction).

[0078] For example, as used herein, the term "clip" refers to a set of features at multiple (e.g., 3, 5, etc.) consecutive time points. Additionally, the term "token" refers to vectorized features for a single clip, map element, etc. Furthermore, a "multi-view image" can be an image composed of stitched-together sub-images (either as a whole image or as a group), where the sub-images were captured at the same time point by corresponding cameras with different views (hence the "multi-view")

[0079] For example, when a set of synchronized multi-view images (e.g., clip input) is input, the control device 500 can sequentially construct clip-level vectorized HD maps (e.g., each HD map can be derived from a corresponding given set of multi-view images). The control device 500 can first extract frame-level BEV features 525 and map queries 520 from the clip input via a first neural network 510. The control device 500 can independently extract the map queries 520 and BEV features 525 from each image frame via the first neural network 510. BEV features 525 are features that match the image view that can be finally displayed to the user and can be presented in various forms such as pixel values, colors, textures, shapes, etc. BEV features 525 can play an important role in understanding and analyzing images. Since BEV features 525 are pixel-based image features, BEV features 525 can also be referred to as "rasterized features". The first neural network 510 can be, for example, a frame-level MapNet implementation and can use the MapTRv2 architecture, but is not limited to this.

[0080] The operation of the first neural network 510 will be described in more detail below.

[0081] The first neural network 510 can extract BEV features 525 and a clipping set of map queries 520 from the synchronized multi-view image 501. The first neural network 510 may include, for example, a backbone network 511, a PV-to-BEV converter 513, and a map decoder 515.

[0082] The backbone network 511 can extract image features of the PV corresponding to the image frame. The backbone network 511 can be, for example, an implementation of ResNet50, but is not necessarily limited to it. The first neural network 510 can convert the image features of the PV extracted by the backbone network 511 into BEV features 525 through the PV-to-BEV converter 513.

[0083] The PV to BEV converter 513 can convert the image features of the PV corresponding to the multi-view image 501 into BEV features 525. The PV to BEV converter 513 can use, for example, IPM techniques to convert the image features of the PV into feature vectors in the BEV space (i.e., BEV features 525), but the examples are not limited to this.

[0084] The PV-to-BEV converter 513 can extract integrated three-dimensional (3D) BEV features 525 covering the vehicle's surrounding environment using various PV-to-BEV conversion techniques. The PV-to-BEV converter 513 can use, for example, BEV feature pooling based on lift, sputtering, and shooting (LSS). The BEV features 525 can be utilized to query a map in the map decoder 515.

[0085] The map decoder 515 can output a frame-level map query 520, which can be directly used to construct a high-quality vectorized map 570 from the BEV features 525. The map decoder 515 can decode from the BEV features 525 and predict the map query 520 corresponding to the map elements using a task-specific header.

[0086] The first neural network 510 can output BEV features 525 as the result of the PV to BEV converter 513 and map query 520 as the result of the map decoder 515.

[0087] Subsequently, the control device 500 can generate clip markers 560 by applying previously extracted information (e.g., map query 520 and BEV features 525) as input to the second neural network 540. Clip markers 560 can include clip-level temporal map information and can directly interact with dense BEV features 525.

[0088] In this scenario, the control device 500 can establish a long-term temporal relationship with the image frame at the current time point by employing a first memory marker 530, which is written to and stored in the memory 527 corresponding to a previous clip. The first memory marker 530 may or may correspond to a clip query.

[0089] The control device 500 can generate clipping tags 560 via a second neural network 540 using a first memory tag 530 read from memory 527 and a map query 520. These clipping tags 560 include clues to temporal map elements in a compact feature space. Clipping tags 560 can correspond to clip-level map information. (Reference) Figure 6 Clip marker 560 is described in more detail. Control device 500 can update BEV feature 525 by employing clip marker 560.

[0090] The control device 500 can update the BEV feature 525 using clipping markers 560, enabling the BEV feature 525 to be generated as a map element that includes hidden elements (hidden relative to the corresponding current time point in the multi-view image 501). The control device 500 can reveal the hidden map elements by updating the BEV feature 525 using clipping markers 560. The size of the clipping markers 560 can be, for example, 50 (number of vectors) × 256 (dimensions of each feature).

[0091] The control device 500 can update the BEV features 525 using the BEV updater 544 of the second neural network 540, and generate a high-quality clip-level vectorized map 570 using the updated BEV features 555. The control device 500 can construct the high-quality clip-level vectorized map 570 by extracting map markers 550 using the updated BEV features 555.

[0092] After inferring the clip-level vectorized map 570, the control device 500 can generate a new second memory marker by using the clip marker 560, map marker 550, and current memory marker (e.g., first memory marker 530) through the second-first neural network 540-3. The control device 500 can generate the second memory marker by using the second-first neural network 540-3 to write the map marker 550 and clip marker 560 into the memory 527.

[0093] For example, the size of map marker 550 could be, for instance, 3(T) × 50 (number of vectors) × 256 (dimensions of each feature). Applying the learning matrix to map marker 550 would result in 3(T) × 50 (vector map) × [2(x,y) × 20 + number of categories]. The size of the updated BEV feature 555 could be, for instance, 3(T) × 100 (width) × 200 (height) × 256 (dimensions of each feature). Here, T represents the time window (clipping length).

[0094] When the size of the first memory marker is 96 (number of vectors) × 256 (dimension of each feature), during the memory reading process, 3 (T) × 50 map queries 520 and 50 clip queries 530, as well as a total of 96 first memory markers and 200 queries (96+200), can be reduced to 50.

[0095] The process of reading memories can be performed in the form of “(1,2,3)→(4,5,6)→(7,8,9)→……”, “(1,2,3)→(2,3,4)→(3,4,5)→……”, or “(1,1,1)→(1,1,2)→(1,2,3)→(2,3,4)→(3,4,5)→……”, these are just some examples.

[0096] The control device 500 can output a second memory tag. The second memory tag can be used to provide a timeline for clip-level reasoning corresponding to a set of subsequent image frames that follows the current set of image frames.

[0097] The second neural network 540 may include, for example, a second-first neural network 540-1, a second-second neural network 540-2, and a second-first neural network 540-3. The second-first neural network 540-1 and the second-first neural network 540-3 may correspond to the same neural network.

[0098] Neural networks 540-1 and 540-3 can read the first memory tag 530 stored in memory 527 or write the second memory tag to memory 527. Neural network 540-1 can refer to the case of reading the first memory tag 530 from memory 527. In addition, neural network 540-3 can refer to the case of writing the second memory tag to memory 527.

[0099] Neural networks 540-1 and 540-3 (2-1) can preserve dense temporal information from previous frames by propagating the tags from previous clip inputs to subsequent clip inputs. Neural network 540-3 (2-1) can retrieve map tags from neural network 540-2 (2-2). 550 and clipping marks 560 is written to memory. Neural network 540-3 (2-1) can efficiently manage long-term map information using a tag-transformer (TTM). The TTM can be a sequential autoregressive transformation model with a memory for sequential visual understanding of the real world. The external memory of the TTM can include a set of tags summarizing previous history (i.e., frames). The memory module of the TTM ensures that new observations are processed along with the content of the memory (rather than the entire history), thus allowing for efficient processing of long sequences at limited computational cost in each step. Neural networks 540-1 and 540-3 (2-1) can be referred to as "inter-clip revealers" (networks that reveal information between clips).

[0100] Neural networks 540-1 and 540-3, for example, can use a tag aggregator to read the first memory tag 530 stored in memory 527, or write the second memory tag into memory 527.

[0101] The second-second neural network 540-2 can generate a vectorized map corresponding to the current frame (e.g., frame t). The second-second neural network 540-2 can generate a vectorized map 570 corresponding to the current frame in the image frames based on map query 520, BEV features 525, and first memory markers 530. The second-second neural network 540-2 may include, for example, a clipping marker generator 542, a BEV updater 544, and a map generator 546. References will follow below. Figure 7 The structure and operation of the second-second neural network 540-2 are described in more detail.

[0102] Neural network 540-3 (2-1) can establish intra-clip connections (information exchange) between map elements by transferring inter-clip markers to the BEV features 525 of subsequent image frames. Neural network 540-3 (2-1) can generate a second memory marker by combining clip markers 560, map markers 550, and a first memory marker 530. Neural network 540-3 (2-1) can generate a second memory marker, including global map information, based on the map markers 550 and clip markers 560 generated in neural network 540-2 (2-2), through the embedding of learnable frames, and store this second memory marker in memory 527. Neural network 540-3 (2-1) can use a Time-Based Mapping (TTM) mechanism to manage the map information for the second memory marker stored in memory 527.

[0103] The second-1 neural network 540-3 can establish global relationships by generating second memory tags using tags generated in the second-2 neural network 540-2 (e.g., map tags 550 and clipping tags 560).

[0104] The reading process in the second neural network 540 can be as follows. The second neural network 540 can generate a compact clipping tag 560, including global map information, by reading the first memory tag 530 and the map query 520. The second neural network 540 can use a tag aggregator to read the first memory tag 530 and the map query 520, which efficiently selects information tags from the input after the TTM. The second neural network 540 can perform location embedding using a location-based memory addressing technique. Location embedding will be described in more detail below.

[0105] Since memory is unavailable in the first / initial clip level pipeline of the second neural network 540, the first memory tag 530 can initially be written by a learnable clip embedding.

[0106] In addition, the writing process in the second neural network 540 can be as follows.

[0107] The second neural network 540 can use a tag aggregator to perform write operations to memory. This is achieved by aggregating clipped tags. 560. Map Marking 550 and the first memory marker 530 can generate new second memory tags. .here, Let C represent the size of the memory tag. Let C represent the dimension corresponding to a feature, and L represent the Lth level of the deep neural network. Additionally, tS represents the time point S frames prior to the current time point t, and t-2S represents the time point 2S frames prior to the current time point t.

[0108] The newly generated second memory tag through the write operation can be used in subsequent clipping processes for read operations on the first layer of the 2-2 neural network 540-2. The control device 500 can perform the write operation using the same location-based memory addressing technique used in the read operation. The write operation can be applied to the last layer of the 2-2 neural network 540-2 to generate a new second memory tag, thereby preserving / propagating the clipping tag. 560 and map markers Information related to 550.

[0109] During the establishment of global map relationships in the second neural network 540, the control device 500 can efficiently execute the clip-level pipeline by avoiding redundant calculations due to time steps.

[0110] The second neural network 540 can efficiently perform online inference by collecting clip-level time map information and learning distinguishable associations.

[0111] A clip-level pipeline can be introduced / adopted to efficiently construct a global map by mapping BEV features 525 into a clip set and propagating map information to subsequent timestamps to infer the second neural network 540 online.

[0112] The second neural network 540 can perform NT / S times of clip-level inference on a sequence of NT frames by using a time step S based on the clip-level pipeline to perform efficient inference. (Reference) Figure 7 The structure and operation of the second-second neural network 540-2 are described in more detail.

[0113] The second neural network 540 may be a framework designed to reveal invisible map information that cannot be captured using only frame-level BEV features 525 (in the current input image). To avoid computationally intensive tasks, the control unit 500 may perform clip-by-clip inference using a time window (clip length) T and a step size S.

[0114] The first neural network 510 can be built on a frame-level MapTRv2 architecture, and therefore can be trained using loss functions used in MapTR and MapTRv2, for example. Additionally, one-to-one loss and auxiliary dense prediction loss can also be used to train the first neural network 510.

[0115] For example, classification loss, point-to-point loss, edge direction loss, etc. can be used to train the second neural network 540.

[0116] Figure 6 An example of clip markers according to one or more embodiments is shown. Figure 6Figure 600 shows clipping markers 560 corresponding to BEV features at three consecutive time points (e.g., time point t-2, time point t-1, and time point t).

[0117] As described above, the set of image features at consecutive time points can be referred to as a "clip", and the vectorized features for a clip can be referred to as a "tag". In other words, clip tag 560 can correspond to the vectorized features of the image at consecutive time points and can include clues for temporal map elements in the feature space.

[0118] The control device can generate compact clip markers 560, including time map information within the clip input, and use the clip markers 560 to update the BEV features corresponding to consecutive time points, so that hidden map elements can be made visible in the predetermined frame to be represented.

[0119] The control device can reveal / convey hidden map elements through the interaction between clip marker 560 and BEV features containing higher-level map information. The control device can establish intra-clip relationships between map elements by leveraging the propagation of information between clips using clip marker 560. For example, the control device can update and visualize / represent hidden (or occluded) BEV features based on clip marker 560 via 1D principal component analysis (PCA) projection. In this case, BEV features can be extracted from a second neural network.

[0120] Figure 7 An example implementation of a second-2 neural network according to one or more embodiments is shown. Figure 700 illustrates the detailed structure and operation of the second-2 neural network. The second-2 neural network 540-2 may be referred to as an "in-clip revealer".

[0121] The second-second neural network 540-2 may include an L-layer sequence. The second-second neural network 540-2 may include, for example, a clipping marker generator 542, a BEV updater 544, and a map generator 546. Each of the clipping marker generator 542, BEV updater 544, and map generator 546 may include various attention layers and feedforward layers. In each attention layer and feedforward layer, for example, normalized layer processing, dropout, residual connections, etc., may be performed.

[0122] Neural network 540-2, version 2-2, can utilize frame-level map queries 520 Q. map BEV Features 525 F BEV and memory read 710 U Read The clip set, including the map marker 550, the updated BEV feature 555, and the clip marker 560, is used to output the map marker 550.

[0123] Clip marker generator 542 can generate compact clip marker 560 by using a first memory marker 530 read from memory and a map query 520. Clip marker generator 542 can extract query Q from the first memory marker 530 and key K and value V from the map query 520.

[0124] The query, key, and value extracted by the clip tag generator 542 can be used to generate clip tags 560 through position embedding during the cross-attention layer, self-attention layer, and feedforward layer of the clip tag generator 542.

[0125] Clip marker generator 542 can query based on frame-level maps. 520 generates clip markers 560. Here, Indicates the clip mark size. This represents the number of map elements predicted by the first neural network, and This represents the number of points for each map element (pixel). The control device can achieve online mapping in the following / subsequent steps by using clip markers 560 generated by clip marker generator 542 to obtain a compact representation including map information.

[0126] BEV updater 544 can use clip tags generated by clip tag generator 542. 560 Updates BEV Features 525 is used to represent hidden map elements. The BEV updater 544 can extract the query Q from the BEV feature 525 and the key K and value V from the clip markers 560 generated by the clip marker generator 542. The query, key, and value extracted by the BEV updater 544 can be used to generate updated BEV features 555 through location embedding, which can be done during the cross-attention layer and feedforward layer of the BEV updater 544.

[0127] BEV updater 544 can update BEV feature 525 by clipping marker 560 corresponding to the hidden region associated with the original BEV feature 525, and output the updated BEV feature. 555. Updated BEV characteristics 555 can be used as a value in map generator 546.

[0128] Map generator 546 can use the updated BEV features generated by BEV updater 544. 555 to generate and / or output clip-level map markers 550. The map generator 546 may include a deformable attention layer, a decoupled self-attention layer, and a feedforward layer.

[0129] Map generator 546 can query maps through deformable attention layers. Extract query Q from 520 and from the updated BEV features. Value V is extracted from layer 555. Additionally, the decoupled self-attention layer can extract the key from the output of the deformable attention layer. The query, key, and value extracted by map generator 546 can be used to generate map markers through location embedding during the process of passing through the deformable attention layer, decoupled self-attention layer, and feedforward layer of map generator 546. 550. When the control device uses the sensing head to construct a vectorized HD map, map markers can be used directly. 550.

[0130] Although the standard transformer structure is permutation invariant, location information combined with temporal information can be used to predict clip-level (e.g., per clip) map elements.

[0131] The control device 500 can use a fixed 3D sinusoidal position embedding for BEV features. The control device 500 can use learnable position embeddings used in the frame-level first neural network as well as newly defined learnable temporal position embeddings for map tagging. The control device 500 can define new learnable position embeddings for clipping tags. Similarly, the control device 500 can define learnable position embeddings for memory tags (e.g., first memory tag 530 and second memory tag) used for reading and writing to the second-1st neural networks 540-1 and 540-3.

[0132] Map markers 550 is written to the memory of the 2-1 neural network 540-3, and when the map of the Lth layer is labeled... When the prediction head is passed through, a vectorized map 560 can be generated.

[0133] Figure 8 An example method for controlling vehicle driving according to one or more embodiments is shown, and Figure 9 An example of a frame for a device for controlling vehicle driving according to one or more embodiments is shown.

[0134] refer to Figure 8 and Figure 9 The control unit can control the driving of the vehicle by operating 810 to 850 based on a vectorized map.

[0135] In operation 810, the control device may receive sensor information 910 including synchronized multi-view images (e.g., 910-1, 910-2, and 910-3), which include image frames at consecutive time points (e.g., time 1, time 2, and time 3) corresponding to the vehicle's driving environment. The sensor information may be detected by various single or multiple sensors (e.g., camera sensors, vision sensors, lidar sensors, multi-camera sensors, and radar sensors).

[0136] In operation 820, the control device can extract features corresponding to consecutive time points from the sensor information 910 received in operation 810. The control device can, for example, refer to the above... Figure 5 The first neural network 510 or feature extraction module 930 described is used to extract features, but the examples are not limited to this.

[0137] In operation 830, the control device can generate map elements corresponding to the driving environment by propagating the features extracted in operation 820 for each time point to a pre-trained neural network 950. The control device can also obtain map information corresponding to the occluded areas at each time point by inputting the features extracted in operation 820 into the pre-trained neural network 950.

[0138] More specifically, the control device can identify map information at different times, including each time point, by combining (or exchanging) the features extracted in operation 820. The control device can identify map information at different times by combining the extracted features using a pre-trained neural network 950. The control device can use the map information at different times to update the features extracted in operation 820. In this case, the pre-trained neural network 950 could be a converter or a 3D convolutional neural network (CNN) that allows reference to information from multiple times, but the example is not limited to these. The pre-trained neural network 950 could be, for example, the reference mentioned above. Figure 5 The second neural network 540 is described.

[0139] The control device can define cues for generating map elements for map information at each time point. The control device can use these cues (as defined for each time point) and combined (or interchanged) features to obtain map information. The control device can also generate map elements at all time points simultaneously using the cues defined for each time point and updated features. For example, the control device can use a DNN-based converter decoder and, based on updated features, generate map elements at all time points simultaneously according to the cues. In this case, cues with the same index among the cues defined for each time point can generate the same map element.

[0140] The control device can generate map elements by defining the same shape characteristics of map elements at consecutive time points using acquired map information. Map elements can have unique indices corresponding to specific time points. Map elements generated at each time point with the same index can generate identical map elements as those generated at different time points.

[0141] In operation 840, the control device can use the map elements generated in operation 830 to generate a vectorized map for each point in time.

[0142] In operation 850, the control unit can control the driving of the vehicle based on the vectorized map generated in operation 840.

[0143] Figure 10A Examples of the structure and operation of a control device 1000 for controlling vehicle driving according to one or more embodiments are shown.

[0144] The control device 1000 allows the neural network 1030 to learn the same shape characteristics of map elements included in the vectorized map at consecutive time points based on features extracted by the feature extraction module 930 from single and / or multi-sensor information 910 received at consecutive time points, thereby allowing the neural network 1030 to generate a high-performance vectorized map 1050 for each time point.

[0145] The neural network 1030 may include a feature exchange module 1033 and a map element generation module 1036. The feature exchange module 1033 performs feature exchange between single and / or multi-sensor information 910 at consecutive time points, and the map element generation module 1036 predicts and / or generates map elements included in the vectorized map at consecutive time points. The control device 1000 can analyze the temporal consistency of map elements at consecutive time points so that the neural network 1030 can learn the same shape characteristics of map elements.

[0146] The acquisition of single and / or multi-sensor information 910 and the feature extraction of the feature extraction module 930 can be performed by the feature extraction section 1010.

[0147] The feature extraction section 1010 can extract features from single or multiple sensor information received over a continuous period of time.

[0148] The feature exchange module 1033 can identify information related to occluded map elements at different times by exchanging the features extracted at each time point.

[0149] The map element generation module 1036 can define prompts for the generation of map elements included in the vectorized map for each time moment, and can generate map elements by utilizing features identified by the exchange between map elements at different times.

[0150] The control device 1000 can predict the vectorized map 1050 corresponding to each time point using map elements. The control device 1000 can train the neural network 1030 based on the comparison results between the vectorized map 1050 and the ground reality (GT).

[0151] The control device 1000 can generate a vectorized map for each time point by exchanging or combining features between each time point using the same sensor information received at consecutive time points. The control device 1000 can analyze the temporal consistency between map elements at consecutive time points so that the neural network 1030 can learn the same shape characteristics.

[0152] More specifically, the feature extraction section 1010 can extract features (Ft) at each time point by receiving single and / or multi-sensor information 910 (e.g., vehicle surrounding information (St)) from a single sensor or multiple sensors at consecutive time points, as shown in Equation 1 below, through the feature extraction module (Enc) 930.

[0153] Equation 1

[0154]

[0155] Various types of sensors capable of recognizing map information (e.g., cameras, LiDAR, and radar) can be used as single or multiple sensors. The feature extraction module 930 can correspond to, for example, a DNN-based encoder. The feature extraction module 930 can also correspond to various types of neural networks or encoders (e.g., CNNs and converters) capable of extracting features from sensor information.

[0156] The control device 1000 can use the feature exchange module (Comm) 1033 to exchange features at corresponding time points. The swaps are performed to generate the improved features at each time point, as shown in Equation 2 below.

[0157] Equation 2

[0158]

[0159] The control device 1000 can exchange features in an image at a corresponding time point, so that features at another time point can compensate for map information obscured by obstacles at the current time point. The feature exchange module (Comm) 1033 can use various neural network modules capable of processing spatiotemporal data, such as DNN-based 3D CNNs and 3D converters. (See below for reference.) Figure 11 The operation of the feature exchange module (Comm) 1033 is described in detail.

[0160] The map element generation module 1036 can define N hints for generating map elements at each time point (t). Features improved by referencing the map element generation module (Dec) 1036 (). ),hint( It can generate N map elements corresponding to the corresponding time point. As shown in Equation 3 below.

[0161] Equation 3

[0162]

[0163] here, This can correspond to the prediction results of the map element generation module 1036.

[0164] The map element generation module 1036 generates map elements for all time points simultaneously by referencing features from all consecutive time points, thereby analyzing the temporal consistency between map elements and enabling the neural network 1030 to learn the same shape characteristics. The map element generation module 1036 can be implemented as any of various neural networks capable of generating map elements by referencing feature information based on prompts. The map element generation module 1036 can be implemented as, for example, a DNN-based converter decoder, but is not necessarily limited to this. Prompts can define feature information for each time point, and prompts with the same index at each time point can generate the same map element. The map element generation module 1036 can also generate map elements by referencing features previously exchanged through the feature exchange module 1033. In this case, map elements with the same index at each time point can generate the same map element at different time points. The map element generation module 1036 can collect the generated map elements and ultimately generate a vectorized map 1050 corresponding to each time point. The following will refer to... Figure 12 The operation of the map element generation module 1036 is described in more detail.

[0165] The control device 1000 can compare and analyze the vectorized map 1050 generated at each time point to calculate the time consistency loss (TCL), as shown in Equation 4 below.

[0166] Equation 4

[0167]

[0168] in, It can represent T consecutive time points. It can represent the prediction result at time point t.

[0169] The control unit can apply a loss function L to make the predictions at two time points similar. This loss function L uses the matching relationship between the predictions based on the vectorized map (e.g., the comparison between the generated vectorized map and the ground truth). Various loss functions, such as L1 loss, can be used as the loss function. The control unit can repeat the above process for all time points (e.g., tN to t) to produce the final loss value.

[0170] The control device can train the feature extraction module 930, the feature exchange module 1033, and the map element generation module 1036 by comparing the generated vectorized map with the ground truth (GT). Two types of loss functions can be used to train the feature extraction module 930, the feature exchange module 1033, and the map element generation module 1036.

[0171] One type of loss is the known loss between the prediction (e.g., map elements) and the ground truth (GT), which may result in the generation of a vectorized map that looks identical to the GT. In this case, map elements can have unique element indices for their respective time points. The control unit can match the GT with map elements so that elements with the same index at different time points generate the same map element. For example, when the map element corresponding to the nth index at time point 1 generates the lane to the left of the vehicle, the control unit can match the map element corresponding to the nth index at time point 2 with the GT to generate the lane to the left of the vehicle. When the same map element is generated with the same index in this way, the control unit can link map elements at different time points to enable tracking of map elements at consecutive time points.

[0172] Another type of loss corresponds to the loss that causes vectorized maps generated at different times to produce the same map elements with the same shape. A large amount of information can be utilized during the learning process. By utilizing vehicle movement information, vectorized maps generated at different times can be distorted to fit different time periods.

[0173] The control device can use loss elimination to remove flickering or distortion that may occur in the vectorized map generated at consecutive time points, such that the vectorized map distorted from different time points produces the same result as the vectorized map at the current time point.

[0174] The control device 1000 can calculate two types of loss (i.e., supervised learning and unsupervised learning) according to the instance matching scheme, and apply the calculated loss to the training of the neural network 1030. (See below for reference.) Figure 13 The method for training the 1030 neural network is described in more detail.

[0175] Figure 10B An example is shown where a control device according to one or more embodiments is installed on a vehicle and identifies lanes during driving.

[0176] When the control device 1000 is installed on or embedded in a vehicle and generates a vectorized map 1050 online, the control device 1000 can generate a vectorized map 1050 for each point in time, such as... Figure 10B As shown.

[0177] Since there are no features stored at previous time points at time point 1, the control device 1000 can extract the features at time point 1 and use these features as input to the feature exchange module 1033, while storing the features at time point 1 in the memory.

[0178] The control device 1000 can use the features of time point 1 (time 1) stored in memory at a previous time point as input to the feature exchange module 1033, instead of re-extracting the features of time point 1 (time 1) at time point 2 (time 2). The control device 1000 can store the features of time point 2 (time 2) in memory.

[0179] Additionally, the control device 1000 can use features stored in the memory at previous time points (e.g., features at time point 1 (time 1) and features at time point 2 (time 2)) as input to the feature exchange module 1033, instead of re-extracting the features of time point 1 (time 1) and time point 2 (time 2) at time point 3 (time 3).

[0180] The control device can generate a vectorized map corresponding to each of consecutive time points (e.g., time 1, time 2, and time 3) as described above, and then use the final vectorized map (time 3) corresponding to the current time point (i.e., time point 3 (time 3)) to control the driving of the vehicle. The driving control of the vehicle may include, but is not limited to, assistance for driving the vehicle (e.g., lane recognition, driving path setting, and driving parameter setting), or setting control parameters for controlling the driving of the vehicle.

[0181] Figure 11 An example of the operation of a feature exchange module according to one or more embodiments is shown. Figure 11Figure 1100 shows the structure of the feature exchange module 1033.

[0182] The size at the corresponding time point can be... Features (e.g., features at time 1, features at time 2, and features at time 3). The input is fed into the feature exchange module 1033. When the feature exchange module 1033 receives T features, it can process them through L convolutional or transformer blocks (e.g., 1110, 1130, and 1150), and the output size is... Improved features .

[0183] Figure 12 An example of how the map element generation module operates is shown. Figure 12 Figure 1200 shows an example of the structure of the map element generation module 1036.

[0184] Map element generation module 1036 can obtain improved features from all time points received from the aforementioned feature exchange module 1033. The value and key can be extracted from the map element prompts corresponding to the map elements. Additionally, the map element generation module 1036 can extract values ​​and keys from the map element prompts corresponding to the map elements. Extract the query from ).

[0185] The map element generation module 1036 can generate features of map elements with reference improved features by sequentially applying the values, keys, and queries extracted at corresponding time points as inputs to L converter decoder blocks 1210, 1230, and 1250. The map element generation module 1036 can generate N map elements corresponding to the corresponding time points by passing the features of the map elements with reference improved features through a 2D vectorized map regression layer 1270. The N map elements can have a size of, for example, T×N×2.

[0186] Figure 13 An example method for training a neural network loss according to one or more embodiments is illustrated. Figure 1300 illustrates the process by which a control device trains a neural network 1030 by calculating two types of loss (supervised learning and unsupervised learning) according to an instance matching scheme.

[0187] The control device can use loss elimination to remove flickering or distortion that may occur in the vectorized map generated at consecutive time points, such that the vectorized map distorted from different time points produces the same result as the vectorized map at the current time point.

[0188] The instance matching loss can be calculated in the following order.

[0189] The control device can generate the prediction results corresponding to time point t 1350 ( ) warped to time point t-11330, such as As shown. The control device can use the prediction results corresponding to time point t-1 (1310) The matching relationship is calculated based on time point t by distorting the time to time point t.

[0190] The control device can perform instance matching between the distorted image and the predicted image, such as... As shown.

[0191] The control device can calculate the instance matching loss between the distorted image and the original image. In this case, the Hungarian algorithm or similar matching algorithm can be used, but the example is not limited to this.

[0192] Based on supervised learning, the control device can calculate the matching relationship with GT(g), and then apply the calculated matching relationship to the predicted matching relationship (e.g., , and The calculation of ). Here, It represents GT, and This represents the prediction result. The function `matching()` performs a match between instances at the same location in two input vectorized maps. When a ground truth (GT) is available, `matching()` can perform a match between the prediction result and the GT, which can result in an accurate match. Correspondingly, a more accurate match can be performed between prediction results at two different points in time.

[0193] Conversely, using unsupervised learning methods, the control device can calculate the matching relationship between the distorted image and the predicted image, such as... As shown. When a ground truth (GT) does not exist, unsupervised learning can be used. Using unsupervised learning, the control device can take the predictions from previous time points as inputs. The predictions are distorted to the current time point to directly match the predictions from previous time points to the results at the current time point. ).

[0194] Figure 14 Examples of devices for controlling vehicle driving according to one or more embodiments are shown. References Figure 14 The control device 1400 may include a communication interface 1410, a memory 1430 including a first neural network 1433 and a second neural network 1436, and a processor 1450. The communication interface 1410, the memory 1430 and the processor 1450 may be connected to each other via a communication bus 1405.

[0195] The communication interface 1410 can receive multi-view images, which include image frames at consecutive time points corresponding to the vehicle's driving environment.

[0196] Memory 1430 can store parameters of a pre-trained neural network. Processor 1450 can use the neural network model with the parameters stored in memory 1430 applied to generate a vectorized map. Additionally, memory 1430 can store various types of data and programs. Memory 1430 may include volatile or non-volatile memory. Memory 1430 may include a high-capacity storage medium (e.g., a hard disk) for storing various types of data.

[0197] The first neural network 1433 can extract BEV features and map queries corresponding to consecutive time points for each image frame. The first neural network 1433 can be, for example, the one described above. Figure 5 The first neural network 510 is described, but is not limited thereto.

[0198] The second neural network 1436 can predict and vectorize map elements contained in an image frame and generate a vectorized map based on BEV features, map queries, and first memory tags stored in memory 1430 corresponding to queries of previous image frames in the image frame. The second neural network 1436 can be, for example, as described above. Figure 5 The second neural network 540 is described, but is not limited thereto.

[0199] The processor 1450 can control the driving of the vehicle based on a vectorized map generated by the second neural network 1436. The processor 1450 can control the driving of the vehicle by generating various control parameters for vehicle steering and assisting vehicle driving based on the vectorized map.

[0200] In addition, the processor 1450 can execute reference Figures 1 to 13 At least one of the described methods, or an algorithm corresponding to at least one of the described methods. Processor 1450 may be a hardware-implemented data processing device having circuitry physically configured to perform a desired operation. The desired operation may include, for example, code or instructions included in a program. Processor 1450 may include, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU). Control device 1400 may include, for example, a microprocessor, CPU, processor core, multi-core processor, multiprocessor, application-specific integrated circuit (ASIC), and field-programmable gate array (FPGA).

[0201] The processor 1450 can execute programs and control the control device 1400. The program code to be executed by the processor 1450 can be stored in the memory 1430.

[0202] Figure 15 An example of a vehicle according to one or more embodiments is shown. References Figure 15 The vehicle 1500 may include a sensor 1510, a neural network 1530, and a processor 1550. The sensor 1510, neural network 1530, and processor 1550 may be connected to each other via a communication bus 1505.

[0203] Sensor 1510 can capture multi-view images, including image frames at consecutive time points corresponding to the vehicle's driving environment. Sensor 1510 may include, for example, a camera sensor, a vision sensor, a lidar sensor, a multi-camera sensor, a radar sensor, etc., but is not limited to these examples.

[0204] The neural network 1530 can extract BEV features and map queries corresponding to consecutive time points of each image frame. The neural network 1530 can predict and vectorize map elements included in the image frame, and generate a vectorized map based on the BEV features, map queries, and a first memory tag stored in memory corresponding to queries from previous image frames in the image frame. The neural network 1530 can be, for example... Figure 5 The first neural network 510 and the second neural network 540 shown can be combined into one neural network, or the first neural network 510 and the second neural network 540 can be separated to form a neural network.

[0205] The processor 1550 can generate control signals for driving the vehicle based on a vectorized map generated by the neural network 1530.

[0206] In addition, the processor 1550 can execute reference... Figures 1 to 14 The described methods include one or more of the methods described above, or algorithms corresponding to one or more of the methods described above. Processor 1550 may be a hardware-implemented data processing device having circuitry physically configured to perform a desired operation. The desired operation may include, for example, code or instructions included in a program. Processor 1550 may include, for example, a CPU, GPU, or NPU. Vehicle 1500 may include, for example, a microprocessor, CPU, processor core, multi-core processor, multiprocessor, ASIC, and FPGA.

[0207] The processor 1550 can execute programs and control the vehicle 1500. The program code to be executed by the processor 1550 can be stored in memory.

[0208] The examples described herein can be implemented using hardware components, software components (instructions), and / or combinations thereof. The processing device can be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers and arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, FPGAs, programmable logic units (PLUs), microprocessors, or any other device capable of responding to and executing instructions in a defined manner. The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and create data in response to the execution of software. For simplicity, the description of the processing device is singular; however, those skilled in the art will understand that a processing device can include multiple processing elements and various types of processing elements. For example, a processing device can include multiple processors, or a single processor and a single controller. Additionally, different processing configurations (e.g., parallel processors) are also possible.

[0209] Software (instructions) may include computer programs, code, instructions, or combinations thereof, to independently or uniformly instruct or configure a processing device to operate as desired. Software and / or data may be stored permanently or temporarily on any type of machine, component, physical or virtual device, or computer storage medium or device capable of providing instructions or data to or being interpreted by the processing device. Software may also be distributed across network-coupled computer systems for distributed storage and execution. Software and data may be stored on one or more non-transitory computer-readable recording media.

[0210] The methods described in the examples above can be recorded in a non-transitory computer-readable medium including program instructions to implement the various operations described above. The medium may also include data files, data structures, etc., alone or in combination with the program instructions. The program instructions recorded on the medium may be specifically designed and constructed for the purposes of the embodiments, or they may be program instructions well known and available to those skilled in the art of computer software. Examples of non-transitory computer-readable media include: magnetic media, such as hard disks, floppy disks, and magnetic tapes; optical media, such as CD-ROMs and DVDs; magneto-optical media, such as optical discs; and hardware devices specifically configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, etc. Examples of program instructions include machine code (e.g., generated by a compiler) and files containing high-level code that can be executed by a computer using an interpreter.

[0211] The aforementioned hardware device can be configured to act as one or more software modules to perform the operations of the above embodiments, and vice versa.

[0212] Computing devices, vehicles, electronic devices, processors, memory, image sensors, vehicle / operation function hardware, ADAS / AD systems, displays, information output systems and hardware, storage devices, and references herein. Figures 1 to 15 Other devices, apparatuses, units, modules, and components described are implemented or represented by hardware components. Examples of hardware components that may be used to perform the operations described in this application, where appropriate, include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (e.g., an array of logic gates, a controller and arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes (or is connected to) one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (e.g., an operating system (OS) and one or more software applications running on the OS) to perform the operations described in this application. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For brevity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may implement a single hardware component, or two or more hardware components. The hardware components may have one or more different processing configurations, examples of which include single processor, discrete processor, parallel processor, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0213] Figures 1 to 15The methods for performing the operations described in this application, as shown, are executed by computing hardware, such as one or more processors or computers implemented as described above, which implement instructions or software to perform the operations described in this application (operations performed by the methods). For example, a single operation or two or more operations may be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be performed by one or more processors or a processor and a controller, and one or more other operations may be performed by one or more other processors or another processor and another controller. One or more processors, or a processor and a controller, may perform a single operation or two or more operations.

[0214] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as machines or special-purpose computers to perform the operations performed by the hardware components and the methods described above. In one example, the instructions or software include machine code that is directly executed by one or more processors or computers, such as machine code generated by a compiler. In another example, the instructions or software include higher-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding descriptions herein, which disclose algorithms for performing the operations performed by the hardware components and the methods described above.

[0215] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed on or in one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), memory of card type such as multimedia card micro or card (e.g., Secure Digital (SD) or Ultimate Digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to: store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and provide instructions or software and any associated data, data files, and data structures to one or more processors or computers, such that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a network-coupled computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0216] Although this disclosure includes specific examples, it will be apparent upon understanding the disclosure of this application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is intended to apply to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents.

[0217] Therefore, in addition to the above disclosure, the scope of this disclosure may also be defined by the claims and their equivalents, and all variations within the scope of the claims and their equivalents are to be included in this disclosure.

Claims

1. A method for controlling the driving of a vehicle, the method comprising: Receive multi-view images, the multi-view images including image frames of the vehicle's driving environment at consecutive time points; For each image frame, extract the bird's-eye view BEV features corresponding to the consecutive time points, and for each image frame, extract the map query corresponding to the consecutive time points; A vectorized map is generated by predicting and vectorizing map elements represented in the image frame, the generation being based on the BEV features, the map query, and a first memory tag stored in memory corresponding to the query of the previously processed image frame; as well as The vehicle is driven based on the vectorized map.

2. The method according to claim 1, wherein, The extraction of the BEV features and the map query includes: Image features of the perspective view PV corresponding to the image frame are extracted using a backbone network; Convert the image features of the PV into the BEV features; Based on the BEV features and the query corresponding to the image frame, extract the frame-level map query for constructing the vectorized map; and Output the BEV features and the map query.

3. The method according to claim 1, wherein, The generation of the vectorized map includes: Read the first memory tag; Based on the map query, the BEV features, and the first memory marker, map markers are generated, including map elements included in the vectorized map, and clipping markers are generated, including vectorized features corresponding to the image frames; and The vectorized map is generated based on the map markers.

4. The method according to claim 3, wherein, The generation of the map markers and the clip markers includes: The clipping tag is generated based on the map query and the first memory tag, and the clipping tag includes clues for map elements in the feature space corresponding to the image frame; Update the BEV features using the clipping markers, such that the BEV features include hidden map elements; and The map markers are generated using the updated BEV features and the map query.

5. The method according to claim 3, wherein, The size of the map query is determined based on the size of the clipping marker, the number of map elements, or the number of points for each map element.

6. The method according to claim 4, wherein, The update of the BEV feature includes: Extract the query from the BEV features; Extract the key and value from the clip marker; and The BEV features are updated using the query, the key, and the value via a cross-attention network and a feedforward network.

7. The method according to claim 4, wherein, The generation of the map markers includes: using a deformable attention network, a decoupled self-attention network, and a feedforward network to generate the map markers based on the map query and the updated BEV features.

8. The method according to claim 7, wherein, The generation of the map markers includes: extracting the query from the map query using the deformable attention network and obtaining values ​​from the updated BEV features to generate the map markers.

9. The method according to claim 1, wherein, The generation of the vectorized map includes: predicting the map elements represented in the image frame using a pre-trained neural network, and vectorizing the map elements for each instance to generate the vectorized map. The pre-trained neural network includes at least one of the following: The second-first neural network is configured to read the first memory tag from the memory, or write the second memory tag to the memory; and The second-second neural network is configured to generate map markers, including map elements included in the vectorized map, and clipping markers, including vectorized features corresponding to the image frames, based on the map query, the BEV features, and the first memory markers, and to generate a vectorized map corresponding to the current frame in the image frames based on the map markers.

10. The method according to claim 9, wherein, The generation of the vectorized map includes: The map markers are written to the memory via the second-first neural network; and The vectorized map is generated as the map markers corresponding to the current frame are passed through the prediction head.

11. The method of claim 9, further comprising: The second memory marker is generated by writing the map marker and the clip marker into the memory using the second-first neural network; as well as Output the second memory tag.

12. The method according to claim 9, wherein, The second-first neural network is configured to retain time information corresponding to the previously processed image frame by reading a first memory tag corresponding to the previously processed image frame and propagating the first memory tag as input to the second-second neural network.

13. The method according to claim 9, wherein, The second-first neural network is configured to establish intra-clip associations between map elements by associating inter-clip information through the propagation of clip markers generated in the second-second neural network.

14. The method according to claim 9, wherein, The second-1 neural network is configured to generate a second memory marker including global map information by embedding learnable frames based on the map markers and clip markers generated in the second-2 neural network, and to store the second memory marker in the memory.

15. The method according to claim 9, wherein, The second-first neural network is configured to generate the second memory tag by combining the clipping tag, the map tag, and the first memory tag.

16. The method according to claim 9, wherein, The second-second neural network is configured to generate the vectorized map by outputting map markers corresponding to the current frame, based on the first memory marker, the BEV feature, and the map query, wherein the current frame has a predetermined time window corresponding to the length of the image frame.

17. The method according to claim 1, wherein, The map elements include pedestrian crossings, roads, lanes, lane edges, buildings, curbs, or traffic lights in the driving environment.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to claim 1.

19. An apparatus for controlling the driving of a vehicle, the apparatus comprising: The communication interface is configured to receive multi-view images, which include image frames of the vehicle's driving environment at consecutive time points. The first neural network is configured to extract bird's-eye view BEV features corresponding to the consecutive time points for each of the image frames, and is configured to extract map queries corresponding to the consecutive time points for each of the image frames. The second neural network is configured to generate a vectorized map by predicting and vectorizing map elements represented in the image frame, the generation being based on the BEV features, the map query, and a first memory tag stored in memory corresponding to the query of the previously processed image. as well as The processor is configured to control the driving of the vehicle based on the vectorized map.

20. A vehicle comprising: The sensor is configured to capture multi-view images, which include image frames at consecutive time points corresponding to the driving environment of the vehicle. The neural network is configured to extract bird's-eye view BEV features and map queries corresponding to the consecutive time points for each of the image frames, and to generate a vectorized map by predicting and vectorizing the map elements represented in the image frames based on the BEV features, the map queries, and a first memory tag stored in memory corresponding to the queries of previously processed image frames in the image frames. as well as The processor is configured to generate control signals for driving the vehicle based on the vectorized map.

Images