Processing environmental data for a vehicle

Abstract

Examples described herein provide a computer-implemented method for processing environmental data of at least one other vehicle relative to a target vehicle. The method includes processing, by a processing device, first data associated with the at least one other vehicle using a self-attention mechanism to generate a first output associated with the at least one other vehicle. The method also includes processing, by the processing device, second data associated with the target vehicle using a stacked residual module to generate a second output associated with the target vehicle. The method further includes concatenating, by the processing device, the first output and the second output to generate a concatenated vector. The method also includes making, by the processing device, a driving decision for the target vehicle based at least in part on the concatenated vector. The method further includes controlling the target vehicle based at least in part on the decision.

Description

Technical Field

[0001] This disclosure relates to vehicles, and more specifically to the processing of environmental data of vehicles. Background Technology

[0002] Modern vehicles (e.g., cars, motorcycles, boats, or any other type of vehicle) can be equipped with vehicle communication systems that facilitate different types of communication between the vehicle and other entities. For example, vehicle communication systems can provide vehicle-to-infrastructure (V2I), vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and / or vehicle-to-grid (V2G) communication. Collectively, these can be referred to as vehicle-to-everything (V2X) communication, which enables the transmission of information from the vehicle to any other suitable entity. Various applications (e.g., V2X applications) can use V2X communication to send and / or receive safety messages, maintenance messages, vehicle status messages, etc.

[0003] Modern vehicles may also include one or more cameras that provide reversing assistance, capture images of the vehicle's driver to determine driver drowsiness or inattention, provide road images while the vehicle is in motion for collision avoidance, and provide structural recognition, such as road signs. For example, a vehicle may be equipped with multiple cameras, and images from multiple cameras (called "panoramic cameras") can be used to create a "panoramic" or "bird's-eye view" of the vehicle. Some cameras (called "remote cameras") can be used to capture remote images (e.g., object detection for collision avoidance, structural recognition, etc.).

[0004] Such a vehicle can also be equipped with sensors for performing target tracking, such as one or more radio radar devices, one or more LiDAR devices, etc. Target tracking involves identifying a target object and tracking it as it moves relative to the vehicle observing it. Images from one or more cameras on the vehicle can also be used to perform target tracking.

[0005] These communication protocols, cameras, and / or sensors can be used in autonomous vehicles. Summary of the Invention

[0006] In one exemplary embodiment, a computer-implemented method is provided for processing environmental data of at least one other vehicle relative to a target vehicle. The method includes processing first data associated with the at least one other vehicle using a self-attention mechanism by a processing device to generate a first output associated with the at least one other vehicle. The method also includes processing second data associated with the target vehicle by the processing device using stacked residual modules to generate a second output associated with the target vehicle. The method further includes concatenating the first and second outputs by the processing device to generate a concatenated vector. The method also includes making a driving decision for the target vehicle by the processing device based at least in part on the concatenated vector. The method further includes controlling the target vehicle based at least in part on the driving decision.

[0007] In an additional embodiment, the first data associated with at least one other vehicle includes at least one of the estimated motion state of the at least one other vehicle, a route allocation, and the vehicle's intent.

[0008] In an additional embodiment, the first data includes a first subset of the first data associated with a first other vehicle and a second subset of the first data associated with a second other vehicle.

[0009] In an additional embodiment, the first subset of the first data is expressed as a first one-dimensional (1D) feature vector, and the second subset of the first data is expressed as a second 1D feature vector.

[0010] In an additional embodiment, processing the first data includes combining the first 1D feature vector and the second 1D feature vector into a two-dimensional (2D) matrix.

[0011] In an additional embodiment, the 2D matrix includes speed data associated with at least one other vehicle and position data associated with at least one other vehicle.

[0012] In an additional embodiment, the second data associated with the target vehicle includes motion status information and route information.

[0013] In an additional embodiment, motion state information is represented in the coordinate system of the road where the target vehicle is located.

[0014] In an additional embodiment, the channel information is represented by a vector of binary values, where a zero value indicates that no channel exists and a one value indicates that a channel exists.

[0015] In an additional embodiment, each of the stacked residual modules includes a first convolutional unit, a corrected linear unit, and a second convolutional unit.

[0016] In an additional embodiment, making a driving decision includes determining a driving strategy and values ​​associated with the current state of the subject in the environment.

[0017] In an additional embodiment, controlling the target vehicle, at least in part, based on decision-making includes controlling the target vehicle based on driving strategies and the current state of the subject in the environment.

[0018] In another exemplary embodiment, a system includes a memory containing computer-readable instructions. The system also includes processing means for executing the computer-readable instructions, which control the processing means to perform operations for processing environmental data of at least one other vehicle relative to a target vehicle. The operations include: processing first data associated with the at least one other vehicle using a self-attention mechanism to generate a first output associated with the at least one other vehicle. The operations also include processing second data associated with the target vehicle using stacked residual modules to generate a second output associated with the target vehicle. The operations further include concatenating the first output and the second output to generate a concatenated vector. The operations also include making driving decisions for the target vehicle based at least in part on the concatenated vector. The operations further include controlling the target vehicle based at least in part on the driving decisions.

[0019] In an additional embodiment, the first data associated with at least one other vehicle includes at least one of the estimated motion state of the at least one other vehicle, a route allocation, and the vehicle's intent.

[0020] In an additional embodiment, the first data includes a first subset of the first data associated with a first other vehicle and a second subset of the first data associated with a second other vehicle.

[0021] In an additional embodiment, the first subset of the first data is expressed as a first one-dimensional (1D) feature vector, and the second subset of the first data is expressed as a second 1D feature vector.

[0022] In an additional embodiment, processing the first data includes combining the first 1D feature vector and the second 1D feature vector into a two-dimensional (2D) matrix.

[0023] In an additional embodiment, the 2D matrix includes speed data associated with at least one other vehicle and position data associated with at least one other vehicle.

[0024] In an additional embodiment, the second data associated with the target vehicle includes motion status information and route information.

[0025] In yet another exemplary embodiment, the computer program product includes a computer-readable storage medium having program instructions embodied therein, wherein the computer-readable storage medium itself is not a transient signal, the program instructions being executable by a processing device to cause the processing device to perform operations for processing environmental data of at least one other vehicle relative to a target vehicle. The operations include: processing first data associated with the at least one other vehicle using a self-attention mechanism to generate a first output associated with the at least one other vehicle. The operations also include processing second data associated with the target vehicle using stacked residual modules to generate a second output associated with the target vehicle. The operations further include cascading the first output and the second output to generate a cascaded vector. The operations also include making a driving decision for the target vehicle based at least in part on the concatenated vector. The operations further include controlling the target vehicle based at least in part on the driving decision.

[0026] The above-described features and advantages, as well as other features and advantages, of this disclosure will become apparent when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description

[0027] Other features, advantages, and details appear only by way of example in the following detailed description, which refers to the accompanying drawings, wherein:

[0028] Figure 1 A vehicle including sensors and processing systems is described according to one or more embodiments of the present document;

[0029] Figure 2 An architecture for supporting the processing of environmental data of a vehicle, according to one or more embodiments described herein, is depicted.

[0030] Figure 3 A flowchart depicts a method for processing environmental data of a vehicle according to one or more embodiments described herein; and

[0031] Figure 4 A block diagram is depicted of a processing system for implementing the techniques described herein, according to exemplary embodiments. Detailed Implementation

[0032] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. It should be understood that throughout the drawings, corresponding reference numerals denote the same or corresponding parts and features. As used herein, the term module refers to processing circuitry, which may include application-specific integrated circuits (ASICs), electronic circuitry, processors (shared, dedicated, or grouped) and memories executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality.

[0033] The technical solutions described herein provide methods for processing environmental data of vehicles. More specifically, one or more embodiments described herein provide methods for encoding and processing environmental data (information) in learning-based autonomous vehicle (AV) driving strategies using a deep attention mechanism.

[0034] The AV driving strategy provides guidance on how the autonomous vehicle should operate. The AV driving strategy encodes and extracts information about the autonomous vehicle's environment for decision-making. This information is used to construct a behavioral / motion plan for the autonomous vehicle to perform. Information can be collected from the autonomous vehicle (referred to as the "target vehicle") and / or from other surrounding vehicles (which may or may not be autonomous) (referred to as "other vehicles").

[0035] Conventional methods for processing environmental data of vehicles are insufficient. One such method is to rasterize different environmental information in a multi-channel "bird's-eye view" image and process it using a convolutional neural network. While this approach can be useful for motion projections of other vehicles, it is inefficient in rasterizing bird's-eye view images compared to projections using a road coordinate system. Another conventional method is convolutional social pooling. In this example, motion information of the vehicle is encoded relative to a road coordinate system by dividing the road into grid cells to provide channel-level lateral discretization and constant longitudinal forward units, where each unit has a multi-channel spatial container, and a joint social tensor is fed into a sequence of convolutional layers. However, due to the 2D structure of the social tensor, this method is limited to a simple readout structure where all channels are parallel to each other. In yet another conventional approach, an architecture is provided in which unordered vector sequences are processed by stacking kernel convolutions of 1-unit size and then performing max-pooling operations in the channel dimension. This method has limited expressive power in capturing interactions between different actuators (vehicles) and limited performance for use cases that require extracting multi-level interactions between input entities.

[0036] One or more embodiments described herein address these and other drawbacks of the prior art by using surrounding vehicle state encoding and neural architecture. According to one or more embodiments described herein, a method is provided that encodes the state of a target vehicle (e.g., an autonomous vehicle), the perceived states of other (surrounding) vehicles, and information about the environment. A neural architecture is then used to process the state / information to generate policy and / or value outputs for learning driving strategies. It should be understood that the techniques described herein can be used to encode the state of a target vehicle even in the absence of other (surrounding) vehicles. More specifically, one or more embodiments described herein provide an efficient and flexible way to encode environmental information for decision-making, such as in the case of autonomous vehicles. For example, state information can be encoded in the presence of a varying number of surrounding vehicles (including the absence of surrounding vehicles).

[0037] Figure 1 A vehicle 100 including a sensor and processing system 110 is depicted according to one or more embodiments described herein. Figure 1 In the example, vehicle 100 includes a processing system 110, cameras 120, 121, 122, 123, cameras 130, 131, 132, 133, a radar sensor 140, and a LiDAR sensor (light radar sensor) 141. Vehicle 100 can be a car, truck, van, bus, motorcycle, boat, aircraft, or other suitable vehicle 100.

[0038] Cameras 120-123 are panoramic cameras that capture images of the exterior and vicinity of vehicle 100. The images captured by cameras 120-123 together form a panoramic view (sometimes referred to as a "top-down view" or "bird's-eye view") of vehicle 100. These images can be used to operate the vehicle (e.g., parking, reversing, etc.). Cameras 130-133 are remote cameras that capture images of the exterior of the vehicle and are further away from vehicle 100 than cameras 120-123. These images can be used, for example, for object detection and avoidance. It should be understood that although eight cameras 120-123 and 130-133 are shown, more or fewer cameras may be implemented in various embodiments.

[0039] The captured images can be displayed on a monitor (not shown) to provide the driver / operator of the vehicle 100 with an external view of the vehicle 100. The captured images can be displayed as live images, still images, or some combination thereof. In some examples, images can be combined to form a composite view, such as a panorama.

[0040] Radar sensor 140 measures the distance to a target object by emitting electromagnetic waves and using the sensor to measure the reflected waves. This information is useful for determining the distance / position of the target object relative to vehicle 100.

[0041] The LiDAR (Light Detection and Ranging) sensor 141 measures the distance to a target object (e.g., another vehicle 154) by illuminating the target with a pulsed laser and measuring the reflected pulse with the sensor. This information is useful for determining the distance / position of the target object relative to the vehicle 100.

[0042] Data generated from cameras 120-123, 130-133, radar sensor 140, and / or LiDAR sensor 141 can be used to detect and / or track target objects relative to vehicle 100. Examples of target objects include other vehicles (e.g., other vehicle 154), vulnerable road users (VRUs) such as pedestrians, bicycles, animals, potholes, oil on the road surface, debris on the road surface, fog, floods, etc.

[0043] Processing system 110 includes a data / communication engine 112, a decision engine 114, and a control engine 116. The data / communication engine 112 receives / collects data from sensors associated with vehicle 100 and / or receives data from other sources such as remote processing system 150 and / or other vehicles 154. The decision engine 114 processes the data to make driving decisions. An example of how the decision engine 114 processes data is provided in... Figure 2 The control engine 116 controls the vehicle 100 to perform driving maneuvers (e.g., changing course, changing speed, etc.). Although not shown, the processing system 110 may include other components, engines, modules, etc., such as processors (e.g., central processing unit, graphics processing unit, microprocessor, etc.), memory (e.g., random access memory, read-only memory, etc.), data storage (e.g., solid-state drive, hard disk drive, etc.).

[0044] Processing system 110 can be communicatively coupled to remote processing system 150, which may be an edge processing node as part of an edge processing environment, a cloud processing node as part of a cloud processing environment, etc. Processing system 110 can also be communicatively coupled to one or more other vehicles (e.g., other vehicles 154). In some examples, processing system 110 is directly communicatively coupled to processing system 150 and / or other vehicles 154 (e.g., using V2V communication), while in other examples, processing system 110 is indirectly communicatively coupled to processing system 150 and / or other vehicles 154 (e.g., via a network). For example, processing system 110 may include a network adapter (not shown) (see example...). Figure 4 (Network adapter 426). The network adapter enables the processing system 110 to send data to and / or receive data from other sources, such as other processing systems including the remote processing system 150 and other vehicles 154, data repositories, etc. As an example, the processing system 110 can send data to and / or receive data from the remote processing system 150 directly and / or via network 152.

[0045] Network 152 represents any one or a combination of different types of suitable communication networks, such as, for example, cable networks, public networks (e.g., the Internet), private networks, wireless networks, cellular networks, or any other suitable private and / or public networks. Furthermore, network 152 may have any suitable communication range associated with it and may include, for example, global networks (e.g., the Internet), metropolitan area networks (MANs), wide area networks (WANs), local area networks (LANs), and personal area networks (PANs). Additionally, network 152 may include any type of medium capable of carrying network services, including but not limited to coaxial cable, twisted pair, optical fiber, hybrid fiber-coaxial (HFC) media, microwave terrestrial transceivers, radio frequency communication media, satellite communication media, or any combination thereof. According to one or more embodiments described herein, remote processing system 150, other vehicles 154, and processing system 110 communicate via vehicle-to-infrastructure (V2I), vehicle-to-vehicle (V2V), vehicle-to-pedestrian (V2P), and / or vehicle-to-grid (V2G) communications.

[0046] This document further describes the features and functions of the components of the processing system 110. The processing system 110 of the vehicle 100 assists in decision-making within the context of autonomous vehicle operation.

[0047] Figure 2An architecture 200 supporting the processing of vehicle environment data is depicted according to one or more embodiments described herein. In this example, architecture 200 includes vehicle 100, vehicle 201a, vehicle 201b, and vehicle 201c. Vehicles 201a, 201b, and 201c are... Figure 1 Examples of other vehicles 154, and collectively referred to as “vehicle 201”.

[0048] like Figure 2 As shown in the example, vehicle 100 and vehicle 201 travel in direction 204 along road 202 having channels 203a and 203b. Specifically, vehicle 100 travels in channel 203a in direction 204, and vehicle 201 travels in channel 203b in direction 204. It should be understood that... Figure 2 The specific arrangement of vehicles 100, 201 and road 202 in this paper is merely an example; other arrangements are also possible. The techniques described herein can be used in many different use cases, such as: using the channel direction (instead of the road direction) to represent the coordinates of vehicle 201; the projection of the position of vehicle 100 onto the channel of vehicle 201; a normalized amount relative to the projection of vehicle 100 (e.g., determining the longitudinal difference of vehicle 100 by subtracting the longitudinal position), etc.

[0049] Each of the vehicles 100 and 201 collects data about itself and sends it to a processing system for processing. According to one example, vehicle 201 collects data about itself and sends the data to the processing system 110 of vehicle 100 for processing. According to another example, vehicles 100 and 201 collect data about themselves and send the data to a remote processing system 150.

[0050] The data (i.e., information) provided by vehicle 201 can indicate the vehicle's perception in the environment and can generate an estimated kinematic state of vehicle 201, channel allocation, vehicle intent (e.g., direction of travel, intent to merge / change channels, etc.), and / or any other information useful for decision-making. At module 206, data about vehicle 201 is processed. Information from vehicle 201 can be in the form of one-dimensional (1D) feature vectors. For example, 1D feature vectors from vehicle 201 can be combined (or concatenated) into a two-dimensional (2D) matrix (e.g., the x-feature of the vehicle), the order of which along the vehicle's dimensions is arbitrary. State table 210 shows an example of such a 2D matrix. In particular, data received from vehicle 201 can be stored as state vectors in state table 210. In this example, state table 210 includes data for vehicle 201a in the first row, data for vehicle 201b in the second row, and data for vehicle 201c in the third row. Data for other vehicles can also be stored in state table 210. For each row, state table 210 includes data for the position (Δs), velocity (v), and any other suitable information (e.g., channel allocation, acceleration, vehicle orientation relative to its channel, channel curvature, etc.). For example, regarding channel allocation, a vehicle traveling on the same channel as vehicle 100 can have a value of "0", while a vehicle on the adjacent channel to the right can have a value of "-1", and a vehicle on the adjacent channel to the left can have a value of "+1". This approach can also be used in non-parallel channel configurations with any other values, so ultimately, any channel can have different values ​​to indicate the vehicle assigned to that channel, regardless of the spatial relationship between channels. It should be understood that the position of the vehicle is determined based on its current position relative to position s=0. For example, position s=0 is... Figure 2 The position is shown as line 205, and the position value of each of the vehicles 201 is determined relative to that position. In this way, the values ​​stored in the state vector table 210 of the vehicles 201 are normalized to the vehicles 100. Thus, in this example, as shown in state table 210, the position value Δs of vehicle 201a is 53, the position value Δs of vehicle 201b is 37, and the position value Δs of vehicle 201c is -27. State table 210 also includes the velocities of vehicles 201a, 201b, and 201c, respectively, of 27, 20, and 22.5.

[0051] To learn meaningful information about the interactions between vehicles 100 and 201, the vehicle matrix (e.g., state table 210) is processed at module 212. Specifically, module 212 processes the vehicle matrix using a self-attention mechanism by stacking N multi-head self-attention blocks, where the order of entities (vehicle vectors) in the matrix is ​​invariant. Any neural network layer in module 212 can be an input-to-output projection, which is learned to be optimal for a given task (e.g., decision making). Stacking these layers increases the expressive power of the neural network formed by these layers. The attention mechanism has several layers and mathematical operations between them, as follows: it learns the projection Q from the input vector to the query vector; it learns the projection K from the input vector to the key vector; and it learns the projection V from the input vector to the value vector. These three projections (Q, K, and V) can be characterized by vectors of any input entity (e.g., the feature vector of vehicle 201), with the three resulting vectors determined for each entity (e.g., each vehicle 201a, 201b, 201c). Then, for each entity, the query vector is matched against the key vectors of other entities (to account for similarity or distance between entities), resulting in a similarity score. The similarity scores of other entities are multiplied by their vector values, and the resulting vectors are summed (e.g., a weighted sum of the value vectors of other entities weighted by their similarity scores). This produces a single 1D vector output for each entity, incorporating pairwise interactions with other entities. Sequentially stacking multiple attention mechanisms has the ability to incorporate multi-level interactions between entities.

[0052] The attention output from module 212 is fed into max-pooling module 214, which performs a max-pooling operation that runs on the feature dimension and aggregates specific feature information from vehicle 201. The max-pooling operation obtains the maximum value of each column from state table 210. This generates a first output associated with vehicle 201, referred to as the global embedding vector, which can be further used with other operations as described further herein and is fed into module 216. Continuing from the above, a stack of multiple self-attention mechanisms (e.g., module 212) produces the same data structure as the input, i.e., a 2D unordered matrix, except that now each entity's vector contains not only its own features but also information about the entity's interactions with other entities. In a sense, each vector represents the state of the vehicle relative to the rest of the environment. This is a "per-vehicle embedding vector," where the values ​​are latent (e.g., their structure is learned, not user-defined). Similar to the method described in this paper, the max-pooling operator at max-pooling module 214 aggregates each vehicle embedding vector into a single fixed-size vector (which could alternatively be average pooling or any other aggregation operator) on the feature dimension. This is a 1D vector representing relevant information about vehicle 201 and their interactions with each other. Since this part of architecture 200 (e.g., modules 210, 212, 214) is invariant to the order in which vehicle information is fed to it, encoding information about the environment in a non-trivial spatial order (e.g., lane merging and lane separation) is relatively straightforward.

[0053] Data (i.e., information) about the vehicle 100 is processed at module 218. Processing at module 218 can occur before, during (e.g., simultaneously, concurrently, etc.), and / or after processing at module 206. Specifically, data about the vehicle 100 (also referred to as the “target vehicle”) can include motion state information represented in the coordinate system of road 202 (e.g., longitudinal and lateral positions and their derivatives). Data about the vehicle 100 can also include channel information for the vehicle 100. As an example, adjacent channel information can be represented by a vector with binary values, where zero (0) indicates the absence of a channel and one (1) indicates the presence of a channel. For example, Figure 2The vector 240 shown indicates that there is no channel to the left of vehicle 100, a channel exists at the current location of vehicle 100, and a channel exists to the right of vehicle 100. Additional information about the environmental attributes or target of vehicle 100 can be cascaded, such as: the encoding of the target channel (e.g., channel 203b) relative to the current channel assignment (e.g., channel 203a) in the event of a desired channel change / merge maneuver, the existence of adjacent channels, the connectivity between adjacent channels (the ability to change channels), road length, etc.

[0054] Information from vehicle 100 is fed into M stacked residual blocks represented by module 218. Module 218 may include a first convolutional unit 220, a rectified linear unit (ReLU) 222, a second convolutional unit 224, and a skip connection 226, which adds the output of module 218 to its input in such a way that if the output of module 218 is all zero (then input = output), the information is provided to the "skipped" module 218. This mechanism improves the robustness of the network training process. Module 218 processes information from vehicle 100 by performing a first convolution operation using the first convolutional unit 220, performing an activation operation using ReLU 222 to transform the result of the first convolution operation, and performing a second convolution operation on the result of the activation operation using the second convolutional unit 224. The first convolutional unit 220 and the second convolutional unit 224 learn optimal weights for the weighted-sum kernel operation based on the input. ReLU 222 uses an activation function that independently takes any input "x" and outputs y = f(x), where the function "f" can be any nonlinear mathematical operation, such as ReLU, LeakyReLU, Tanh, Sigmoid, etc. At module 226, the raw data from vehicle 100 and the result of the second convolution operation can be combined. The output of module 218 (e.g., a second output associated with vehicle 100) is fed into module 216.

[0055] A first output associated with vehicle 201 (from module 206) and a second output associated with vehicle 100 (from module 218) are input to module 216, which combines or concatenates the two outputs into a concatenated vector. In the case of learning a decision task via the Actor-Critic paradigm (a Reinforcement Learning technique), the concatenated vector is fed into modules 228 and 230, which are fully connected layers responsible for making decisions regarding the operation of vehicle 100. Module 228 (also referred to as the “actor head”) outputs a score (or distribution) of actions available to the policy (“policy output”), and module 230 (also referred to as the “evaluator head”) determines a value associated with the current state of the agent in the environment (“value output”). According to one or more embodiments described herein, each of modules 228 and 230 may contain fully connected layers, convolutional layers, and / or residual modules.

[0056] Figure 3 A flowchart depicts a method 300 for processing environmental data of a vehicle according to one or more embodiments described herein. Method 300 can be performed by any suitable system or device, such as… Figure 1 Processing system 110 Figure 4 The processing system 400 or any other suitable processing system and / or processing device (e.g., processor). Now refer to Figure 1 and / or Figure 2 The component description method is 300, but is not limited to it.

[0057] At box 302, processing system 110 uses a self-attention mechanism (e.g., see...). Figure 2 Module 206) processes the first data associated with at least one other vehicle to generate a first output associated with at least one other vehicle. For example, processing system 110 (or another suitable processing system) receives data from vehicles 201a, 201b, 201c and processes it as described herein. This may include, for example, receiving data from vehicle 201 as a 1D feature vector, combining the data into a 2D matrix (e.g., state table 210), processing the data using one or more self-attention blocks, and using a max-pooling operation to determine the maximum value from each column of the 2D matrix.

[0058] At block 304, processing system 110 uses stacked residual modules to process second data associated with static portions of the target vehicle and its environment (e.g., adjacent waterways, etc.) to generate a second output associated with the target vehicle. For example, processing system 110 (or another suitable processing system) receives (or collects) data about vehicle 100 and uses stacked residual blocks to process the data (e.g., see...). Figure 2 (Module 218). The processing may include performing a first convolution operation using the first convolution unit 220, performing an activation operation using ReLU 222 to transform the result of the first convolution operation, and performing a second convolution operation on the result of the activation operation using the second convolution unit 224.

[0059] According to one or more embodiments, blocks 302 and 304 may operate simultaneously, concurrently, sequentially, or in any other suitable order.

[0060] At block 306, processing system 110 concatenates the first and second outputs to generate a concatenated vector. The first output associated with vehicle 201 (from module 206) and the second output associated with vehicle 100 (from module 218) are input to module 216, which combines or concatenates the two outputs into a concatenated vector.

[0061] At block 308, processing system 110 makes driving decisions for the target vehicle based at least in part on cascaded vectors. For example, cascaded vectors are fed into modules 228 and 230, which are fully connected layers and responsible for making decisions about the operation of vehicle 100. Driving decisions may include driving strategies and values ​​associated with the agent's state in the environment. For example, module 228 determines the driving strategy (e.g., changing course), and module 230 determines the "state value." In an "executor-evaluator" paradigm (e.g., a private case of RL algorithms, hence the two modules 228 and 230), the "value output" of module 230 is primarily (though not necessarily) used for training purposes and is largely ignored during inference (during driving), while the "policy output" of 228 has scores for some or all available actions, allowing the autonomous agent to select the action with the highest score to execute during inference.

[0062] At box 310, processing system 110 controls the target vehicle at least in part based on this decision. Controlling the target vehicle may include implementing a driving strategy. For example, if the driving strategy is to merge a channel from channel 203A into channel 203B, and the value associated with the driving strategy is to increase the speed by 5 mph, then processing system 110 may cause vehicle 100 to increase its speed and turn toward channel 203B. This improves vehicle operation by enabling vehicle 100 to implement driving decisions based on its own data and data (e.g., motion data) from other vehicles (e.g., vehicles 201a, 201b, 201c). This allows vehicle 100 to operate more efficiently (e.g., vehicle 100 could achieve its objectives, such as driving to a specific location, more efficiently by having better behavior from vehicle 201). As a result of this technology, the vehicle 100 can also operate more autonomously (e.g., requiring less driver input / takeover) because it is able to extract the essence of driving decisions more effectively from the available inputs.

[0063] It may also include other processes, and it should be understood that... Figure 3 The processes described herein are illustrative, and other processes may be added or existing processes may be removed, modified or rearranged without departing from the scope and spirit of this disclosure.

[0064] It should be understood that this disclosure can be implemented in conjunction with any other type of computing environment now known or developed in the future. For example, Figure 4 A block diagram of a processing system 400 for implementing the techniques described herein is depicted. In the example, the processing system 400 has one or more central processing units (processors) 421a, 421b, 421c, etc. (collectively or collectively referred to as processors 421 and / or processing devices). In aspects of this disclosure, each processor 421 may include a Reduced Instruction Set Computer (RISC) microprocessor. Processors 421 are connected via a system bus 433 to system memory (e.g., random access memory (RAM) 424) and various other components. Read-only memory (ROM) 422 is connected to the system bus 433 and may include a basic input / output system (BIOS) that controls certain basic functions of the processing system 400.

[0065] Further depictions include an input / output (I / O) adapter 427 and a network adapter 426 connected to the system bus 433. The I / O adapter 427 may be a Small Computer System Interface (SCSI) adapter that communicates with the hard disk 423 and / or storage device 425 or any other similar component. The I / O adapter 427, hard disk 423, and storage device 425 are collectively referred to herein as mass storage device 434. An operating system 440 for execution on the processing system 400 may be stored in the mass storage device 434. The network adapter 426 interconnects the system bus 433 with an external network 436, enabling the processing system 400 to communicate with other such systems.

[0066] A display (e.g., a display monitor) 435 is connected to the system bus 433 via a display adapter 432, which may include a graphics adapter for improving the performance of graphics-intensive applications and video controllers. In one aspect of this disclosure, adapters 426, 427, and / or 432 may be connected to one or more I / O buses, which are connected to the system bus 433 via an intermediate bus bridge (not shown). Suitable I / O buses for connecting peripheral devices (such as hard disk controllers, network adapters, and graphics adapters) typically include common protocols such as Peripheral Component Interconnect (PCI). Additional input / output devices are shown connected to the system bus 433 via a user interface adapter 428 and the display adapter 432. A keyboard 429, a mouse 430, and a speaker 431 (or other suitable input and / or output, such as a touchscreen in an infotainment system) may be interconnected to the system bus 433 via a user interface adapter 428, which may include, for example, a super I / O chip integrating multiple device adapters into a single integrated circuit. One or more of cameras 120-123 and 130-133 are also connected to system bus 433.

[0067] In some aspects of this disclosure, the processing system 400 includes a graphics processing unit 437. The graphics processing unit 437 is a dedicated electronic circuit designed to manipulate and modify memory to accelerate the creation of images in a frame buffer intended for output to a display. Typically, the graphics processing unit 437 is highly efficient in manipulating computer graphics and image processing and has a highly parallel architecture, making it more efficient than general-purpose CPUs used for algorithms that perform parallel processing of large blocks of data.

[0068] Therefore, as configured herein, the processing system 400 includes processing capabilities in the form of a processor 421, storage capabilities including system memory (e.g., RAM 424) and mass storage 434, input devices such as a keyboard 429 and a mouse 430, and output capabilities including a speaker 431 and a display 435. In some aspects of this disclosure, a portion of the system memory (e.g., RAM 424) and the mass storage 434 jointly store an operating system 440 to coordinate the functionality of the various components shown in the processing system 400.

[0069] Various examples of this disclosure have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described techniques. The terminology used herein has been chosen to best explain the principles of the technology, its practical application, or a technical improvement upon technology found in the market, or to enable those skilled in the art to understand the technology disclosed herein.

[0070] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, this technology is intended to be limited to the specific embodiments disclosed, but will include all embodiments falling within the scope of this application.

Claims

1. A computer-implemented method for processing environmental data of at least one other vehicle relative to a target vehicle, the method comprising: The processing device uses a self-attention mechanism to process the first data associated with the at least one other vehicle to generate a first output associated with the at least one other vehicle; The processing device uses stacked residual modules to process the second data associated with the target vehicle to generate a second output associated with the target vehicle; The first output and the second output are cascaded by the processing device to generate a cascaded vector; The processing device makes driving decisions for the target vehicle based at least in part on the cascaded vectors; as well as The target vehicle is controlled at least in part based on the driving decisions; The method further includes: processing the self-attention mechanism of the vehicle matrix by stacking N multi-head self-attention modules, wherein the order of the vehicle vectors in the vehicle matrix remains unchanged and multiple attention mechanisms are stacked sequentially.

2. The computer-implemented method of claim 1, wherein the first data associated with the at least one other vehicle includes at least one of the estimated motion state of the at least one other vehicle, a route allocation, and the vehicle's intent.

3. The computer-implemented method of claim 1, wherein, The first data includes a first subset of the first data associated with a first other vehicle and a second subset of the first data associated with a second other vehicle.

4. The computer-implemented method according to claim 3, wherein the first subset of the first data is expressed as a first one-dimensional feature vector, and wherein the second subset of the first data is expressed as a second one-dimensional feature vector.

5. The computer-implemented method of claim 4, wherein processing the first data includes combining the first one-dimensional feature vector and the second one-dimensional feature vector into a two-dimensional matrix, wherein the two-dimensional matrix includes speed data associated with the at least one other vehicle and position data associated with the at least one other vehicle.

6. The computer-implemented method of claim 1, wherein the second data associated with the target vehicle includes motion status information and route information.

7. The computer-implemented method of claim 6, wherein, The motion state information is represented in the coordinate system of the road where the target vehicle is located.

8. The computer-implemented method according to claim 6, wherein the channel information is represented by a vector of binary values, where a value of 0 indicates that no channel exists and a value of 1 indicates that a channel exists.

9. The computer-implemented method of claim 1, wherein, Each of the stacked residual modules includes a first convolutional unit, a corrected linear unit, and a second convolutional unit.

10. The computer-implemented method of claim 1, wherein making the driving decision includes determining a driving strategy and values ​​associated with the current state of a subject in the environment, and wherein controlling the target vehicle based at least in part on the decision includes controlling the target vehicle based on the driving strategy and the current state of the subject in the environment.

Images