Automobile perception based on fused sensor maps

Abstract

The embodiment of the invention relates to automobile perception based on fused sensor maps. The present disclosure relates to a method for generating a fused automotive sensor data map, and to an automotive control unit configured to perform the method, and to a vehicle comprising the automotive control unit. A plurality of automotive sensor data maps are obtained, each automotive sensor data map based on sensor data captured by a corresponding automotive sensor of a vehicle. Furthermore, at least one calibration matrix is obtained, which defines a transformation between the sensor data nodes of the at least two vehicle sensor data maps. A fused automotive sensor data map is generated based on the plurality of automotive sensor data maps and the at least one calibration matrix, the fused automotive sensor data map comprising the plurality of automotive sensor data maps and a plurality of fused edges. The fused automotive sensor data map is provided to an automotive perception function implemented by a map neural network.

Description

Technical Field

[0001] The present invention relates generally to automotive computer vision and, more specifically, to processing sensor data from a plurality of automotive sensors of a vehicle. Background Art

[0002] In order to implement advanced driver assistance systems (ADAS), modern vehicles typically include multiple automotive sensors of various sensor types that can provide sensor data to the vehicle based on the vehicle's environment. The sensor data of multiple automotive sensors can be processed by a vehicle processing unit to identify objects such as other vehicles, vulnerable road users (VRUs), traffic signs, traffic lights, and lane markings. The sensor data of multiple sensors can be combined in a two-dimensional grid or a three-dimensional grid so that the sensor data can be jointly processed using processing methods used in image processing. In order to form such a grid, empty cells need to be added, which increases storage requirements and prolongs processing time because empty cells also need to be processed. In addition, the formation of such a grid may cause discretization of sensor data, resulting in information loss.

[0003] In order to solve these problems, there is a method for forming an initial automobile sensor data graph based on sensor data of one automobile sensor among multiple automobile sensors. Based on the initial automobile sensor data graph, sensor data of other automobile sensors among multiple sensors are added to each node of the initial automobile sensor data graph to enrich the information of the nodes in the automobile sensor data graph, thereby forming a connected automobile sensor data graph. Then, the connected automobile sensor graph can be processed by a machine learning model (such as an artificial neural network). However, the connected automobile sensor graph only includes the nodes and edges of the initial automobile sensor. Since the edge encodes the relative distance and position information between the data points of the automobile sensor used to form the initial automobile sensor, the connected automobile sensor graph does not include any relative distance and position information of other automobile sensors among multiple automobile sensors. As the nodes of the initial automobile sensor data graph are enriched, only a subset of the data from other automobile sensors among multiple automobile sensors is used, which results in information loss. In addition, since each node of the connected automobile sensor graph includes sensor data of various sensors and therefore includes various sensor types, it may be necessary to train the machine model multiple times.

[0004] Therefore, an object of the present disclosure is to combine sensor data of multiple automotive sensors together without losing information and to process the sensors in a manner that requires less training effort as well as less computational effort. Summary of the invention

[0005] To achieve this goal, the present disclosure provides a method for generating a fused automotive sensor data graph. The method includes obtaining a plurality of automotive sensor data graphs. Each automotive sensor data graph is based on sensor data captured by a corresponding automotive sensor in a plurality of automotive sensors. Each automotive sensor data graph includes a plurality of sensor data nodes and a plurality of sensor data edges. Each sensor data node includes sensor data captured by a corresponding automotive sensor. Each sensor data edge defines a sensor distance relationship between two sensor data nodes in the plurality of sensor data nodes. The method also includes: obtaining at least one calibration matrix, the at least one calibration matrix defining a transformation between corresponding multiple sensor data nodes in at least two automotive sensor data graphs; and generating a fused automotive sensor data graph based on the plurality of automotive sensor data graphs and the at least one calibration matrix. The fused automotive sensor data graph includes a plurality of fused sensor data nodes and a plurality of fused sensor data edges. The plurality of fused automotive sensor data nodes include all the plurality of sensor data nodes. The plurality of fused automotive sensor data edges include all the plurality of sensor data edges and a plurality of fused edges. Each fused edge defines a fused distance relationship between two sensor data nodes in two plurality of sensor data nodes. Finally, the method includes providing the fused vehicle sensor data graph to a vehicle perception function implemented by a graph neural network and configured to generate perception data based on the fused vehicle sensor data graph.

[0006] The present disclosure also provides an automobile control unit, comprising at least one processing unit and a memory coupled to the at least one processing unit and configured to store machine-readable instructions, wherein the machine-readable instructions enable the at least one processing unit to execute a method for generating a fused automobile sensor data map.

[0007] The present disclosure also provides a vehicle, including a plurality of sensors and a vehicle control unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Examples of the present disclosure will be described with reference to the following drawings, wherein like reference symbols represent like elements.

[0009] Figure 1 A flow chart of a method for generating a fused automotive sensor data map according to an example of the present disclosure is provided.

[0010] Figure 2 Processing sensor data based on generating a fused automotive sensor data graph according to an example of the present disclosure is illustrated.

[0011] Figure 3A and Figure 3B Transformations between data nodes of an automotive sensor data graph according to an example of the present disclosure are illustrated.

[0012] Figure 4 A diagram of automotive sensor data for a camera and a calibration matrix for images captured by the camera according to an example of the present disclosure is illustrated.

[0013] Figure 5 A vehicle including a plurality of automotive sensors according to an example of the present disclosure is illustrated.

[0014] Figure 6 An automotive control unit according to an example of the present disclosure is illustrated.

[0015] It should be understood that the above-mentioned figures are by no means intended to limit the present disclosure. On the contrary, these figures are provided to help understand the present disclosure. Those skilled in the art will readily appreciate that the various aspects of the present invention shown in one figure may be combined with the various aspects in another figure or may be omitted without departing from the scope of the present disclosure. DETAILED DESCRIPTION

[0016] The present disclosure generally provides a method for generating a fused automotive sensor data graph, an automotive control unit configured to generate a fused automotive sensor data graph, and a vehicle including the automotive control unit. The fused automotive sensor data graph combines sensor data captured by multiple automotive sensors of the vehicle in a single graph. To this end, the method and the automotive control unit obtain multiple automotive sensor data graphs, which are sensor data based on multiple automotive sensor data, respectively, and add all sensor data nodes and all sensor data edges as fused automotive sensor data nodes and fused automotive sensor data edges to the fused automotive sensor data graph. In addition, the method and the automotive control unit generate multiple fused edges between sensor data nodes of different automotive sensor data graphs based on a transformation defined by at least one calibration matrix. The fused edges also form part of the fused automotive sensor data edges. Therefore, the fused edges make the fused automotive sensor data graph a connected graph, that is, due to the fused edges, there is a sequence of fused automotive sensor data edges connecting each pair of fused automotive sensor data nodes. The fused automotive sensor data graph is then provided to an autonomous perception function implemented by a graph neural network to generate perception data, i.e., data defining various aspects of the vehicle environment.

[0017] Since the fused automotive sensor data graph includes all the sensor data nodes and sensor data edges of the automotive sensor data graph, the fused automotive sensor data graph retains all the information provided by multiple automotive sensors. In addition, since each fused automotive sensor node includes sensor data of a specific type of sensor, training the graph neural network is simplified compared to a sensor data graph that includes multiple types of sensor data in a node. This is due to the fact that the neurons of the graph neural network that receive the sensor data of the corresponding fused automotive sensor node as input only need to be trained for one type of sensor data. In addition, considering the structure of the fused automotive sensor data graph, all types of sensor data can be trained at the same time.

[0018] The general concept will now be explained with reference to the accompanying drawings, in which Figure 1 A flowchart of a method for generating a fused automotive sensor data graph is provided, and Figures 2 to 4 Further illustrations are given of general aspects of the method or specific aspects thereof. Figure 5 The diagram shows a vehicle including a plurality of vehicle sensors and a vehicle control unit. Finally, Figure 6 An example of an automotive control unit is illustrated in more detail.

[0019] Figure 1 A flow chart of a method 100 for generating a fused automotive sensor data map is provided. Sub-steps of the steps of the method 100 are shown as boxes within the boxes of the corresponding steps. Optional steps are shown as dashed boxes. The method 100 may be performed by Figure 6 The vehicle control unit 800 is used to execute.

[0020] In step 130, the method 100 obtains a plurality of vehicle sensor data maps. Each vehicle sensor data map is based on sensor data captured by a corresponding vehicle sensor 200 of the plurality of vehicle sensors.

[0021] like Figure 5As shown in , the vehicle may include vehicle sensors 200 at various locations in and around the vehicle. Each vehicle sensor 200 may be configured to capture sensor data indicating the environment of the vehicle. To this end, the vehicle sensor 200 may be a radar sensor, which may be configured to emit radio waves so as to determine the distance, angle, and speed of objects around the vehicle based on reflected radio waves. The vehicle sensor 200 may be a light detection and ranging (LIDAR) sensor, which is configured to emit a laser beam so as to determine the distance, angle, and speed of objects around the vehicle based on reflected laser beams. The vehicle sensor 200 may be a camera that captures an image of the vehicle environment. The vehicle sensor 200 may be a thermal imaging camera that captures an image of the vehicle environment based on infrared radiation. It should be understood that LIDAR sensors, radar sensors, or cameras are provided only as examples of sensor types of the vehicle sensor 200. For example, the vehicle sensor may also be an ultrasonic sensor. More generally, the vehicle sensor 200 may be any type of sensor capable of capturing sensor data indicating the vehicle environment. It should also be understood that the plurality of vehicle sensors may include a plurality of sensors of various types. In addition, the same type of automotive sensor 200 may exhibit different properties, such as by being configured to capture sensor data at different ranges, such as close range, mid-range, and long range. For example, a vehicle may include three close range radar sensors, one each at the front and rear of the vehicle, a mid-range to long-range radar sensor at the rear of the vehicle, a lidar sensor at the front of the vehicle, a rear camera at the rear of the vehicle, a front camera at the front of the vehicle, a front camera at the rearview mirror, and a rear close to mid-range radar sensor in each door-mounted exterior mirror. It will be appreciated that a vehicle may include more than one sensor. Figure 5 More or fewer automotive sensors may be shown in and discussed in the above examples.

[0022] The information contained in the sensor data may depend on the type of automotive sensor providing the sensor data. If the automotive sensor is a camera, the sensor data may correspond to image data, such as RGB values ​​and brightness values ​​for each pixel captured by the camera. If the automotive sensor is a radar sensor, the sensor data may correspond to radar data, for example, it may include the return time, intensity, angle, and frequency of the detected radio waves. If the automotive sensor is a LIDAR sensor, the sensor data may correspond to LIDAR data, for example, it may include the return time, angle, and intensity of the detected laser beam. More generally, the sensor data of each automotive sensor 200 may be raw sensor data captured by the corresponding automotive sensor. In some examples, the sensor data may also be sensor data processed by the corresponding automotive sensor 200 before generating the corresponding automotive sensor data map, for example, the processing is to reduce the amount of sensor data to meet storage or complexity requirements, as will be discussed with reference to steps 110 and 120.

[0023] Each automobile sensor data graph includes multiple sensor data nodes and multiple sensor data edges. Therefore, each automobile sensor data graph G S can be expressed as shown in equation (1):

[0024] G Sn =(V Sn ,E Sn ) (1)

[0025] In equation (1), V S It indicates multiple sensor data nodes which can also be called vertices. S indicates multiple sensor data edges, and n indicates the nth car sensor data graph G Sn .

[0026] Multiple sensor data nodes V S It can be expressed as a set of sensor data nodes, including k sensor data nodes v, as shown in equation (2):

[0027] V S =*v1,…,v k + (2)

[0028] Each sensor data node v includes sensor data captured by one of the vehicle sensors 200 discussed above, which may be raw sensor data or processed sensor data.

[0029] Multiple sensor data edge S can be expressed as a set of sensor data nodes including l sensor data edges e, as shown in equation (3):

[0030] E S =*e1,…,e l + (3)

[0031] Each sensor data edge e defines a sensor distance relationship between two sensor data nodes v. Based on the sensor distance relationship, the sensor data edge defines the relative distance between sensor data points captured by one of the automobile sensors 200. Taking a camera as an example, each sensor data node v may include information of a pixel captured by the camera or average information of a pixel group. Each sensor data edge in this example defines the relative distance between the pixel or pixel group and other pixels or pixel groups captured by the camera. Taking a LIDAR sensor as an example, the sensor data may be a point cloud. Therefore, each sensor data node may correspond to the average value of a point or a group of points in the point cloud. Therefore, each sensor data edge e in this example defines the relative distance between points or groups of points in the point cloud. Therefore, the distance relationship may refer to both a two-dimensional distance and a three-dimensional distance. It is understandable that the distance relationship may be used to infer the location information of the corresponding sensor data within each automobile sensor data graph.

[0032] The sensor distance relationship is used as a rule to define which relative distances should be included in the automotive sensor data graph. Therefore, the sensor distance relationship can be one of n nearest neighbor relationships, fixed radius relationships, and weighted distance relationships. In the case where the sensor distance relationship is an n nearest neighbor relationship, the multiple sensor data edges include edges incident on a sensor data node v corresponding to the n shortest relative distances between a sensor data point associated with a sensor data node v and a sensor data point of an adjacent sensor data node. In the case where the sensor distance relationship is a fixed radius relationship, the multiple sensor data edges include edges incident on a sensor data node v, the edge corresponding to the relative distance between a sensor data point associated with a sensor data node v and a sensor data point of an adjacent sensor data node, the relative distance being less than a threshold distance. In the case where the sensor distance relationship is a weighted distance relationship, the multiple sensor data edges include edges between all sensor data nodes of the corresponding sensor data graph, wherein each relative distance is weighted by the average distance between all sensor data nodes of the corresponding automotive sensor data graph. It should be understood that this distance relationship is merely an example of a distance relationship and that sensor data edges within each respective vehicle sensor data map may be defined based on any distance relationship suitable for use as a rule defining which relative distances are indicated in the respective vehicle sensor data map.

[0033] To obtain a plurality of vehicle sensor data maps, method 100 may include steps 131 and 132. In step 131, method 100 may receive sensor data captured by each vehicle sensor. In step 132, method 100 may generate each vehicle sensor data map based on the received sensor data.

[0034] The method 100 may include step 110, wherein the method 100 generates each automotive sensor data graph using a feature extractor configured to extract one or more features of the corresponding automotive sensor. The feature extractor may be any type of data processing method configured to identify one or more features. More precisely, the feature extractor may be configured to transform the sensor data from a sensor data space to a feature space, wherein the number of features to be extracted defines the dimensionality of the feature space. Therefore, the sensor data nodes of the automotive sensor data graph generated using the feature extractor may include vectors indicating one or more features of the identified sensor data. For example, the feature extractor may be a neural network, such as a residual neural network (ResNet). By using a feature extractor to generate each automotive sensor data graph, the amount of sensor data and thereby the amount of sensor data nodes may be reduced, resulting in a smaller automotive sensor data graph and thereby a smaller fused automotive sensor data graph, without or as little loss of information about the vehicle environment as possible.

[0035] The method 100 may include step 130, wherein the method 100 reduces the plurality of nodes and the plurality of edges of each automotive sensor data graph. To reduce the plurality of nodes and the plurality of edges in step 130, the method 100 may employ one of an autoencoder, principal component analysis, clustering, or distance-based grouping.

[0036] Although steps 110 and 120 are illustrated and discussed as steps before step 130, i.e., the step of obtaining a plurality of automobile sensors, it is understood that steps 110 and 120 may be sub-steps of step 130, similar to steps 131 and 132. That is, as part of obtaining a plurality of automobile sensor data graphs, method 100 may generate a plurality of automobile sensor data graphs using a feature extractor based on the sensor data received in step 131, and may then reduce a plurality of nodes and a plurality of edges, as discussed with respect to step 120.

[0037] The method 100 may include step 140 , in which the method 100 translates each vehicle sensor data map from a sensor coordinate system of the corresponding sensor to a vehicle coordinate system. Figure 5Each vehicle sensor 200 shown in FIG. 1 has a sensor coordinate system, as exemplarily illustrated by a corresponding three-dimensional Cartesian coordinate system for two vehicle sensors 200 . In order to enable subsequent fusion of multiple vehicle sensor data maps and processing of the sensor data contained in the multiple vehicle sensor data maps in step 170 , each vehicle sensor data map can be translated from the corresponding sensor coordinate system to the vehicle coordinate system. Figure 5 As shown, the vehicle coordinate system may have an origin centered at a vehicle component, such as the center of a rear axle of the vehicle. The vehicle coordinate system may also be centered at any other component of the vehicle or the center of mass of the vehicle. It should be understood that translating each automotive sensor data graph includes translating the sensor data contained in the corresponding sensor data node and the relative distance contained in the corresponding sensor data edge so that the sensor data and the relative distance are referenced to the vehicle coordinate system.

[0038] In step 150, the method 100 obtains at least one calibration matrix. The at least one calibration matrix defines a transformation between corresponding multiple sensor data nodes of at least two vehicle sensor data maps. In other words, the calibration matrix enables mapping sensor data nodes of two or more vehicle sensor data maps to each other. Figure 4 Let's discuss the concept of a calibration matrix.

[0039] Figure 4 An example of a calibration matrix 4002 overlaid on sensor data 2101 of an automobile sensor 2001, illustrated as a camera image, and an automobile sensor data graph 3001 generated based on the sensor data 2101 is provided. The calibration matrix 4002 corresponds to the position of the sensor data of the automobile sensor 2002, and indicates the position of all data points of the sensor data relative to the automobile sensor data graph 3001. As can be seen from this example, by indicating the position of sensor data points of the sensor data of one automobile sensor 200 relative to the sensor data points of the sensor data of another automobile sensor 200, the calibration matrix thus defines a transformation or mapping between corresponding multiple sensor data nodes.

[0040] To achieve the purposes discussed above, at least one calibration matrix can be based on the calibration of two automotive sensors relative to each other. That is, the calibration matrix can be based on determining how sensor data captured by one automotive sensor 200 is related to sensor data captured by another automotive sensor 200 in view of the vehicle's environment.

[0041] The method 100 may obtain at least one calibration matrix in step 150. That is, if the method 100 obtains a single calibration matrix, the calibration matrix determined for one automotive sensor 200 relative to another automotive sensor 200 may be used as an approximation of the sensor data position mapping between all automotive sensors 200 to simplify the subsequent generation of a fused automotive sensor map. Depending on the accuracy requirements and the differences between the positions of the plurality of automotive sensors, step 150 may also include step 151, in which the method 100 obtains a plurality of calibration matrices, each calibration matrix defining a transformation between a corresponding plurality of sensor data nodes of two corresponding automotive sensor data maps. In other words, if the method 100 is implemented with step 151, the method 100 may then generate a fused automotive sensor data map based on one calibration matrix for each pair of automotive sensors 200. It will be appreciated that method 100 may obtain a single calibration matrix, multiple calibration matrices, or any number of calibration matrices between these two extremes, such as a calibration matrix that transforms or maps sensor data points of sensor data of a front vehicle sensor 200 between each other, a calibration matrix that transforms or maps sensor data points of sensor data of a rear vehicle sensor 200 between each other, and two calibration matrices that similarly map sensor data points of sensor data of a side vehicle sensor 200 between each other.

[0042] At step 150 , method 100 generates a fused vehicle sensor data map based on a plurality of vehicle sensor data maps and at least one calibration matrix.

[0043] The fused automobile sensor data graph includes a plurality of fused sensor data nodes and a plurality of fused sensor data edges. Therefore, each fused automobile sensor data graph G F It can be expressed as shown in equation (4):

[0044] G F =(V F ,E F ) (4)

[0045] In equation (4), V F It indicates multiple fusion sensor data nodes which can also be called vertices. F Marks the edge of multiple fused sensor data. Multiple V F Including all multiple sensor data nodes. Based on equations (1) and (2), multiple fusion sensor data nodes V F It can be expressed as shown in equation (5):

[0046] V F =*V S1 ,…,V Sn + (5)

[0047] In addition, multiple fusion sensor data edge E F Including all multiple sensor data edges and multiple fusion edges E fu Based on equations (1) and (3), multiple fused sensor data edges E F can be expressed as shown in equation (6):

[0048] E F ={E S1 ,…,E Sn ,E fu} (6)

[0049] Multiple fused edges E fu can be expressed as multiple m fused edges e fu , as shown in equation (7):

[0050] E F ={e fu1 ,…,e fum} (7)

[0051] Each fused edge fu A fusion distance relationship between two sensor data nodes in the two plurality of sensor data nodes is defined. Similar to the sensor data edge e discussed above, the fusion edge e fu The relative distance between a sensor data point captured by one vehicle sensor 200 and another sensor data point captured by another vehicle sensor 200 is defined based on the fused distance relationship. Therefore, the sensor data edge e and the fused edge e fu The difference between is that the sensor data edge defines the relative distance between sensor data nodes of the same automobile sensor data graph, while the fusion edge defines the relative distance between sensor data nodes of different automobile sensor data graphs. Therefore, the fusion edge connects multiple sensor data nodes to each other to ensure that the fused automobile sensor data graph is a connected graph.

[0052] Similar to the sensor distance relationship, the fused distance relationship may be one of an n-nearest neighbor relationship, a fixed radius relationship, and a weighted distance relationship.

[0053] Step 160 may include step 161, wherein the method 100 generates each fused edge e between two sensor data nodes of the two plurality of sensor data nodes by determining a distance between the two sensor data nodes based on at least one calibration matrix. fuIf method 100 obtains a plurality of calibration matrices as part of step 150, step 160 may include step 162, wherein method 100 generates each fused edge e between two sensor data nodes of the plurality of sensor data nodes by determining a distance between the two sensor data nodes based on a corresponding calibration matrix of the plurality of calibration matrices. fu In other words, the method 100 may determine relative distances between sensor data nodes of different automotive sensor data graphs based on at least one calibration matrix, and include the determined relative distances in the fusion edge graph. fu middle.

[0054] Step 160 may also include a sub-step (not shown) in which method 100 fuses the sensor data contained in each sensor data node. In other words, as part of step 160, method 100 may combine the sensor data contained in each sensor node to further reduce storage and computation requirements by mapping the included sensor data to a reduced dimensional sensor data space. This sensor data fusion across the vehicle sensor data graph may also improve the quality of the results of 170.

[0055] As mentioned above about Figure 4 As discussed, at least one calibration matrix defines a transformation or mapping between corresponding plurality of sensor data nodes by indicating the positions of sensor data points of sensor data of one vehicle sensor 200 relative to sensor data points of sensor data of another vehicle sensor 200 . Figure 3A and Figure 3B The generated blended edge is shown in view of the indication provided by at least one calibration matrix. fu .

[0056] Figure 3A Shown are car sensor data graph 3001 and car sensor data graph 3002, which have been connected by exemplary fused edges 5101 and 5102. Figure 3A In FIG. 5 , a data sensor node is highlighted to illustrate the generation of fused edges 5101 based on at least one calibration matrix. Figure 3B In the embodiment, the relative position of the highlighted data sensor node relative to the vehicle sensor data map 3001 and the corresponding transformation 400 determined based on at least one calibration matrix are determined. T , showing the highlighted data sensor node. Based on the transformation 400 T , the relative distance between the highlighted data sensor node and the car sensor data graph 3001 is determined and included in the fused edge 5101, as shown Figure 3B The changed positions of the sensor nodes are indicated by the arrows and highlighted data.

[0057] In step 170, method 100 provides the fused vehicle sensor data graph to the vehicle perception function. The vehicle perception function is implemented by a graph neural network and is configured to generate perception data based on the fused vehicle sensor data graph. The vehicle perception function can be any type of vehicle computer vision function that is configured to obtain information about the vehicle environment from sensor data of multiple vehicle sensors contained in the fused vehicle sensor data graph. For example, the vehicle perception function can be configured to implement semantic segmentation, instance segmentation, and / or object detection. It should be understood that these functionalities are merely provided as exemplary functionalities implemented by the vehicle perception function, and the vehicle perception function can implement any vehicle computer vision function that helps the vehicle implement one or more ADAS provided in the vehicle. Therefore, the perception data can be configured to be processed by one or more ADAS.

[0058] In some examples of the present disclosure, in addition to the processing implemented by the graph neural network, the vehicle perception function may also include more processing steps. For example, in order to improve the perception data provided by the vehicle perception function, deep fusion-based processing may be included, that is, by updating the sensor data between the fused vehicle sensor data nodes and thereby by exchanging information between the fused vehicle sensor data nodes. As another example, the vehicle perception function may also include Kalman filtering.

[0059] A graph neural network can be any type of neural network that accepts a graph as input and outputs one or more pieces of information as perception data, such as recognized objects. As discussed above, since each fused automobile sensor data node in the fused automobile sensor data graph only includes sensor data of a specific type of automobile sensor, the training of the graph neural network becomes less complex because each neuron only needs to be trained with training data corresponding to the corresponding sensor type.

[0060] Figure 2 The method 100 is illustrated to process the vehicle sensor data to generate a fused vehicle sensor data graph using a graph neural network to process the fused vehicle sensor data graph. To this end, Figure 2 The diagram illustrates vehicle sensors 2001 - 2003 , vehicle sensor data graphs 3001 - 3003 , a calibration matrix 400 , a fused vehicle sensor data graph 500 , and a vehicle perception process 600 .

[0061] For each of the vehicle sensors 2001 to 2003, Figure 2Exemplary sensor data 2101 to 2103, and sensor processing 2201 to 2203 are illustrated. If method 100 obtains multiple automotive sensor data maps without generating them, sensor processing 2201 to 2203 may optionally implement steps 110 and 120. If method 100 generates automotive sensor data maps, sensor processing 2201 to 2203 may be omitted.

[0062] Figure 2 The calibration matrix 400 illustrated in FIG. 4 includes three separate calibration matrices 4001 to 4003. The calibration matrix 400 is illustrated in this manner to illustrate the fact that there may be at least one calibration matrix up to a plurality of calibration matrices 4001 to 4003, depending on how many calibration matrices are obtained in step 150.

[0063] Figure 2 The fused automobile sensor graph 500 illustrated in FIG. 5 includes an automobile sensor data graph 3001, which is connected to an automobile sensor data graph 3002 via two fused edges, which is again connected to an automobile sensor data graph 3003 via two fused edges. It should be understood that this example is merely an oversimplified example. For example, the automobile sensor data graph 3003 may also be connected to the automobile graph 3001 based on the generation of multiple fused edges. In addition, the fused graph 500 may be a three-dimensional graph, because at least some of the sensor data may be a point cloud.

[0064] The vehicle perception processing 600 may include a vehicle perception function 610, as discussed above with respect to step 170, which generates perception data 610O. The perception data 610O may include various acquired information, as indicated by various boxes within the perception data 610O. The perception data 610O may then be provided to a plurality of ADAS 6201-6203.

[0065] As mentioned above, the method 100 may be implemented by the vehicle control unit 800 , which will be discussed below.

[0066] Figure 6 An automotive control unit 800 is shown configured to perform the method 100. The automotive control unit 800 may include a processor 810, a graphics processing unit (GPU) 820, an automotive processing system 830, a memory 840, a removable storage device 850, a storage device 860, a cellular interface 870, a global navigation satellite system (GNSS) interface 880, and a communication interface 890.

[0067] The processor 810 can be any kind of single-core or multi-core processing unit using a reduced instruction set (RISC) or complex instruction set (CISC). Exemplary RISC processing units include ARM-based cores or RISC V-based cores. Exemplary CISC processing units include x86-based cores or x86-64-based cores. The processor 810 can execute instructions that cause the automotive control unit 800 to perform method 100d. The processor 810 can be directly coupled to any component of the computing device 800, or can be directly coupled to the memory 830, the GPU 820, and the device bus.

[0068] The GPU 820 can be any kind of processing unit that is optimized for processing graphics-related instructions or more generally for parallel processing of instructions. Thus, the GPU 420 can be configured to generate information, such as ADAS information or telemetry data display, to the vehicle driver, for example, via a head-up display (HUD) or a display arranged within the driver's field of view. The GPU 820 can be coupled to the HUD and / or display via connection 820. The GPU 820 can also execute a portion of the method 100 to enable fast and parallel processing of instructions related to the method 100, such as processing sensor data nodes while generating a fused automotive sensor data graph. It should be noted that in some embodiments, the processor 810 can determine that the GPU 820 does not need to execute instructions related to the method 100. The GPU 820 can be directly coupled to any component of the computing device 400, or can be directly coupled to the processor 810 and the memory 830. In some embodiments, the GPU 420 can also be coupled to the device bus

[0069] The automotive processing system 830 may be any kind of system on a chip that is configured to provide trillions of operations per second (TOPS) to enable the automotive control unit 800 to implement one or more ADAS and the method 100 while driving. The automotive processing system 830 may interface only with the processor 810, or may interface with other devices via a system bus.

[0070] Memory 840 may be any kind of fast memory that enables processor 810, GPU 820, and vehicle processing system 830 to store instructions for fast retrieval during processing instructions, and to cache and buffer data. Memory 840 may be a unified memory coupled to processor 810, GPU 820, and vehicle processing system 830 to enable memory 840 to be allocated to processor 810, GPU 820, and vehicle processing system 830 as needed. Alternatively, processor 810, GPU 820, and vehicle processing system 830 may be coupled to separate processor memory 840a, GPU memory 840b, and vehicle processing system memory 840c.

[0071] The removable memory 850 may be a storage device that can be removably coupled to the vehicle control unit 800. Examples include a digital versatile disc (DVD), a compact disc (CD), a universal serial bus (USB) storage device (such as an external SSD), or a magnetic tape. It should be noted that the removable memory 850 may store data (such as instructions of the method 100), or may be omitted.

[0072] Storage device 860 may be a storage device that enables storage of program instructions and other data. For example, storage device 860 may be a hard disk drive (HDD), a solid state disk (SSD), or some other type of non-volatile memory. For example, storage device 860 may store instructions of method 100.

[0073] The removable memory 850 and the storage device 860 may be coupled to the processor 810 via a system bus. The system bus may be any kind of bus system that enables the processor 810 and the optional GPU 820 and the vehicle processing system 830 to communicate with other devices of the vehicle control unit 800. For example, the bus 440 may be a peripheral component interconnect express (PCIe) bus or a serial AT attachment (SATA) bus.

[0074] The cellular interface 870 may be any kind of interface that enables the vehicle control unit 800 to communicate via a cellular network, such as a 4G network or a 5G network.

[0075] The GNSS interface 880 may be any kind of interface that enables the vehicle control unit 800 to receive position data provided by a satellite network, such as the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), or Galileo.

[0076] The communication interface 890 may enable the computing device 400 to interface with external devices directly or via a network via the connection 480C. For example, the communication interface 480 may enable the computing device 400 to couple to a wired or wireless network, such as Ethernet, Wifi, a controller area network (CAN) bus, or any bus system suitable in a vehicle. For example, the vehicle control unit 800 may be coupled to the vehicle sensor 200 to perform the method 100, or may be coupled to the engine and brakes to control the vehicle based on the sensory data generated by the method 100. The communication interface may also include a USB port or a serial port to enable direct communication with external devices.

[0077] The vehicle control unit 800 may be integrated with the vehicle, for example located directly under the vehicle's cab or in the vehicle's trunk.

[0078] The present invention can be further illustrated by the following examples.

[0079] In an example, a method for generating a fused automotive sensor data graph includes obtaining a plurality of automotive sensor data graphs, each automotive sensor data graph being based on sensor data captured by a corresponding automotive sensor of a plurality of automotive sensors. Each automotive sensor data graph includes a plurality of sensor data nodes and a plurality of sensor data edges. Each sensor data node includes sensor data captured by a corresponding automotive sensor. Each sensor data edge defines a sensor distance relationship between two sensor data nodes of the plurality of sensor data nodes. The example method also includes obtaining at least one calibration matrix. The at least one calibration matrix defines a transformation between corresponding plurality of sensor data nodes in at least two automotive sensor data graphs. The example method also includes generating a fused automotive sensor data graph based on the plurality of automotive sensor data graphs and the at least one calibration matrix. The fused automotive sensor data graph includes a plurality of fused sensor data nodes and a plurality of fused sensor data edges. The plurality of fused automotive sensor data nodes includes all of the plurality of sensor data nodes. The plurality of fused automotive sensor data edges includes all of the plurality of sensor data edges and a plurality of fused edges. Each fused edge defines a fused distance relationship between two sensor data nodes of two of the plurality of sensor data nodes. The example method also includes providing the fused automotive sensor data graph to an automotive perception function. The vehicle perception function is implemented by a graph neural network and is configured to generate perception data based on a fused vehicle sensor data graph.

[0080] In an example approach, perception data may be configured to be processed by one or more advanced driver assistance systems.

[0081] The example method may also include translating each vehicle sensor data map from a sensor coordinate system of the corresponding sensor to a vehicle coordinate system.

[0082] The example method may also include generating each vehicle sensor data map using a feature extractor configured to extract one or more features of the corresponding vehicle sensor.

[0083] The example method may also include reducing the number of nodes and the number of edges of each automotive sensor data graph using one of an autoencoder, principal component analysis, clustering, or distance-based grouping.

[0084] In the example method, obtaining a plurality of vehicle sensor data maps may include receiving sensor data captured by each vehicle sensor and generating each vehicle sensor data map based on the sensor data captured by the corresponding vehicle sensor.

[0085] In the example method, the sensor distance relationship may be one of an n-nearest neighbor relationship, a fixed radius relationship, and a weighted distance relationship.

[0086] In the example method, the at least one calibration matrix may be based on a calibration of two vehicle sensors relative to each other.

[0087] In the example method, generating a fused automotive sensor data graph may include generating each fused edge between two sensor data nodes of the two plurality of sensor data nodes by determining a distance between the two sensor data nodes based on at least one calibration matrix.

[0088] In this example method, obtaining at least one calibration matrix may include obtaining a plurality of calibration matrices, each calibration matrix defining a transformation between a corresponding plurality of sensor data nodes of two corresponding vehicle sensor data graphs.

[0089] In the example method, generating a fused automotive sensor data graph may include generating each fused edge between two sensor data nodes of the plurality of sensor data nodes by determining a distance between the two sensor data nodes based on a corresponding calibration matrix of the plurality of calibration matrices.

[0090] In this example method, the fused distance relationship may be one of an n-nearest neighbor relationship, a fixed radius relationship, and a weighted distance relationship.

[0091] In an example, an automobile control unit includes at least one processing unit and a memory coupled to the at least one processing unit and configured to store machine-readable instructions. The machine-readable instructions cause the at least one processing unit to obtain a plurality of automobile sensor data graphs. Each automobile sensor data graph is based on sensor data captured by a corresponding automobile sensor in a plurality of automobile sensors. Each automobile sensor data graph includes a plurality of sensor data nodes and a plurality of sensor data edges. Each sensor data node includes sensor data captured by a corresponding automobile sensor. Each sensor data edge defines a sensor distance relationship between two sensor data nodes in the plurality of sensor data nodes. The machine-readable instructions also cause the at least one processing unit to obtain at least one calibration matrix. The at least one calibration matrix defines a transformation between corresponding multiple sensor data nodes in at least two automobile sensor data graphs. The machine-readable instructions also cause the at least one processing unit to generate a fused automobile sensor data graph based on the plurality of automobile sensor data graphs and the at least one calibration matrix. The fused automobile sensor data graph includes a plurality of fused sensor data nodes and a plurality of fused sensor data edges. The plurality of fused automobile sensor data nodes include all of the plurality of sensor data nodes. The plurality of fused automobile sensor data edges include all of the plurality of sensor data edges and a plurality of fused edges. Each fused edge defines a fused distance relationship between two sensor data nodes in two of the plurality of sensor data nodes. The machine-readable instructions also cause at least one processing unit to provide the fused vehicle sensor data graph to a vehicle perception function. The vehicle perception function is implemented by a graph neural network and is configured to generate perception data based on the fused vehicle sensor data graph.

[0092] In the example automotive control unit, the machine-readable instructions further cause the at least one processing unit to perform any one of the aforementioned example methods.

[0093] In an example, a vehicle includes a plurality of vehicle sensors and the aforementioned example vehicle control unit.

[0094] The foregoing description has been provided to illustrate automotive perception based on fused sensor maps. It should be understood that this description is by no means meant to limit the scope of the present disclosure to the precise embodiments discussed throughout the description. Instead, those skilled in the art will appreciate that the examples of the present disclosure may be combined, modified, or streamlined without departing from the scope of the present disclosure as defined by the following claims.

Claims

1. A method (100) for generating a fused automotive sensor data graph (500), the method comprising: A plurality of vehicle sensor data maps are obtained (130), each vehicle sensor data map (3001-3003) being based on sensor data (2101-2103) captured by a corresponding vehicle sensor (200, 2001-2003) of the plurality of vehicle sensors, wherein: Each automobile sensor data graph (3001-3003) includes a plurality of sensor data nodes and a plurality of sensor data edges. Each sensor data node includes sensor data captured by a corresponding vehicle sensor (200, 2001-2003), and Each sensor data edge defines a sensor distance relationship between two sensor data nodes among the plurality of sensor data nodes; Obtaining (150) at least one calibration matrix (400, 4001-4003), the at least one calibration matrix (400, 4001-4003) defining a transformation between corresponding plurality of sensor data nodes of at least two vehicle sensor data graphs (3001-3003); The fused vehicle sensor data map (500) is generated (160) based on the plurality of vehicle sensor data maps and the at least one calibration matrix (400, 4001-4003), wherein: The fused automobile sensor data graph (500) comprises a plurality of fused sensor data nodes and a plurality of fused sensor data edges, The plurality of fused automobile sensor data nodes include all the plurality of sensor data nodes, and The plurality of fused automotive sensor data edges include all of the plurality of sensor data edges and the plurality of fused edges, and Each fused edge defines a fused distance relationship between two sensor data nodes of the two plurality of sensor data nodes; and The fused vehicle sensor data graph (500) is provided (170) to a vehicle perception function (610), which is implemented by a graph neural network and is configured to generate perception data (610O) based on the fused vehicle sensor data graph (500).

2. The method (100) of claim 1, wherein the perception data (610O) is configured to be processed by one or more advanced driver assistance systems (6201-6203).

3. The method (100) according to any one of the preceding claims, further comprising: Each vehicle sensor data map (3001-3003) is translated (140) from the sensor coordinate system of the corresponding vehicle sensor (200, 2001-2003) to the vehicle coordinate system.

4. The method (100) according to any one of the preceding claims, further comprising: Each vehicle sensor data map (3001-3003) is generated (110) using a feature extractor configured to extract one or more features of the corresponding vehicle sensor (200, 2001-2003).

5. The method (100) according to any one of the preceding claims, further comprising: The plurality of nodes and the plurality of edges of each automotive sensor data graph (3001-3003) are reduced (120) using one of an autoencoder, principal component analysis, clustering, or distance-based grouping.

6. The method (100) of any one of the preceding claims, wherein obtaining the plurality of vehicle sensor data maps comprises: receiving (131) said sensor data (2101-2103) captured by each vehicle sensor (200, 2001-2003); and Each vehicle sensor data map (3001-3003) is generated (132) based on the sensor data (2101-2103) captured by the corresponding vehicle sensor (200, 2001-2003).

7. The method (100) of any one of the preceding claims, wherein the sensor distance relationship is one of an n-nearest neighbor relationship, a fixed radius relationship, and a weighted distance relationship.

8. The method (100) according to any of the preceding claims, wherein the at least one calibration matrix (400, 4001-4003) is based on a calibration of two vehicle sensors (200, 2001-2003) relative to each other.

9. The method (100) of any one of the preceding claims, wherein generating (160) the fused automotive sensor data map comprises: Each fused edge between two sensor data nodes of the two plurality of sensor data nodes is generated (161) by determining a distance between the two sensor data nodes based on the at least one calibration matrix.

10. The method (100) according to any one of the preceding claims, wherein obtaining (150) the at least one calibration matrix (400, 4001-4003) comprises: A plurality of calibration matrices are obtained (151), each calibration matrix (400, 4001-4003) defining a transformation between a corresponding plurality of sensor data nodes of two corresponding vehicle sensor data graphs (3001-3003).

11. The method (100) of claim 10, wherein generating (160) the fused automotive sensor data graph comprises: Each fused edge between the two sensor data nodes in the two pluralities of sensor data nodes is generated (161) by determining a distance between the two sensor data nodes based on a corresponding calibration matrix (400, 4001-4003) in the plurality of calibration matrices.

12. The method (100) according to any of the preceding claims, wherein the fused distance relation is one of an n-nearest neighbor relation, a fixed radius relation, and a weighted distance relation.

13. An automobile control unit (800), comprising: at least one processing unit (810); as well as a memory (830, 850, 860) coupled to the at least one processing unit (810) and configured to store machine-readable instructions, wherein the machine-readable instructions cause the at least one processing unit to: A plurality of vehicle sensor data graphs are obtained, each vehicle sensor data graph (3001-3003) being based on sensor data (2101-2103) captured by a corresponding vehicle sensor (200, 2001-2003) of the plurality of vehicle sensors, wherein: Each automobile sensor data graph (3001-3003) includes a plurality of sensor data nodes and a plurality of sensor data edges. Each sensor data node includes the corresponding vehicle sensor (200, 2001-2003) captured sensor data, and Each sensor data edge defines a sensor distance relationship between two sensor data nodes among the plurality of sensor data nodes; obtaining at least one calibration matrix (400, 4001-4003), the at least one calibration matrix (400, 4001-4003) defining a transformation between corresponding plurality of sensor data nodes of at least two vehicle sensor data graphs (3001-3003); Generating a fused vehicle sensor data map (500) based on the plurality of vehicle sensor data maps and the at least one calibration matrix (400, 4001-4003), wherein: The fused automobile sensor data graph (500) comprises a plurality of fused sensor data nodes and a plurality of fused sensor data edges, The plurality of fused automobile sensor data nodes include all the plurality of sensor data nodes, and The plurality of fused automotive sensor data edges include all of the plurality of sensor data edges and the plurality of fused edges, and Each fused edge defines a fused distance relationship between two sensor data nodes of the two plurality of sensor data nodes; and The fused automotive sensor data graph (500) is provided to an automotive perception function (610), which is implemented by a graph neural network and is configured to generate perception data (610O) based on the fused automotive sensor data graph (500).

14. The automotive control unit according to claim 13, wherein the machine-readable instructions further cause the at least one processing unit (810) to perform the method according to any one of claims 2 to 12.

15. A vehicle comprising: a plurality of vehicle sensors (200), and An automotive control unit (800) according to any one of claims 13 and 14.

Images