Data processing method and system for iterative sensor fusion, computer program product and computer readable medium for implementing method

Abstract

The invention relates to a data processing method for iterative sensor fusion by a computer system, the method comprising the steps of:-receiving sensor data from at least a first data source type and a second data source type, the first data source type being adapted to provide sensor data that exclusively determines an entity, and the second data source type being adapted to provide sensor data that exclusively determines an entity; the second data source type is adapted to provide sensor data of the tracked entity,-generating, by the data adaptation module, a data source identifier for each sensor data, where the data source identifier comprises a data source type identifier corresponding to the data source type of the received sensor data, where the data source type identifier comprises a data source type identifier corresponding to the data source type of the received sensor data; the first data source type identifier corresponds to a first data source type and the second data source type identifier corresponds to a second data source type,-in a sampling step, a data set to be associated comprising the received latest sensor data as an element is compiled, and-a data association step is iteratively performed on the data set. The invention also relates to a computer system, a computer program product and a computer readable medium for implementing the method.

Description

Technical Field

[0001] The present invention relates to a data processing method and system for iterative sensor fusion, and also relates to a computer program product and a computer readable medium for implementing the method.

[0002] Vehicle-to-everything (V2X) technology allows entities involved in ground transportation to share and receive various types of information in a distributed manner, preferably in real-time transmission of standardized message formats via dedicated radio channels. This information may include, for example, the type of the sender device (the device of the sharing entity), absolute position, dimension, direction, speed and other main attributes. V2X messages may also contain confidence values ​​for certain shared information. V2X messages are digitally signed, and the signer also sends a certificate proving its authentication. Next-generation systems can also send information about external objects and entities, such as unconnected vehicles, pedestrians, animals, etc., by broadcasting their onboard sensor outputs. Each perceived entity is represented by similar data listed previously. The main goal of this technology is to improve traffic safety. Recently, the penetration of V2X-enabled devices has increased dynamically, and the speed of the technology's spread is expected to increase further. The ultimate goal is to integrate every relevant participant, including pedestrians, cyclists or e-scooter users.

[0003] Two main directions have begun to develop, greatly increasing the amount of information shared within the V2X ecosystem. First, the percentage of participants supporting V2X has been increasing over the past few years as car manufacturers have begun to make V2X an integral part of their latest models. However, as most traffic participants are equipped with V2X technology, there is often a growing need to integrate non-V2X entities, especially in areas where traffic is statistically more dangerous than average.

[0004] To achieve this goal, more and more roadside units (RSUs) have begun to be equipped with smart sensors, which usually also perform object detection and tracking, and can be used as inputs to dedicated sensor sharing messages (e.g., collective sensing messages in Europe and sensor data sharing messages in the United States). In addition, since most vehicles are equipped with multiple sensors for various safety functions, their detection results can also be shared within these standardized messages, providing a wealth of information to the receiving V2X participants.

[0005] More and more information needs to be processed on the receiver side, and this is where sensor fusion plays a key role. Sensor fusion is responsible for maintaining a local dynamic map, i.e., keeping track of entities (creating and deleting entities), correlating measurements with each other, and estimating entity states based on measurements belonging to each entity. In a V2X environment, this problem may grow to keeping track of hundreds of entities (vehicles, pedestrians, cyclists), which requires efficient processing methods. Background Art

[0006] The field of sensor fusion technology has a rich literature resource that solves various emerging challenging problems. In order to improve the safe operation of the vehicle, sensor fusion is often applied in the field of autonomous or self-driving vehicle technology to combine the measurements of different sensors. An example is shown in US2019 / 0351899 A1.

[0007] Most state-of-the-art data association methods perform the task on a probabilistic basis, i.e., they compute probability values ​​to describe the likelihood that a measurement belongs to a specific entity and aim to find an optimal global solution over the entire set of measurements and entities. In complex cases, there are multiple possibilities and, therefore, the required computations can explode.

[0008] Several methods and schemes for data association, information fusion, track initialization and maintenance are known in the art. Among the simpler information fusion algorithms, the nearest neighbor or strongest neighbor methods have gained a high popularity. They select a few measurements and continue to use them for calculations, but may lose useful information in the future. In the general case, when there are few or no assumptions about the characteristics of the input data, there are several methods that outperform these methods in terms of stability and accuracy.

[0009] In contrast to proximity-based algorithms, probabilistic methods assign weights to measurements based on the likelihood that the measurement belongs to a given track. Those with low likelihood are used as track initializers, while those that reach sufficiently high over time are confirmed. These methods are called probabilistic data association (PDA) methods. The results of these methods are used as weighted inputs to object tracking algorithms, such as a Kalman filter (KF), an extended Kalman filter (EKF), or an unscented Kalman filter (UKF), which will provide an estimated state and the corresponding covariance. This approach is described by Y. Bar-Shalom et al. in "The probabilistic data association filter", IEEE Control Systems Magazine, Vol. 29, No. 6, pp. 82-100 (2009).

[0010] These composite methods are called probabilistic data association filters (PDAF) and have several variants based on initial assumptions about the detection characteristics and the number of objects to be tracked, see TE Fortmann et al., "Multi-target tracking using joint probabilistic data association", 19th IEEE Conference on Decision and Control, Including Symposium on Adaptive Processes, Albuquerque, New Mexico, USA (1980). One of the composite methods is Joint PDAF for Multi-Object Tracking (JPDAF), where it is assumed that at most one measurement may come from a true object (i.e., measurements are otherwise false measurements or clutter) and that each measurement may belong to several tracks, so possible associations are enumerated and weights are calculated for each track.

[0011] Another approach is the family of multi-detection PDAF (MD-PDAF), see for example BK Habtemariam et al., "Multiple Detection Probabilistic Data Association filter for multistatic target tracking", 14th International Conference on Information Fusion, Chicago, IL, pp. 1-6 (2011), and B. Habtemarian et al., "A Multiple-Detection Joint Probabilistic Data Association Filter", IEEE Journal on Selected Topics in Signal Processing, Vol. 7, No. 3, pp. 461-471 (2013). These methods allow multiple measurements from objects within range of a single sensor. The advantage over the basic approach is that in each possible association, the probability of being associated with those measurements that should be from the target is higher (more realistic). The disadvantage is that the number of possible cases increases.

[0012] An alternative to the above approach is known as probability hypothesis density (PHD) filtering and multiple hypothesis tracking (MHT) and was disclosed by Kusha Panta et al. in "Probability hypothesis density filter versus multiple hypothesis tracking", Proceedings of the International Society for Optical Engineering 5429, Signal Processing, Sensor Fusion and Target Recognition XIII (2004). The probability hypothesis density filter is a practical alternative to the optimal Bayesian multi-target filter based on finite set statistics. It only propagates the first order moments, rather than the full multi-target posteriori. Recently, a sequential Monte Carlo (SMC) implementation of the PHD filter has been used for multi-target filtering with good results.

[0013] Another group of methods called multiple hypothesis tracking (MHT) is disclosed by Samuel S. Blackman in "Multiple hypothesis tracking for multiple target tracking" in IEEE Journal of Aerospace and Electronic Systems 19.1, pp. 807-812 (2004). A track hypothesis tree is constructed whose branches represent possible data association results (track hypotheses). The probability of the track hypothesis is calculated by evaluating the quality of the data association results that the branch has. In order to make this method tractable, the hypothesis tree needs to be pruned, that is, some heuristic method needs to be used to remove branches with lower priority.

[0014] US2021 / 0101612 A1 discloses a system for providing local dynamic map (LDM) data, wherein various information bits, including external information received via a wireless communication channel, are used for sensor fusion. The process according to US2021 / 0101612 A1 includes receiving data from vehicles, mobile devices, and other data sources (such as RSUs), and determining whether any information should be integrated into the LDM data model based on the received data. The process may also include determining a trust metric for the received data, and the data is only used when its trust metric is above a trust metric threshold.

[0015] US2022 / 0159425 A1 discloses methods and apparatus for determining and receiving V2X messages. These messages contain a series of information, and part of this information may be related to sensor data. Quality indicators and / or accuracy indicators may also be part of the V2X messages.

[0016] US2022 / 0272626 A1 discloses a method, system, and device for wireless communication supporting energy-saving iterative sensor fusion. Sensing data can be converted from the vehicle to the RSU or from the RSU to the vehicle.

[0017] US2020 / 0228946 A1 discloses an apparatus and method for V2X communication, which has various data elements (DE) and / or data frames (DF) summarized in Tables 1-3 of the application. Sensor ID and object ID are used in the communication, wherein the object ID is a unique identifier assigned to the tracked object and the object ID is maintained as long as the object is tracked.

[0018] In view of the known methods, there is a need for a data processing method that reduces the computational requirements for correlating various sensor data from different data sources, so that it allows real-time data association and sensor fusion. Summary of the invention

[0019] The main object of the present invention is to provide a data processing method for iterative sensor fusion which is free of the disadvantages of the prior art methods to the greatest extent possible.

[0020] As mentioned above, most known methods require high computing power, especially if the information is received from multiple data sources, ie from various types of sensors through different communication channels.

[0021] Another object of the present invention is to provide a data processing method and system that allows data association of sensor data from various different sources, wherein the data association is performed in real time.

[0022] Furthermore, the object of the present invention is to provide a non-transitory computer program product for implementing the steps of the method according to the present invention on one or more computers, and a non-transitory computer-readable medium comprising instructions for executing the steps of the method on one or more computers.

[0023] The objects of the invention are achieved by a method according to claim 1. The objects of the invention are also achieved by a system according to claim 12, a non-transitory computer program product according to claim 17 and a non-transitory computer readable medium according to claim 18. Preferred embodiments of the invention are defined in the dependent claims.

[0024] An advantage of the method according to the invention is that it reduces computational requirements by reducing the number of possible associations required to assign each sensor data to an entity. It has been recognized that certain types of data sources, such as V2X messages containing sensor data, belong explicitly and exclusively to a single entity, and that different information shared at different moments of the same type of data source should belong to different entities. Thus, for example, information received from different V2X entities does not need to be matched to each other, since it is known a priori that these information belong to different entities, and computational power can be saved by not matching these information to each other.

[0025] Another advantage of the method according to the invention is that if a piece of sensor information has been assigned to an entity, this information does not need to be matched with any other data or entity, which also reduces the total number of possible combinations and thus reduces computing power.

[0026] It is also recognized that the characteristics of certain types of data sources (such as smart sensors, i.e. sensors capable of tracking entities) can also be used to reduce the computational requirements of the method according to the invention. In certain embodiments of the method, the results of previous data associations are saved and can be reused in further iteration cycles, because if sensor data from such a data source has already been matched with an entity, then this association is likely to still be valid, so it is sufficient to check the validity of the previous association and only associate any additional sensor data, which further reduces the computational requirements of the method, because less information needs to be matched with each other.

[0027] In certain embodiments of the method according to the invention, assigning identifiers based on the origin of the information may also be used to reduce the number of possible associations. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Preferred embodiments of the present invention are described below by way of example with reference to the accompanying drawings, in which

[0029] Figure 1 is a block diagram of a preferred embodiment of a system according to the present invention,

[0030] Figure 2 shows a preferred way of generating an identifier according to an embodiment of the method of the present invention, Figure 3-9 An exemplary implementation of an embodiment of the method according to the invention is shown, and

[0031] Fig.10 An exemplary use case of the method according to the invention is shown. DETAILED DESCRIPTION

[0032] The present invention relates to a data processing method for iterative sensor fusion, the method being implemented by a computer system. The data processing method according to the present invention comprises the steps of receiving sensor data from at least a first data source type and a second data source type, wherein the first data source type is suitable for providing sensor data exclusively determining an entity, and the second data source type is suitable for providing sensor data of a tracked entity. Preferably, the first data source type is a V2X message containing sensor data, and the second data source type is a smart sensor.

[0033] The data processing method according to the present invention also includes a step of generating a data source identifier for each sensor data by a data adaptation module, wherein the data source identifier includes a data source type identifier corresponding to a data source type from which the sensor data is received, wherein a first data source type identifier corresponds to a first data source type, and a second data source type identifier corresponds to a second data source type.

[0034] In certain preferred embodiments of the present invention, sensor data from a third data source type may also be received, wherein the third data source type is preferably a conventional sensor that is not suitable for tracking an entity; therefore, it can only provide individual measurement data that needs to be associated with the entity.

[0035] The data processing method according to the present invention further comprises the step of compiling a data set to be associated including the latest received sensor data as an element in the sampling step. Preferably, the sampling step is implemented by a sampling module (see Figure 1 ). In the sampling step, the latest and most recent sensor data is collected from any data source. As a result, some intermittent data from sensors with more frequent sampling frequencies can be omitted. This also reduces the computational requirements of the method and also ensures that only the most relevant and latest sensor data is processed.

[0036] The data processing method according to the present invention further comprises a data association step, which comprises the step of generating an entity selection set to collect all available sensor data about the entity and generating a dedicated sensor selection set for each entity from which sensor data is received.

[0037] The process of generating an entity selection set includes the steps of selecting elements of the dataset to be associated having a first data source type identifier and moving them to the entity selection set. The process of generating an entity selection set also includes the steps of matching the elements of the dataset having the first data source type identifier with each element of the dataset having the second data source type identifier, selecting each element of the dataset having the second data source type identifier that matches the same entity as the elements of the dataset having the first data source type identifier, and moving each selected element of the dataset to the entity selection set. Through this process, each sensor data in the dataset to be associated belonging to the same (i.e., first) entity is collected.

[0038] The data processing method according to the present invention further comprises the step of iteratively performing the data association step until the data set no longer has elements having the first data source type identifier, i.e., creating a different entity selection set for each individual entity. The method according to the present invention utilizes the characteristics of the first data source type, i.e., the sensor data provided by the data source type exclusively determine an entity, i.e., the sensor data received from different data sources of the first data source type belong to different entities. Therefore, they do not need to be matched with each other, but these sensor data can be included in different entity selection sets.

[0039] The data processing method according to the present invention further comprises a step of matching the remaining elements of the data set having the second data source type identifier with each other and forming one or more groups of elements of the data set, wherein each group belongs to the same entity. This step ensures that even if no sensor data about the entity is received from the first data source type, one or more sensor data can be related to the entity, and these sensor data need to be associated with each other.

[0040] Preferably, the method according to the invention further comprises the step of storing the entity selection set in a mapping cache. Even more preferably, each entity selection set is stored in a mapping cache. Furthermore, one or more groups of elements of a data set are preferably also stored in a mapping cache, in particular if the group contains at least two elements, i.e. an association is established between at least two elements of the data set to be associated. The information stored in the mapping cache can be used for further processing steps, for example, when generating another data set to be associated. Preferred embodiments of the further processing steps are described in more detail below.

[0041] Preferably, the method according to the present invention further comprises the step of compiling another data set to be associated including the latest received sensor data as an element. When compiling the other data set to be associated, the mapping cache may be queried to check whether there is a previous association between the elements of the other data set to be associated. This is preferably performed by the following steps:

[0042] - select the elements of the other dataset to be associated with that have the first data source type identifier and move them to the entity selection set,

[0043] - checking whether the mapping cache includes an element of the dataset to be associated with the first data source type identifier, and if the mapping cache includes an element of the dataset to be associated with the first data source type identifier, adding the corresponding element of the dataset with the second data source type identifier to the corresponding entity selection set,

[0044] - performing a data association step on the remaining elements of the other dataset to be associated, and

[0045] - Update mapping cache with entity selection set.

[0046] The above steps ensure that the mapping cache is always up-to-date and that minimal association steps are performed, thus reducing the computational requirements of the method.

[0047] In a preferred embodiment of the method according to the invention, the method further comprises a sensor fusion step of combining sensor data of the selected set of entities by a sensor fusion module.

[0048] The sensor data provided by any data source is preferably location data of an entity, dynamic data of an entity, motion data of an entity or dimensional data or entity type. Even more preferably, the sensor data may include confidence intervals, for example, the location data of an entity may include uncertainty of said location data, to further assist the data association process.

[0049] The method according to the invention preferably further comprises the step of storing the entity selection set in an entity database.The entity database is preferably accessible by other modules or applications that need to use data about the entities.

[0050] Preferably, the method according to the invention is adapted to be able to receive sensor data from a third data source type, for example a conventional sensor that is not capable of tracking an entity, so that each sensor data provided by the third data source type needs to be associated each time it occurs. This preferred embodiment of the method comprises the step of generating a data source identifier, the data source identifier comprising the third data source type identifier for each sensor data received from the third data source type, and including the sensor data having the third data source type identifier in the data set to be associated. The method preferably also comprises the step of generating an extended entity selection set, and in the data association step, preferably matching said elements of the data set having the first data source type identifier with each element of the data set having the third data source type identifier, selecting each element of the data set having the third data source type identifier that matches the same entity as said element of the data set having the first data source type identifier, and moving each selected element of the data set to the extended entity selection set.

[0051] Preferably, sensor data with the third data source type identifier are not stored in the mapping cache, since there is no way to ensure that a new measurement provided by the same sensor corresponds to the same entity as a previous measurement. Therefore, even if an association is established between sensor data with the third data source type identifier and sensor data with the first or second data source type identifier, such information cannot be used in other data association steps.

[0052] The method according to the present invention may further comprise the step of storing the entity selection set and the extended entity selection set in an entity database.

[0053] Preferably, the method according to the invention further comprises a step of creating unified sensor data by the data adaptation module 24. This step not only facilitates data association, but also facilitates possible sensor fusion, since sensor data in a unified format can be more easily compared and / or fused.

[0054] The method according to the invention preferably also comprises the step of determining a trust score for each data source from which the sensor data is received. Fig.10 Have a discussion.

[0055] Figure 1 A preferred embodiment of a data processing system according to the invention is shown, which system is also suitable for executing the steps of the data processing method according to the invention.

[0056] The system includes a data association module 30, which is adapted to perform data association of sensor data received from different data sources. Possible data sources may include a first data source type, such as a V2X entity 14 that may share a V2X message 20 (e.g., a CAM or BSM message) that includes sensor data, a second data source type, such as a smart sensor 12 that may preferably provide tracked sensor data 18 as sensor data, and a third data source type, such as a traditional sensor 10 that may provide traditional sensor data 16 as sensor data. In certain embodiments of the system, the number of different data source types may vary. Furthermore, the system is preferably adapted to receive sensor data from multiple instances of the same data source type.

[0057] The sensor data from different data sources are fed into corresponding adapters 22, wherein the adapters 22 may be part of the data adaptation module 24. The adapters 22 are preferably adapted to provide a unified data representation of the different data sources. This has the advantage that other modules of the system can more easily process the unified data.

[0058] The data association module 30 receives entity data elements 26 from the adapter 22 or the data adaptation module 24, wherein the entity data elements 26 need to be associated by the data association module 30, that is, entity data elements 26 belonging to the same entity need to be found. Preferably, the entity data elements 26 are compiled into a data set 70 to be associated (see Figure 3 The data set 70 to be associated may be compiled by the data association module 30 or the sampler module 28. The sampler module 28 preferably creates an ordered data set 70.

[0059] Preferably, the data adapter module 24 preferably assigns a data source identifier to each entity data element 26, and the sampler module 28 also assigns them fusion data identifiers to assist in the data association and / or sensor fusion process. Preferably, the sampler module 28 creates an ordered data set 70 based on the data source identifiers. More details of the identifiers will be provided in conjunction with Figure 2 Have a discussion.

[0060] The data association module 30 preferably performs the data association step according to the data association method of the present invention. The system preferably comprises a mapping cache 34 adapted to store sensor data associated with entities. The mapping cache 34 is preferably in bidirectional communication connection with the data association module 30 to store already associated data arranged by the entities and to let the data association module 30 check whether there is already an association between the sensor data, i.e. between the elements of the data set 70 to be associated. If an association already exists, it can speed up the association process and reduce the computational requirements by reducing the number of possible associations. Preferably, the data association module 30 generates.

[0061] The system preferably includes a matching module 42 in bidirectional communication connection with the data association module 30. The matching module 42 is preferably adapted to receive a pair of sensor data to be matched and return a matching probability 40 to the data association module 30. Based on the matching probability, the data association module 30 can determine whether the pair of sensor data belongs to the same entity. If the pair of sensor data may belong to the same entity, it can be saved in the entity database 50 or in the mapping cache 34, and further, the pair of sensor data can be forwarded to a data fusion module 38 adapted to perform sensor fusion. If the pair of sensor data can already be found in the mapping cache 34, the mapping cache 34 can be updated. Even more preferably, the data fusion module 38 receives all relevant (associated) sensor data belonging to the same entity and performs sensor fusion on one entity at a time. The data fusion module 38 preferably creates fused data for the entity, which can be stored in the entity database 50.

[0062] Preferably, the data stored in the entity database 50 can be used by any application in the application layer 52. Possible applications in the application layer may be filtering or security applications. Fig.10 An example of this is described in more detail.

[0063] The entity database 50 preferably provides the matching module 42 with information about previous entity states 48 of entities, with the help of which a more accurate matching can be performed.

[0064] Preferably, the system comprises an iterative fusion module 44, which comprises the data association module 30, the data fusion module 38 and / or the matching module 42. The system according to the present invention preferably also comprises a sensor fusion module 46, which preferably comprises the iterative fusion module 44, the sampler module 28 and / or the mapping buffer 34.

[0065] According to a preferred example of the system of the present invention, the direction of the data flow is indicated by arrows. In other embodiments of the system, there may be further communication or data channels between the elements, modules of the system.

[0066] In order to assist the data association method according to the invention, each received sensor data, ie any measurement value belonging to an entity, is tagged by an identifier, which is a more complex structure than a unique integer value. Figure 2 A preferred structure of various identifiers used in different processing stages of some embodiments of the method according to the invention is shown.

[0067] When a new measurement (i.e., sensor data) is injected into the system according to the present invention, a data source identifier (DataSourceId) is generated and assigned to the measurement. The data source identifier preferably includes a numeric identifier number 60 (or tag), which preferably indicates the kind of entity (e.g., object or traffic information) described by the measurement. The data source identifier preferably also includes a data source type identifier 62, which is Figure 2 In the figure, it is represented by S. The data source type identifier 62 corresponds to the data source type from which the measurement value comes. As an example, preferably, two or three different data source types can be defined and used based on the characteristics of the sensor. Preferably, the following two data source types can be defined.

[0068] First data source type. A data source belonging to the first data source type can exclusively determine an entity. Thus, sensor data from different data sources of the first data source type describe different entities, whereas sensor data from the same data source of the first data source type is expected to describe properties of the same entity. The first data source type may also be referred to as an "exact" type, since such sensor data corresponds exactly to an entity. Figure 3-9 In the figure, the letter "E" represents sensor data with a first data source type identifier. An example of a first data source type is a V2X message containing sensor data. The V2X message is preferably a CAM or BSM message. Preferably, different adapters 22 can handle different message types. Therefore, when a measurement value (sensor data) is received via a V2X message, it will be assigned a first data source type identifier. The V2X message typically includes a unique identifier (digital identifier number 60), which can also be used to generate a data source identifier. The unique identifier can be a 48-bit identifier.

[0069] A special case of the first data source type may be an ego entity. An ego entity typically consists of local measurements, e.g. GNSS or more accurate positioning information and measurements retrieved from onboard electronics, e.g. in the case of an ego vehicle, the source of sensor data may be the vehicle's CAN network.

[0070] Second data source type. Data sources belonging to the second data source type are capable of tracking entities, i.e., making multiple measurements of the same entity. The second data source type may also be referred to as a "tracked" type, since such data sources can track entities. Figure 3-9 In FIG. 1 , the letter “T” represents sensor data with a second data source type identifier. Typically, the measurement values ​​provided by the smart sensor will be assigned a second data source type identifier.

[0071] Alternatively, V2X messages can also provide "tracking" type sensor data. For example, sensor data received via CPM "perception" messages (according to EU regional standards) and SDSM "sensor data sharing" messages (according to US standards). These V2X messages share the sensor data of the own sensors with other ITS participants. The content of these received V2X messages is assigned a second data source type identifier, just like the sensor data of the own sensors on the receiver side.

[0072] If the sensor data is received from any other data source, it may be assigned a third data source type identifier.

[0073] A third data source type. Sensor data received from a data source that cannot ensure that its data exclusively belongs to a certain entity and the data source cannot track a certain entity, a third data source type identifier will be assigned to such sensor data. As an example, sensor data received from a traditional sensor (not a smart sensor) can be assigned a third data source type identifier. The third data source type can also be called a "virtual" type; in Figure 3-9 In the mapping cache, the letter "D" represents sensor data with a third data source type identifier. Preferably, sensor data with a third data source type identifier is not stored in the mapping cache, because due to the lack of tracking, it cannot be ensured that consecutive measurements from the same sensor are related to the same entity. Therefore, any sensor data with a third data source type identifier needs to be matched with sensor data with a first data source type identifier and / or a second data source type identifier, and also needs to be matched with other sensor data with a third data source type identifier.

[0074] Preferably, the data source identifier is provided by the data adapter module 24 (see Figure 1 ) is assigned to the sensor data.

[0075] When compiling the data sets to be associated, preferably by the sampling module 28, the information contained in the data source identifier can be extended, preferably by adding the registration key (or feed identifier 66) of the adapter 22 that pushes the measurements (sensor data) into the sensor fusion module 46 of the system according to the present invention. In cases where it is known that multiple adapters 22 process information from the same entity, for example, the motion data and dimensional data of the same entity are processed by different adapters 22, the tracking identifier 64 can be added to the data source identifier. The same tracking identifier 64 will be assigned to the sensor data from the same entity, so the tracking identifier 64 can be used to assist and speed up the data association process.

[0076] In addition, an entity type flag 68 may also be added to the data source identifier. The entity type flag 68 may be used to determine the final data entry type in the entity database 50. As an example, different entity type flags 68 may provide categories such as object, intersection, road information, etc. The entity type flag 68 may also assist in the data association step, since sensor data with an entity type flag 68 of intersection type does not need to be matched with sensor data with an entity type flag 68 of road information type. Preferably, only sensor data with an entity type flag 68 of object type needs to be fused, and other categories, such as intersection or road information type, do not need to be included in the sensor fusion, so such sensor data may be moved directly to the entity database 50 after data association.

[0077] Such an extended data source identifier may be referred to as a fusion data identifier (FusionDataId), which contains further information related to purposes such as data fusion. Preferably, the fusion data identifier is generated by the sampling module 28 .

[0078] Preferably, the fused data identifier is also stored in the mapping cache 34 to make the next processing cycle of data association easier and more efficient.

[0079] After the data association step and / or the sensor fusion step, an entity identifier may be generated for each entity. When the sensor data related to the entity is stored in the entity database 50, the entity identifier may also be stored therewith.

[0080] Preferably, the entity identifier is converted from a data source identifier having a digital identifier number 60 (from a V2X message), or in other cases, the entity identifier is assigned by the iterative data association module 30. In the case where the entity identifier is generated by the iterative data association module 30, a modifier flag 69 may be set to indicate that the entity is "virtual", i.e., the entity has no sensor data with the first data source identifier. The entity identifier is preferably a unique identifier that can be used to distinguish entities stored in the entity database 50. Thus, the entity identifier can be used by any application stored in the application layer 52.

[0081] The data source type identifier 62, tracking identifier 64, and / or feed identifier 66 are typically used for data association and / or sensor fusion, so they do not need to be included in the entity identifier because applications are unlikely to make use of the information they carry. However, it is preferred to include the entity type flag 68 in the entity identifier because other applications can benefit from knowing the type of entity stored in the entity database 50. Figure 2 Constructing the identifier facilitates the data association and / or sensor fusion process to the greatest extent possible while using the least amount of storage capacity.

[0082] Figure 3-9 The steps of an exemplary embodiment of the method according to the invention are shown.

[0083] Figure 3 An example of a data set 70 to be correlated is shown, which is preferably generated by the sampling module 28. The data set 70 to be correlated preferably contains the latest sensor data received from one or more data sources. Figure 3 , two elements of the data set 70 have a first data source type identifier ("exact" type), three elements of the data set 70 have a second data source type identifier ("tracked" type), and three elements of the data set 70 have a third data source type identifier ("virtual" type). The elements of the data set 70 to be associated are represented by their respective fused data identifiers, where the letters represent the data source type identifier 62 (i.e., E-exact, T-tracked, D-virtual), and the numbers correspond to the tracking identifier 64. The feed identifier 66 is not included in the fused data identifier.

[0084] Preferably, the data set 70 is sorted according to the data source type identifier, i.e., the first element is the sensor data with a first data source type identifier, followed by elements with a second data source type identifier, and finally the data set 70 includes elements with a third data source type identifier. The sorting of the data set 70 is preferably performed by the sampler module 28. The data set 70 may be stored in a data cache.

[0085] Figure 4 The first sub-step of the data association step is shown. First, sensor data having a first data source type identifier is selected, e.g. Figure 4 Then, each element of the data set 70 having the second data source type identifier is checked to find sensor data describing the same entity (same object) as the selected element E1. Preferably, the selected element E1 is matched with each sensor data having the second data source type identifier (i.e., T1-T3). This matching is preferably performed by the matching module 42 (see Figure 1 ) is performed, preferably based on statistical methods. Figure 4In the example of, among the three elements with the second data source type identifier (T1-T3), only the element represented by T1 corresponds to the same entity, so it is selected and preferably stored in the mapping cache 34 together with the selected element E1. In the mapping cache 34, the entity identifier 72 can be assigned to the selected sensor data. It should be noted that more elements of the data set 70 with the second data source type identifier can correspond to the same entity, so each element of the data set 70 with the second data source type identifier needs to be matched with the selected element E1. By using the tracking identifier 64, the number of matches can be further reduced. According to Figure 4 In the example shown, the tracking identifier 64 is included in the fused data identifier, wherein the same tracking identifier 64 corresponds to the same entity, and therefore, elements E1 and T1 can be immediately selected as matching sensor data. In a similar case, only elements without the tracking identifier 64 need to be matched with elements with the first data source type identifier.

[0086] Figure 5 , wherein the elements of the data set 70 having the third data source type identifier (D1, D3, D4) are checked to find sensor data describing the same entity (same object) as the selected element E1. Preferably, this step is also performed by the matching module 42, wherein the matching module 42 preferably uses the same statistical method to select the elements of the data set 70 having the third data source type identifier, which elements correspond to the same entity as the selected element E1. It should be noted that even if the element represented by D1 has been selected as the element corresponding to the same entity as the selected element E1, it is preferably not stored in the mapping cache 34, because this type of association cannot be used in the next association cycle, so it does not need to be stored in the mapping cache 34, but the memory usage of the mapping cache 34 can be saved to store data of other entities.

[0087] according to Figure 6 , all sensor data corresponding to the same entity I1 are preferably stored in the entity database 50, ie the entity database preferably contains sensor data with a third data source type identifier.

[0088] Since the elements of the data set 70 to be associated have been moved to the entity selection set, the size of the data set 70 is reduced (i.e., the data set 70 contains fewer elements), and therefore the number of possible combinations of the elements of the data set 70 is also reduced, which improves the performance of the method according to the invention by reducing the computational requirements. This reduction also makes it possible to perform the method according to the invention on an on-board unit of a vehicle, which usually has limited computational power.

[0089] Figure 7represents the step of selecting elements of the data set 70 having a first data source type identifier (E2) and matching them with the remaining elements having a second data source type identifier. Figure 7 In the example shown, only the element represented by T2 matches the selected element E2, that is, these two elements of the data set 70 belong to the same entity I2. As described above, both elements represented by E2 and T2 are stored in the mapping cache 34. Figure 7 , no element of the data set 70 belongs to the same entity I2, so only the elements represented by E2 and T2 are stored in the entity database 50.

[0090] exist Figure 7 After the steps shown, there are no more elements in the data set with the first data source type identifier; therefore, Figure 8 It is shown that the remaining elements of the data set 70 match each other, that is, the remaining elements are grouped together, wherein each group corresponds to a different entity. Only groups with elements (which have the second data source type identifier) ​​are stored in the mapping cache 34, whereas each group is stored in the entity database 50. As an example, Fig. 9 It is shown that a group comprising only elements having a third data source type identifier is also stored in the entity database 50 .

[0091] exist Fig. 9 After the steps shown, there are no more elements in the data set 70, so another data set can be compiled from the latest sensor data. In the case where the associated entity already exists in the mapping cache 34, the elements of the other data set can be looked up in the mapping cache 34, and if a previous association already exists, no matching needs to be performed by the matching module, but the mapping cache 34 and the entity database 50 can be updated with the new sensor data. For elements of the other data set that are not present in the mapping cache 34, the above-mentioned association steps need to be performed in a similar manner.

[0092] Preferably, the matching of two elements of the dataset 70 to be associated and / or another dataset is performed by statistical checking. For example, the sensor data may describe the movement of an entity, for example in the form of a state vector, in which case the sensor data may include the position, velocity and / or orientation of the entity. The sensor data may also include an estimate and its accuracy.

[0093] When a possible piece of sensor data is found that may be added to the current set of measurements belonging to the same entity, preferably a combined value is calculated, which is a weighted average of the sensor data of the current set of measurements and the possible piece of sensor data, and the accuracy is obtained by covariance intersection (for example, by two-dimensional covariance intersection for position and one-dimensional covariance intersection for speed and direction). Preferably, the values ​​of position, speed and direction before and after adding (merging) the new piece of data are checked against each other, and a similarity measure is derived from the ratio of the accuracy of the new piece of data and the combined data. If the similarity measure exceeds a threshold, the merged state with the new piece of data is accepted and iteration continues.

[0094] It should be noted that other motion parameters, such as acceleration, yaw rate, etc., may also be part of the state vector, however, these other motion parameters are typically not included in the decision making process.

[0095] Preferably, the elements of each entity selection set are fused together (preferably in a sensor fusion step) to provide aggregate information about the entity. Once the sensor data belonging to the same entity is updated, the sensor fusion step will be performed again. It is worth noting that sensor fusion algorithms have an inherent property that in some cases, incorrect measurements may be combined. This may occur when the difference between the motion parameters of two entities is below a threshold for a period of time, so these motion parameters are merged together as if they belong to the same entity. However, if there is a significant difference in the motion parameters of the two entities, the sensor fusion should react. Therefore, the data association needs to be constantly checked and reset if an incorrect fusion is detected. Preferably, during this check, the same method as in the sensor fusion step will be performed.

[0096] The result of the data association method according to the present invention can be utilized by various applications, for example, it can support the performance of sensor fusion as described above, as well as further filtering and / or safety applications.

[0097] Fig.10 An exemplary application is shown. Preferably, the method according to the invention further comprises the step of evaluating the data source providing the sensor data. Preferably, the step of evaluating the data source providing the sensor data is performed separately for each data source.

[0098] Preferably, an indicator for positive confidence and an indicator for negative confidence are applied, wherein the confidence value is determined based on verification of the received sensor data with sensor data from different reliable data sources. For example, the ego vehicle can verify the received data by comparing it with its own trusted measurements. Furthermore, the received sensor data can be cross-checked for inconsistencies and consistency.

[0099] If a data source is found to provide malicious or incorrect data, e.g., wrong location, the trust value (or trust score) of the data source needs to be reduced. Therefore, preferably, the new data source has an initial trust score, wherein the initial trust score is a predetermined value that is not an extreme value of the possible trust score. Preferably, the initial trust score is near the middle of the full range of possible trust scores. For example, if the trust score is between 0 and 100, it is preferred that the predetermined value of the trust score is around 50. It allows sensor data from new, previously unknown entities to be considered, because on the one hand, losing an important piece of information (sensor data) may lead to an accident, and on the other hand, fully trusting an unknown data source that may share incorrect data may also be dangerous. Preferably, if the information provided by an object or entity (e.g., sensor data) is found to contain anomalies, its trust score can be reduced (negative trust), or if other (preferably reliable) entities are found to confirm or re-assure the accuracy of the provided information, the trust score of such an object or entity can be increased (positive trust).

[0100] The main benefits of determining a trust score are as follows. Data sources that misbehave can be detected, preferably through a set of standardized and unique proprietary checks. Evidence of misbehavior or a probability measure can also be provided to security applications that can use or ignore such information. Data sources that have been flagged (due to providing incorrect sensor data) can be filtered, so the amount of false positive triggers for security applications can be greatly reduced. Misbehavior detection can also identify ghost vehicles and other cybersecurity threats.

[0101] The trust score can be used by various applications as a filtering or triggering criterion, i.e., some information from data sources with a trust score below a predetermined threshold can be filtered out / omitted. The trust score (preferably a single value) can be added as metadata for each object or entity stored in the vehicle's environmental perception database or local dynamic map, which is a fused single database representing the vehicle's perception of objects / entities around it.

[0102] Preferably, the vehicle continuously receives incoming V2X information and updates its own database (eg, entity database 50 ), as described in conjunction with the data association method described above.

[0103] In a preferred embodiment, the trust score is in the range of -100 to +100, where the initial trust score is zero. The initial trust score can increase to positive values ​​(e.g., up to +100) or decrease to negative values ​​(e.g., until -100). The trust score preferably represents the trust that the ego vehicle has in a particular abstract entity, and by trust it is intended to represent the probability that the information received from that entity is correct. Preferably, the information represented by the trust score is not a measured confidence (e.g., position confidence or location confidence), as confidence has significantly different goals and uses. As an example, the position confidence can also be fused according to the fusion rule, and it can be used as part of the fused abstract object position confidence.

[0104] Preferably, the trust score corresponds to an abstract entity (object), however, it is also preferred to assign a source trust score to each data source. The data source may be an Intelligent Transportation System (ITS) station or a single sensor. Preferably, an initial source trust score may also be defined and assigned to each new data source that has not been evaluated before. Preferably, the source trust score has a similar range as the trust score, and the initial source trust score is preferably zero. Events or information indicating that a particular data source is not trustworthy will result in a reduction in the source trust score. Such events may be the identification of improper behavior of an ITS station or inconsistent V2X data provided by said ITS station. An example of the latter may be that the dynamic data sent is not consistent with the measurements of the onboard sensors of the ego vehicle (e.g., the remote vehicle and the ego vehicle are driving very close, the sensors report the same speed, the ego vehicle is driving at 100 km / h, while the remote vehicle reports a speed of 50 km / h). Another example is an inconsistency with other ITS stations, when one station reports the confirmed vacancy of an area using the free space description of the collective sensing service, and another vehicle sends information about the presence of a vehicle in the area. In this case, the source trust scores of both ITS stations will be reduced. An increase in the source trust score can be achieved by maintaining consistency over time or by confirming the sent V2X data by the ego vehicle's own sensors.

[0105] It would be preferable if a single software module or algorithm was able to vary both the trust score and the source trust score by alternative / parallel methods. To calculate the final trust score for an entity, the source trust scores of the data sources providing information about the entity may be used. Preferably, the final trust score for the entity may be calculated by using the source trust scores as weights when combining the various pieces of information about the entity. For example, an indication of misconduct about a station from an ITS station with a negative source trust score would be considered to have a lower weight. Another example is that trust scores increase over time when multiple ITS stations confirm the presence of a vehicle, so a remote vehicle being tracked by three other vehicles using a collective sensing service should increase the vehicle's trust score compared to a situation where only two vehicles are continuously reporting.

[0106] ADAS functions and applications can also take the trust score into account at any time to influence the weight of information while fusing it with other data from different data sources (e.g., on-board sensors) and verify the triggering of actions based on this information (e.g., warning the driver or making steering changes). If an entity has a higher trust score, events based on calculations involving such objects, such as proximity applications (e.g., forward collision warnings), should be triggered in more cases.

[0107] according to Fig.10 In the example of FIG. 8 , two vehicles that can share their information with each other and thus verify the accuracy of the received information through their own measurements can be assigned a higher trust score 80 if the shared information is successfully verified. As shown in the exemplary scenario, the two vehicles are also within the line of sight of the camera / traffic light, which can verify the information shared by the vehicles. If improper behavior of the ghost vehicle can be detected, a lower trust score 84 is assigned to the corresponding vehicle. For vehicles communicating via secure CAM messages, a neutral trust score 82 can be assigned; if more information about the vehicle and the sensor data it provides becomes available, the neutral trust score 82 can be increased or decreased later.

[0108] Furthermore, the invention relates to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to perform an embodiment of the method according to the invention.

[0109] The computer program product may be executed by one or more computers.

[0110] The invention also relates to a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform an embodiment of the method according to the invention.

[0111] The computer readable medium may be single or comprise multiple separate components.

[0112] Of course, the present invention is not limited to the preferred embodiments described in detail above, but other variations, modifications and developments may be performed within the scope of protection determined by the claims. In addition, all embodiments defined by any arbitrary combination of dependent claims belong to the present invention.

[0113] Reference Symbols List

[0114] 10 Traditional Sensors

[0115] 12 Smart Sensors

[0116] 14V2X Entity

[0117] 16 Traditional sensor data

[0118] 18 Tracking sensor data

[0119] 20V2X Message

[0120] 22 Adapter

[0121] 24Data adapter module

[0122] 26 Entity Data Elements

[0123] 28 Sampler Modules

[0124] 30Data Association Module

[0125] 34 Mapping Cache

[0126] 38 Data Fusion Module

[0127] 40 Matching probability

[0128] 42 matching modules

[0129] 44 Iteration Fusion Module

[0130] 46 sensor fusion module

[0131] 48 Previous Entity State

[0132] 50 Entity Database

[0133] 52 Application Layer

[0134] 60 digital identifier number

[0135] 62 Data source type identifier

[0136] 64 Tracking Identifier

[0137] 66 Feed Identifier

[0138] 68 Entity type flag

[0139] 69 modifier flags

[0140] 70 Datasets

[0141] 72Entity identifier

[0142] 80 High Trust Score

[0143] 82 Neutral Trust Score

[0144] 84 Low Trust Score

Claims

1. A data processing method for iterative sensor fusion by a computer system, the method comprising the following steps: - receiving sensor data from at least a first data source type and a second data source type, wherein the The first data source type is adapted to provide sensor data that exclusively identifies an entity, and the second The second data source type is suitable for providing sensor data of the tracked entity. - generating a data source identifier for each of the sensor data by a data adaptation module (24), wherein the data source identifier comprises a data source type identifier corresponding to a data source type receiving the sensor data, wherein a first data source type identifier corresponds to the first data source type and a second data source type identifier corresponds to the second data source type, - compiling, in a sampling step, a data set to be associated comprising as elements the latest sensor data received, - performing a data association step, including the step of generating an entity selection set by: o selecting elements of the data set (70) to be associated having the first data source type identifier and moving them to the entity selection set, o compare said element of the data set (70) having said first data source type identifier with Each element of the data set (70) having the second data source type identifier is Match, and o selecting the element corresponding to the data set (70) having the first data source type identifier each element of the data set (70) having the second data source type identifier that matches the same entity, and moving each selected element of the data set (70) Move to the entity selection set, - iteratively performing the data association step until the data set (70) no longer has any data with the elements of the first data source type identifier, and - matching the remaining elements of the data set (70) having the second data source type identifier with each other, And forming one or more groups of elements of the data set (70), wherein each group belongs to the same entity.

2. The method according to claim 1, wherein: The first data source type is a V2X message (20) containing sensor data, and the second data source type is a smart sensor (12).

3. The method according to claim 1 or claim 2, further comprising the step of storing the entity selection set in a mapping cache (34).

4. The method according to claim 3, further comprising the steps of: - compiling the latest sensor data received as an element of another data set to be associated (70), - selecting elements of another data set (70) to be associated having said first data source type identifier and moving them to said entity selection set, - checking whether the mapping cache includes an element of the dataset (70) to be associated with the first data source type identifier, and if the mapping cache (34) includes an element of the dataset (70) to be associated with the first data source type identifier, adding a corresponding element of the dataset (70) with the second data source type identifier to the corresponding entity selection set, - performing said data association step on the remaining elements of the further data set (70) to be associated, and - Updating the mapping cache (34) with the entity selection set.

5. The method according to any of the preceding claims, further comprising a sensor fusion step of combining the sensor data of the selected set of entities by a data fusion module (38).

6. A method according to any one of the preceding claims, wherein: Sensor data is the location data of an entity, the dynamic data of an entity, the dimensional data of an entity, or the type of an object.

7. The method according to any of the preceding claims, further comprising the step of storing the selected set of entities in an entity database (50).

8. The method according to any one of the preceding claims, characterized in that: - receiving sensor data from a third data source type, - generating, for each of the sensor data received from said third data source type, a data source identifier comprising a third data source type identifier, - including sensor data having said third data source type identifier in the data set (70) to be associated, - Generates an extended entity selection set, - in said data association step, matching said element of the data set (70) having said first data source type identifier with each element of the data set (70) having said third data source type identifier, and - Selecting each element of the data set (70) having the third data source type identifier that matches the same entity as the element of the data set (70) having the first data source type identifier, and moving each selected element of the data set (70) to the expanded entity selection set.

9. The method according to claim 8, further comprising the step of storing the entity selection set and the extended entity selection set in an entity database (50).

10. The method according to any of the preceding claims, further comprising the step of creating uniform sensor data by the data adaptation module (24).

11. A method according to any preceding claim, further comprising the step of determining a trust score for each data source receiving sensor data.

12. A computer system, adapted to execute the steps of the method according to any one of claims 1 to 11, the system comprising: - means for receiving sensor data from at least a first data source type and a second data source type, wherein the first data source type is adapted to provide sensor data exclusively determining an entity and the second data source type is adapted to provide sensor data of a tracked entity, a data adaptation module (24) adapted to generate a data source identifier for each of the sensor data, wherein the data source identifier comprises a data source type identifier corresponding to a data source type receiving the sensor data, wherein a first data source type identifier corresponds to the first data source type and a second data source type identifier corresponds to the second data source type, - means for compiling a data set to be associated comprising as an element the latest sensor data received, - a data association module (30) adapted to perform data association on said data sets to be associated.

13. The system according to claim 12, further comprising a sampler module (28) as means for compiling the data sets to be associated.

14. The system according to claim 12 or claim 13, further comprising a matching module (42) adapted to - matching an element of a data set having said first data source type identifier with each element of a data set having said second data source type identifier, and / or - matching the remaining elements of the data set having the second data source type identifier with each other and forming one or more groups of elements of the data set, wherein: Each group belongs to the same entity.

15. The system according to any one of claims 12-15, further comprising a mapping cache (34) for storing entity selection sets.

16. The system of claim 16, further comprising a data fusion module (38) for combining sensor data for the selected set of entities.

17. A non-transitory computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 11.

18. A non-transitory computer-readable medium comprising instructions which, when executed by a computer, cause the computer to perform the method of any one of claims 1-11.

Images