METHODS AND SYSTEMS FOR DATA FUSION IN A VEHICLE

Abstract

The disclosure relates to a method and system for fusing multi-sensor data in a vehicle. The method includes processing first sensor data to determine a first set of parameters for each of a plurality of objects and processing second sensor data to determine a second set of parameters for each of the plurality of objects. The method further includes generating an intermediate data object for each of the plurality of objects using the first set of parameters and generating a bounding box corresponding to the intermediate data object. The method further includes determining one or more data objects from the second set of parameters and determining the one or more data objects within the bounding box of the intermediate data object as candidate objects.The method further comprises determining a candidate object that satisfies two or more predetermined fusion conditions as a related object and generating a fused data object based on the determination by performing data fusion of the intermediate data object and the related object.

Description

PREAMBLE FOR DESCRIPTION:In the following description, the invention and the manner in which it is carried out will be explained in more detail:DESCRIPTION OF THE INVENTIONTechnical FieldThe present disclosure relates to a technical field of autonomous vehicles, and more particularly, to a method and system for data fusion for an autonomous vehicle.BACKGROUND OF THE DISCLOSUREModern vehicles, such as smart drive vehicles, are equipped with assisted and conditional automation systems, such as advanced driver assistance systems (ADAS). The ADAS may be deployed at various levels defined by the Society of Automotive Engineers (SAE). For example, the automation may correspond to the SAE levels L2 and L3. ADAS plays the most important role in vehicles to enable safe and comfortable driving. Advanced functions of SAE L2 systems such as automatic lane change and adaptive cruise control and SAE L3 systems such as drive pilot must handle passing vehicles and motorized two-wheels such as motor wheels with a higher safety margin. In particular, when detecting emergency vehicles in the rear (when the vehicle enters traffic), the system must initiate an emergency corridor. Particularly, when emergency vehicles such as emergency trucks, fire fights, etc. are detected behind the vehicle, the smart vehicle must generate ADAS events in time and precision to make the emergency vehicles travel a path. The ADAS events include automatic braking, adaptive cruise control, pedestrian detection, collision avoidance, road sign detection, lane keeping, cross traffic warning, parking assist, collision warning, blind spot detection, etc. The smart vehicle may generate each ADAS event by merging the data of one or more sensors. The one or more sensors may be a camera, a radio detection and ranging (RADAR), light detection and edge (LiDAR), and other types of sensors configured in the vehicle. Merging the data of one or more sensors is a decisive part of the generation of the ADAS events.Existing methods, such as those described in CN114631117A, include merging data from a plurality of sensors using a series of neural networks and retransmitting data within the neural networks. However, the existing methods require high computational resources to create a data fusion, which may also be time consuming.Therefore, there is a need to provide a system and method for data fusion that requires less computational resources and allows timely and accurate ADAS events to solve one or more of the problems noted above.The information disclosed in this Background of the Disclosure section is only for enhancement of understanding of the general background of the disclosure and should not be taken as an acknowledgement or an indication that this information is prior art that is already known to a person of ordinary skill in the art.SUMMARY OF THE DISCLOSUREThe disclosure relates to a method for multisensor data fusion in a vehicle. The method includes processing first sensor data to determine a first set of parameters for each of a plurality of objects and processing second sensor data to determine a second set of parameters for each of the plurality of objects. The method further includes generating an intermediate data object for each of the plurality of objects using the first set of parameters and generating a bounding box corresponding to the intermediate data object. The method further comprises determining one or more data objects from the second set of parameters and determining the one or more data objects within the bounding box of the intermediate data object as candidate objects. Further, the method includes determining a candidate object that satisfies two or more predetermined fusion conditions as a corresponding object, and generating a fused data object based on the determination by performing data fusion of the intermediate data object and the corresponding object.The disclosure relates to a system for data fusion in a vehicle. The system includes a memory and a processor communicatively coupled to each other. The processor is configured to process first sensor data to determine a first set of parameters for each of a plurality of objects and process second sensor data to determine a second set of parameters for each of the plurality of objects. The processor is further configured to generate an intermediate data object for each of the plurality of objects using the first set of parameters and create a bounding box corresponding to the intermediate data object. The processor is further configured to determine one or more data objects from the second set of parameters and determine the one or more data objects within the bounding box of the intermediate data object as candidate objects. Further, the processor is configured to determine a candidate object satisfying two or more predetermined fusion conditions as a related object, and generate a fused data object based on the determination by performing data fusion of the intermediate data object and the related object.The foregoing summary is illustrative only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, other aspects, embodiments and features will become apparent by reference to the drawings and the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGSThe accompanying drawings, which form a part of this disclosure, illustrate exemplary embodiments and together with the description, explain the principles disclosed. In the figures, the leftmost digit(s) of a reference numeral indicates the figure in which the reference numeral first appears. In the figures, the same numbers are used to indicate like features and components. Some embodiments of systems and / or methods according to embodiments of the present subject matter will now be described, by way of example only, and with reference to the accompanying figures, in which: FIG. 1 illustrates an example architecture of a system for merging multisensor data in a vehicle according to an embodiment of the present disclosure; FIG. 2 shows a detailed block diagram of a multi-sensor data fusion system (MSDFS) according to an embodiment of the present disclosure; FIGS. 3a-3c show an example method of data mapping for performing data fusion, according to an embodiment of the present disclosure; FIGS. 4a-4b show an example method for generating fused data objects for data fusion, according to an embodiment of the present disclosure; and FIG. 5 shows a flow diagram of a method for data fusion in the vehicle 102 according to another embodiment of the present disclosure.Those skilled in the art should appreciate that all block diagrams included herein represent conceptual views of systems embodying the principles of the present subject matter. It will also be understood that all flowcharts, flowcharts, state transition diagrams, pseudo code, and the like represent various processes that are substantially embodied in a computer readable medium and that can be executed by a computer or processor, regardless of whether such a computer or processor is explicitly illustrated.DETAILED DESCRIPTIONAs used herein, the word "exemplary" is used in the sense of "serving as an example, instance, or illustration.". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be considered preferred or advantageous over other embodiments. Although the disclosure is susceptible to various modifications and alternative forms, a specific embodiment thereof has been shown by way of example in the drawings and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the specific forms disclosed, but on the contrary, the disclosure is intended to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.The terms "comprises," "comprises," "includes," or other variants thereof are intended to cover a non-exclusive inclusion, such that a structure, apparatus, or method comprising a list of components or steps not only includes those components or steps, but may also include other components or steps not expressly listed or associated with such structure or apparatus or method. In other words, one or more elements in a system or device initiated with "comprises... a" does not exclude, without further limitations, the presence of other elements or additional elements in the system or method.In the following detailed description of the embodiments of the disclosure, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is therefore not to be taken in a limiting sense.The present disclosure provides methods and systems for data fusion of a first sensor, e.g., a camera sensor, and a second sensor, e.g., a radar sensor. The system processes sensor data to evaluate data for first objects received from the first sensor and second objects received from the second sensor in the environment of the vehicle. The system also determines whether the first object is associated with the second object based on one or more parameters. The one or more parameters may include, but are not limited to, a distance between the first object and the second object, an object classification of the first object and the second object, object-to-lane information of the first object and the second object, a width overlap between the first object and the second object, and a difference between a motion parameter such as the speed of the first object and the second object. When it is determined that the first object is associated with the second object, the system performs a fusion based on a weighted fusion of the parameters of the first object and the parameters of the second object and generates a fused data object. The system also evaluates historical fusion data to generate parameters of the fused data object in scenarios where the first sensor or the second sensor provides a false or unknown classification of the object. In addition, the system predicts the location or position of each object using a quadratic polynomial equation in both the x and y directions to reduce the use of the computational resources required for data fusion.FIG. 1 illustrates an example architecture of a system for data fusion in a vehicle according to an embodiment of the present disclosure.As shown in FIG. 1, system architecture 100 may include one or more components configured to perform data fusion of sensor data in a vehicle 102 along a route. In one example, the vehicle 102 may be any type of transport vehicle, such as, but not limited to, a motorcycle, a car, a bus, a truck, and the like. In another example, the vehicle 102 may be any service vehicle, such as, but not limited to, a lift truck or fire truck. In one embodiment, system architecture 100 may include, but is not limited to, a data fusion system (MSDFS) 104 in an electronic control unit (ECU) of vehicle 102. The system architecture 100 also includes a first sensor 106 and a second sensor 108 configured within the vehicle 102 and communicatively coupled to the MSDFS 104. In some embodiments, system architecture 100 may also include a plurality of sensors deployed in vehicle 102. The system architecture 100 also includes a sensor fusion database 110 communicatively coupled to the ECU of the vehicle 102.In operation, the MSDFS 104 may be configured to process first sensor data to determine a first set of parameters for each of a plurality of objects and process second sensor data to determine a second set of parameters for each of the plurality of objects. The MSDFS 104 may be configured to generate an intermediate data object for each of the plurality of objects using the first set of parameters and create a bounding box corresponding to the intermediate data object. Further, the MSDFS 104 may be configured to determine one or more data objects from the second set of parameters and determine the one or more data objects within the boundary frame of the intermediate data object as candidate objects. The MSDFS 104 may be further configured to determine a candidate object that satisfies two or more predetermined fusion conditions as an associated object. The MSDFS 104 may be configured to generate a fused data object based on the determination by performing data fusion of the intermediate data object and the associated object.In one embodiment, the first sensor 106 may be any image sensor configured to capture one or more images of an external environment of the vehicle 102, also referred to herein as first sensor data. In an embodiment, the first sensor 106 may be a multi-purpose camera mounted at a rear position of the vehicle 102. The first sensor 106 may be communicatively coupled to the MSDFS 104. In some embodiments, the first sensor 106 may include a plurality of image sensors mounted at one or more locations in the vehicle 102. For example, a plurality of cameras may be mounted to a front portion, a rear portion, a left portion, a right portion, and an upper portion of the vehicle 102. In some other embodiments, the first sensor 106 may include any type of camera, such as, but not limited to, an infrared camera. In an example, the first sensor 106 may detect one or more vehicles traveling on the rear of the vehicle 102.In one embodiment, the second sensor 108 may be a sensor configured to sense an area of one or more objects in the exterior environment of the vehicle 102. In one embodiment, the second sensor 108 may be a Radio Detection and Ranging (RADAR) sensor, or a Light Detection and Ranging (LiDAR) sensor, or an Ultrasound (USS) sensor. In some embodiments, the second sensor 108 may be comprised of a plurality of RADAR sensors mounted at the rear position of the vehicle 102. In an example, the plurality of RADAR sensors may include a first RADAR sensor deployed at a left rear position and a second RADAR sensor deployed at a right rear position of the vehicle 102. In operation, the second sensor 108 may be configured to transmit one or more radio signals to one or more objects and receive one or more reflected signals from the one or more objects. The second sensor 108 may be further configured to determine the distance of the one or more objects based on the reflected signals and the transmitted signals. In an example, the second sensor 108 may detect one or more vehicles traveling on the rear of the vehicle 102.The sensor fusion database 110 may store historical fusion data of one or more objects for a plurality of times. The historical fusion data may include one or more fusion parameters, such as, but not limited to, an object classification, a position, and an identifier of the one or more objects.FIG. 2 shows a detailed block diagram 200 of the MSDFS 104, according to an embodiment of the present disclosure.The MSDFS 104 may include, without limitation, a processor 202, a memory 204, and a plurality of modules 206 and data 208. The processor 202 may be any hardware processing system, e.g., a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a system on chip (SOC), an electronic control unit (ECU), or other type of processing system. The memory 204 may be any type of hardware data storage device, such as read only memory (ROM), random access memory (RAM), temporary or permanent storage system.The plurality of modules 206 may include a data processing module 210, a fusion time prediction module 212, a data association module 214, a fusion generation and update module 216, and an object tracking module 218. The data 208 may include the sensor data 220, object parameters 222, intermediate data object 224, candidate objects 226 massociated objects 228, fused data objects 230, and historical data buffers 232. In some embodiments, the plurality of modules 206 may also be configured within the processor 202.In one embodiment, the data processing module 210 may be configured to receive sensor data 220 from the first sensor 106 and the second sensor 108 and process the sensor data. The data processing module 210 may receive first sensor data from the first sensor 106 to determine a first set of parameters for each of a plurality of objects. The first set of parameters may be estimated based on sensor characteristics, sensor data processing characteristics, and sensor-object characteristics. The data processing module 210 may receive second sensor data and process second sensor data to determine a second set of parameters for each of the plurality of objects. The plurality of objects may correspond to a plurality of vehicles external to the vehicle 102 traveling on the same road on which the vehicle 102 is traveling.The first set of parameters may include, among other things, a size, a position, one or more motion parameters such as a velocity, an acceleration, a variance, a covariance, an identifier, and a classification of an object received from the first sensor 106. Similarly, the second set of parameters may include, but is not limited to, a size, a position, one or more motion parameters such as a velocity, an acceleration, a variance, a covariance, an identifier, and a classification of an object received from the second sensor 108. The data processing module 210 may facilitate approximating a weight or quality of each object. The data processing module 210 may store the first set of parameters and the second set of parameters as object parameters 222 in the memory 204 and send the object parameters 222 to the sensor fusion database 110 for storage.The fusion time prediction module 212 may be configured to generate an intermediate data object 224 for each of the plurality of objects using the first set of parameters. The fusion time prediction module 212 may evaluate the first set of parameters to recognize a plurality of objects to perform data fusion for a current time and may generate an intermediate data object 224 for each of the plurality of objects. The intermediate data object 224 may correspond to an external object and may include one or more parameters, such as, but not limited to, the first set of parameters.The data assignment module 214 may be configured to determine whether one of the objects detected by the second sensor is associated with the intermediate data object 224 to perform the data fusion. In some embodiments, the data assignment module 214 may be configured to determine whether one of the objects detected by the first sensor is associated with the intermediate data object 224 to perform data fusion. The data assignment module 214 may create a bounding box for the intermediate data object 224 based on a classification of the intermediate data object 224 and a distance of the intermediate data object 224 from the vehicle 102. The shape of the bounding box may depend on the shape of the object. For example, a car has a smaller bounding box as compared to a truck having a larger bounding box. The bounding rectangle may have, without being limited thereto, a two-dimensional shape with dimensions such as length and width or a three-dimensional shape with the dimensions length, width and height together with the shape orientation and the yaw rate.For example, a bounding box may have a rectangular shape surrounding an object and providing information about one or more parameters of the object. The one or more parameters may include, among other things, a position, an object classification, and a confidence value of the bounding box. In some embodiments, the bounding box may be an adaptive bounding box that can dynamically change the one or more parameters of the object or the shape of the object based on historical or current data. The data assignment module 214 may determine one or more data objects from the second set of parameters. The data association module 214 may further determine some of the one or more data objects that are within the bounding box of the intermediate data object 224 as candidate objects 226. Alternatively, the data association module 214 may discard some of the one or more data objects that are not within the bounding box of the intermediate data object 224.FIGS. 3 a- 3 c show an example method of data mapping for performing data fusion according to an embodiment of the present disclosure.FIG. 3 ashows an example intermediate data object 224 for which a bounding box 304 was generated. The data assignment module 214 may determine an example data object 306 from the second set of parameters and determine whether the data object 306 is a candidate object 226 for the intermediate data object 224. Because the data object 306 is outside the bounding box 304 of the intermediate data object 224, the data object 306 may not be a candidate object 226 for the intermediate data object 224. Thus, the data assignment module 214 may discard the data object 306 for fusion with the intermediate data object 224.FIG. 3 b shows another example data object 308 that may be considered for fusion with intermediate data object 224. Since data object 308 is within bounding box 304, data object 308 may be a candidate object 226 for intermediate data object 224. Similarly, FIG. 3 c illustrates that an example data object 310 may also function as a candidate object 226 for the intermediate data object 224.Thereafter, the data association module 214 may be configured to determine an association between one or more candidate objects 226 and the intermediate data object 224 based on predetermined fusion conditions. A first predetermined fusion condition may be to determine whether a distance between the intermediate data object and the candidate object is less than a threshold distance. The threshold distance may be determined as a maximum distance between the intermediate data object and the candidate object when the intermediate data object and the candidate object belong to the same external object.A second predetermined fusion condition may be to determine whether a difference between a position of the intermediate data object in a spatial dimension and a position of the candidate object in the spatial dimension is less than a threshold. The threshold may be determined as a minimum width overlap between the intermediate data object and the candidate object when the intermediate data object and the candidate object belong to the same external object.A third predetermined fusion condition may be to determine whether a difference between a motion parameter of the intermediate data object and the motion parameter of the candidate object is less than a threshold motion parameter difference. The threshold value for the motion parameter difference may be determined as a maximum difference between the motion parameter of the intermediate data object and the motion parameter of the candidate object when the intermediate data object and the candidate object belong to the same external object. In some embodiments, the motion parameter may be either speed or acceleration.The data association module 214 may determine a candidate object 226 that satisfies two or more predetermined fusion conditions as an associated object 228 for the intermediate data object 224. For example, in FIG. 3 b, the candidate object 308 satisfies the second and third predetermined fusion conditions, and thus the candidate object 308 is associated with the intermediate data object 224. Alternatively, if two or more candidate objects 226 meet two or more predetermined fusion conditions, the data association module 214 determines a candidate object 226 having a minimum distance evaluated in the first predetermined fusion condition and a minimum difference evaluated in the second predetermined fusion condition. Further, the data assignment module 214 may determine the candidate object 226 as the associated object 228 for the intermediate data object 224. For example, in FIG. 3 c, candidate objects 308 and 310 both satisfy two or more conditions. However, since candidate object 308 has the smallest distance and the smallest difference, candidate 310 is considered an associated object 228 for intermediate data object 224.Fusion generation and update module 216 may be configured to generate a new fused data object or update an already present fused data object based on the data mapping. Fusion generation and update module 216 may generate a new fused data object 230 by fusing intermediate data object 224 and associated object 228 if there is an associated object 228 for intermediate data object 224. Alternatively, fusion data generation and update module 216 may generate one or more fused data objects if there is no associated object 228 for intermediate data object 224. In this scenario, fusion generation and update module 216 may generate a first fused data object for intermediate data object 224 and a second fused data object for candidate object 226.FIGS. 4a-4b illustrate an example method for generating fused data objects for data fusion, according to an embodiment of the present disclosure.FIG. 4a illustrates a method for creating a merged data object 402 by merging the intermediate data object 306 and the candidate object 308. FIG. 4 b illustrates a method for generating a first fused data object 404 for the intermediate data object 306 and a second fused data object 406 for the candidate object 308.Fusion generation and update module 216 may perform a weighted fusion of intermediate data object 224 and associated object 228 to generate fused data object 230 based on a weight assigned to each of the first and second parameters. In some embodiments, the weighted fusion may be similar to one or other fusion approach or derivatives as mentioned in "B. Duraisomy, T. Schwarz, and C. Wöhler, Track level fusion algorithms for automotive safety applications.". The fusion generation and update module 216 may predict one or more parameters of the fused data object for each of a plurality of times based on one or more weighted fusion and historical fusion data stored in a historical data buffer 232. The fusion generation and update module 216 may update the one or more parameters of the fused data object 230 based on the prediction for each time point. Additionally, fusion generation and update module 216 may delete fused data object 230 from memory 204 when neither first sensor 106 nor second sensor 108 captures fused data object 230 for a threshold amount of time, e.g., 60 ms.The object tracking module 218 may be configured to determine one or more parameters of the fused data object 230 based on the weighted fusion or the historical data. The one or more parameters may include, but are not limited to, an object classification, an identifier, or a position of the fused object. The object tracking module 218 may determine that an object classification of the intermediate data object 224 is one of the predefined objects or unknown. The object tracking module 218 may predict the object classification of the merged data object 230 based on the weighted fusion when the object classification of the intermediate data object 224 is determined to be the predefined object. Alternatively, the object tracking module 218 may predict the object classification of the fused data object 230 based on the historical fusion data when the object classification of the intermediate data object 224 is determined to be unknown. For example, intermediate data object 224 has been classified as "car" at an earlier time, and therefore fused data object 230 is also classified as "car.". In this example, at the current time when intermediate data object 224 was classified as "unknown", the classification of fused data object 230 is predicted to be "auto" from the classification at the previous time.In one embodiment, the object tracking module 218 may determine that at least one of the identifiers of the intermediate data object 224 and the identifiers of the associated object 228 is unknown or different from an identifier of the fused object at an earlier time. In this embodiment, the object tracking module 218 may predict that the fused data object identifier 230 is the same as the fused data object identifier at an earlier time. For example, if an identifier of the associated object 228 is "1" and that of the fused data object 230 is "135" at the previous time and the identifier of the associated object 228 is "2" at the current time, the identifier of the fused data object 230 is maintained as "135" based on the data at the previous time.In another embodiment, the object tracking module 218 may predict a position of the intermediate data object 224 and the position of the associated object 228. The object tracking module 218 may further predict the position of the fused data object 230 based on the position of the intermediate data object 224 and the associated object 228. The object tracking module 218 may use motion and target models represented by polynomial equations to evaluate the position of the intermediate data object 224 and the position of the associated object 230 as shown below. where x(t) is a position of the object in the x direction at a time "t", a is an acceleration of the object in the x direction, v is the speed of the object in the x direction, and x is a distance of the object in the x direction. where y(t) is a position of the object in the y direction at a time "t", a is an acceleration of the object in the y direction, v is the speed of the object in the y direction, and y is the distance of the object in the y direction.The object tracking module 218 may further generate a list of the predicted fused data objects 230 for the vehicle 102 to one or more ADAS systems to generate one or more ADAS events. Each fused data object 230 may be assigned a priority order based on a distance between the fused data object 230 and the vehicle 102. In one embodiment, the fused data object 230 that is closer to the vehicle 102 is assigned a higher priority and a fused data object 230 that is farther from the vehicle 102 is assigned a lower priority.In this way, the MSDFS 104 may provide stable, robust, and unique fusion data for each vehicle or pedestrian detected at the rear of the vehicle 102. The MSDFS 102 also provides accurate information about the position, speed, acceleration, yaw rate, and classification for each object detected at the rear and side of the vehicle 102. All targets detected by the sensors are fused and tracked with unique identifiers for the entire life cycle of the object. The system requires the least amount of computing resources by performing unique preprocessing steps, historical fusion data maintenance for sensor targets, data fusion, and association techniques. The system assists trajectory planning in the case of emergency vehicles, motorbikes, cyclists, and pedestrians approaching vehicle 102 from behind and from the side.The system provides a higher safety margin in detecting vehicles passing, and we have developed a multi-sensor data fusion module (e.g., cameras, radar, ultrasonic sensors, short range lidars) that can efficiently combine the information of the sensors covering the rear and side regions of the system vehicle to provide a consistent environmental view (at each operation) and powered two-wheels such as motorcycle.FIG. 5 shows a flow diagram of a method for data fusion in the vehicle 102 according to another embodiment of the present disclosure.The method 500 may be described in the general context of computer-executable instructions. The computer-executable instructions may generally include routines, programs, objects, components, data structures, procedures, modules, and functions that perform particular functions or implement particular abstract data types. The order in which method 500 is described is not intended to be limiting, and any number of the described method blocks may be combined in any order to implement the method. Moreover, individual blocks may be deleted from the methods without compromising the scope of the subject matter described herein. Moreover, the method may be implemented in any suitable hardware, software, firmware, or combination thereof.At block 502, the MSDFS 104 may process first sensor data to determine a first set of parameters for each of a plurality of objects and process second sensor data to determine a second set of parameters for each of the plurality of objects.At block 504, the MSDFS 104 may generate an intermediate data object 224 for each object of the plurality of objects using the first set of parameters.In block 506, the MSDFS 104 may create a bounding box corresponding to the intermediate data object 224.In block 508, the MSDFS 104 may determine one or more data objects from the second set of parameters.In block 510, the MSDFS 104 may determine that one or more data objects within the boundary frame of the intermediate data object are considered candidate objects 226.In block 512, the MSDFS 104 may determine a candidate object 226 that satisfies two or more predetermined fusion conditions as an associated object 228.In block 514, the MSDFS 104 may generate a fused data object 230 based on the determination by performing data fusion of the intermediate data object 224 and the associated object 228.The steps illustrated are for purposes of explaining the exemplary embodiments shown, and it should be understood that ongoing technological development will alter the manner in which particular functions are performed. These examples are illustrative and not restrictive. Moreover, the boundaries of the functional blocks have been arbitrarily set here for convenience of description. Alternatives (including equivalents, extensions, variations, deviations, etc. from those described herein) will be apparent to persons skilled in the relevant field(s) based on the teachings herein. Such alternatives fall within the scope of the disclosed embodiments.Also, the words "consisting of", "with", "containing", and "including", and other similar forms, are intended to be equivalent in meaning and have an open end such that an element or elements following any of these words are not intended to be an exhaustive list of such element or elements or to be limited to the listed element(s). It is also to be understood that the singular forms used herein and in the appended claims include "a", "an" and "the" plural references unless the context clearly dictates otherwise. Finally, the language used in the specification has been chosen primarily for readability and guidance purposes, and not to delineate or rewrite the subject matter. Accordingly, the disclosure of the embodiments of the disclosure is intended to illustrate, but not limit, the scope of the disclosure. As regards the use of plural and / or singular terms, those skilled in the art may translate the plural to the singular and / or the singular to the plural terms as appropriate to the context and / or application. The various singular / plural permutations may be expressly listed herein for clarity.Reference Number:100 System architecture 102 vehicle 104 multi-sensor data fusion system 106 first sensor 108 second sensor 110 database for sensor fusion 202 processor 204 memory 206 module 208 data 210 data processing module 212 fusion time prediction module 214 data assignment module 216 fusion generation and update module 218 object tracking module 220 sensor data 222 object parameter 224 intermediate data object 226 candidate object 228 associated object 230 fused data object 232 historical data buffer 304 bounding box 306, 308, 310 example data objects 402, 404, 406 example fused data objectsReferences included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Cited Non-Patent LiteratureB. Duraisamy, T. Schwarz and C. Wöhler, Track level fusion algorithms for automotive safety applications

[0040]

Claims

A method for merging multisensor data in a vehicle, the method comprising: processing first sensor data to determine a first set of parameters for each of a plurality of objects and processing second sensor data to determine a second set of parameters for each of the plurality of objects; generating an intermediate data object of each of the plurality of objects using the first set of parameters; creating a bounding box corresponding to the intermediate data object; determining one or more data objects from the second set of parameters; determining the one or more data objects within the bounding box of the intermediate data object as candidate objects; determining a candidate object that satisfies two or more predetermined merging conditions based on one or more probabilistic models as an associated object; generating a fused data object based on the determination by performing multisensor data fusion of the intermediate data object and the associated object.The method of claim 1, wherein the predetermined melting conditions comprise: a distance between the intermediate data object and the candidate object is less than a threshold distance; a difference between a position of the intermediate data object in a spatial dimension and a position of the candidate object in the spatial dimension is less than a threshold; and a difference between a motion parameter of the intermediate data object and the motion parameter of the candidate object is less than a threshold speed difference.The method of claim 1, wherein performing the data fusion of the intermediate data object and the associated object comprises: applying a weighted fusion of the intermediate data object and the associated object to generate the fused data object based on a weight associated with each of the first set of parameters and the second set of parameters; and predicting one or more parameters of the fused data object for each of a plurality of times based on one or more of the weighted fusion and historical fusion data; and updating the one or more parameters of the fused data object based on the prediction.The method of claim 3, wherein the prediction comprises: determining that an object classification of the intermediate data object is either a predefined object or unknown; predicting the object classification of the fused data object based on the weighted fusion upon determining that the object classification of the intermediate data object is the predefined object; or predicting the object classification of the fused data object based on the historical fusion data upon determining that the object classification of the intermediate data object is unknown.The method of claim 4 further comprising: determining that at least one of the identifier of the intermediate data object and the identifier of the associated object is unknown or different from an identifier at a prior time; and predicting that the identifier of the fused data object is the same as an identifier of the fused data object at a prior time.The method of claim 3, wherein the prediction comprises: predicting a position of the intermediate data object and the position of the associated object; and predicting a position of the fused data object based on the position of the intermediate data object and the associated object.The method of claim 1, further comprising: determining that a candidate object is not associated with the intermediate data object; generating a first fused data object based on the intermediate data object; and generating a second fused data object based on one or more candidate objects.A system for merging multisensor data in a vehicle, the system comprising: a memory; and a processor communicatively coupled to the memory, the processor configured to: process first sensor data to determine a first set of parameters for each of a plurality of objects and process second sensor data to determine a second set of parameters for each of the plurality of objects; generate an intermediate data object of each of the plurality of objects using the first set of parameters; create a bounding box corresponding to the intermediate data object; determine one or more data objects from the second set of parameters; determine one or more data objects within the bounding box of the intermediate data object as candidate objects; a candidate object that satisfies two or more predetermined fusion conditions is determined as an associated object based on one or more probabilistic models; and a fused data object is generated based on the determination by performing multisensor data fusion of the intermediate data object and the associated object.The system of claim 8, wherein the predetermined fusion conditions comprise: a distance between the intermediate data object and the candidate object is less than a threshold distance; a difference between a position of the intermediate data object in a spatial dimension and a position of the candidate object in the spatial dimension is less than a threshold; and a difference between a motion parameter of the intermediate data object and the motion parameter of the candidate object is less than a threshold speed difference.The system of claim 8, wherein to perform the data fusion of the intermediate data object and the associated object, the processor is configured to perform the steps of: applying a weighted fusion of the intermediate data object and the associated object to generate the fused data object based on a weight associated with each of the first set of parameters and the second set of parameters; and predicting one or more parameters of the fused data object for each of a plurality of times based on one or more of the weighted fusion and historical fusion data; and updating the one or more parameters of the fused data object based on the prediction.

Images