Modular architecture for asynchronous late fusion of objects

Abstract

The present disclosure relates to systems and methods for asynchronous late fusion of measurements. State information and intermediate parameters can be computed as measurements arrive and are stored. When late sensor measurements arrive out of time order from when the measurements were generated, the late measurements are stored in time order, not in the order the measurements arrived. The state information is then recomputed to account for the late arriving sensor measurements, with state outputs propagated forward in time order using previously computed intermediate parameters to speed up the computation. In this way, more accurate modified state information is efficiently generated, accounting for any late arriving measurements.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application Serial No. 63 / 240,552, filed September 3, 2021, the entire contents of which are expressly incorporated herein by reference.

[0003] introduction

[0004] This disclosure relates generally to autonomous navigation. More specifically, this disclosure relates to the asynchronous late fusion of objects in applications such as autonomous navigation. Summary of the Invention

[0005] Applications such as autonomous navigation often rely on object fusion and other methods for accurate object detection and vehicle path determination. Specifically, vehicles, such as autonomous vehicles, typically employ multiple different sensors to detect nearby objects. While arrays of different sensors can sometimes effectively detect objects, they also present challenges. For example, individual sensors (such as those used in Advanced Driver Assistance Systems (ADAS)) may differ in their operating timing, causing some measurements to arrive after certain outputs have been calculated. Consequently, the calculated state information may be based on incomplete or outdated sensor measurements, reducing the accuracy and reliability of applications such as autonomous navigation.

[0006] Therefore, this paper discloses systems and methods for processing sensor data, such as ADAS sensor measurements, in a modular manner without requiring significant process changes for different sensors and sensor timings, where these measurements arrive out of order and asynchronously. Sensor measurements can arrive asynchronously and be rearranged in chronological order. State information can then be recalculated after the rearrangement. For example, when vehicle sensor information is received, the system can calculate state information, such as the state of objects around the vehicle, e.g., object shape, position, speed, etc. Subsequently, later sensor measurements may arrive, with a time index preceding some other slightly earlier received measurements. This can occur when, for example, some sensors experience delays in transmitting their measurements due to, for example, hysteresis, high computational overhead, etc. State information can be recalculated from the later sensor measurements and any slightly earlier sensor measurements to account for the correct time at which the later sensor measurements were obtained. That is, previously calculated state information is recalculated to account for the later-arriving sensor measurements. This modified state information can then be used for any desired purpose.

[0007] In some embodiments of this disclosure, later measurement results may arrive in a different epoch. That is, state information calculations can be performed within a specified epoch, and it is possible that later measurement results may arrive in a different epoch, later than their associated time index. For example, a set of sensor measurements may have already been received during a first epoch and have a time index falling within that first epoch. Meanwhile, later sensor measurements may also have a time index falling within the first epoch, but due to some delay, may actually arrive in a subsequent second epoch. The later-arriving measurements can then be moved to the appropriate time index of the first epoch, and the state information for that first epoch can be recalculated accordingly.

[0008] Various embodiments of this disclosure envision any suitable method for storing sensor measurement results. In some embodiments, sensor measurement results may be stored in a buffer in chronological order of their time indices. If a later sensor measurement result arrives, where its time index is earlier than one or more measurements that have already arrived, a measurement result with a slightly later time index can be retrieved from the buffer (e.g., thrown out), the later sensor measurement result can be pushed onto the buffer, and the slightly later measurement result can be pushed back onto the buffer in chronological order of its time index. That is, later sensor measurement results can be inserted into the buffer in chronological order of their time indices, such that received sensor measurement results are stored in chronological order of their time indices rather than in the order of their arrival. Thus, even if a later sensor measurement result is received, where its time index is between the indices of two other measurements already stored in the buffer, the later sensor measurement result is still stored between these two existing measurements.

[0009] Sensor measurements can be any measurement obtained from any sensor. In some embodiments of this disclosure, the measurements can originate from sensors located on or coupled to the vehicle. This includes autonomous vehicles and any sensors that may be used therein. For example, the measurements can be radio detection and ranging (radar) measurements or light detection and ranging (LiDAR) measurements. Measurements can also include images from sensors that detect light or other radiation of any wavelength, such as visible light sensors, infrared or ultraviolet light sensors, etc.

[0010] State information can be any information calculated from sensor measurements and describing the state of the detected object. In some embodiments, this state information may include quantities such as object fusion output. More specifically, sensor measurements such as images and LiDAR output can be used for object fusion operations to determine the probability of an object being present at a particular location. Embodiments of this disclosure envision using any one or more types of measurements to determine the probability of an object being present at a particular location. Furthermore, any other quantities of interest can be determined from the received sensor measurements.

[0011] The system of this disclosure can also calculate any other quantity from sensor measurements or from any calculated state information. As an example, the system can determine an occlusion region associated with a determined object. Once an object is determined from, for example, one or more object fusion operations, its position and size can be used to determine the associated occlusion region, where the object blurs the field of view of an observer located at or near the sensor's position. The occlusion region can be used for any purpose, such as preventing object fusion calculations within the occlusion region, since no object can be seen in the occlusion region anyway, thus saving computational resources.

[0012] Embodiments of this disclosure also cover any configuration of a system for performing any measurement result fusion and sensor measurement result arrangement methods and processes described herein. Embodiments of this disclosure also cover any object, such as a vehicle, in conjunction with any such system for performing any measurement result fusion and sensor measurement result arrangement methods and processes described herein. Attached Figure Description

[0013] The above and other objects and advantages of this disclosure will become apparent from the following specific embodiments taken in conjunction with the accompanying drawings, wherein similar reference characters always refer to similar parts, and wherein:

[0014] Figure 1A The image is a top view of a road and vehicles thereon according to an embodiment of this disclosure, conceptually illustrating the operation of a system and method for asynchronous late fusion of measurement results;

[0015] Figure 1B This is a top view of the object fusion output according to an embodiment of the present disclosure, conceptually illustrating the operation of a system and method for asynchronous late fusion of measurement results;

[0016] Figure 2 This is a top view of an exemplary sensor for asynchronous late fusion of measurement results according to an embodiment of this disclosure and its corresponding field of view;

[0017] Figure 3 This is a timeline showing the reception and storage of late sensor measurement results according to an embodiment of this disclosure;

[0018] Figure 4 A block diagram of components of a system for determining the drivable space of a vehicle according to some embodiments of the present disclosure is shown;

[0019] Figure 5 This is a block diagram of the logical components of a system for asynchronous late fusion of measurement results according to an embodiment of this disclosure;

[0020] Figure 6This is a flowchart illustrating an exemplary method for asynchronous late fusion of measurement results according to an embodiment of the present disclosure; and

[0021] Figure 7 This is a top view conceptually illustrating the determination of the occlusion area according to an embodiment of this disclosure. Detailed Implementation

[0022] In one embodiment, this disclosure relates to a system and method for asynchronous late fusion of measurement results. State information can be computed as measurement results arrive and are stored. When late sensor measurements arrive out of the chronological order in which they were generated, the late measurements are stored in chronological order, not in the order of arrival. The state information is then recalculated to account for the late-arriving sensor measurements, wherein the state output is propagated forward chronologically. This generates more accurate modified state information that takes into account any late-arriving measurements.

[0023] Figure 1A This is a top view of a road and vehicles thereon according to an embodiment of the present disclosure, conceptually illustrating the operation of a system and method for asynchronous late fusion of measurement results. Here, vehicle 100 may carry multiple onboard sensors having fields of view 120, 130, and 140. These sensors can detect objects such as moving vehicles, sidewalks, and crosswalks when they enter the corresponding fields of view 120, 130, and 140 of the sensors. In some embodiments, the sensors of vehicle 100 may acquire measurement results at discrete times, such as at fixed intervals. Therefore, the processor of vehicle 100 may receive sensor outputs at fixed times. For example, when vehicle 150 is within field of view 120, vehicle 100 may receive measurement results acquired at time t1 via one or more sensors having field of view 120. Similarly, vehicle 100 may receive measurement results acquired at a slightly later time t2 from sensors having both fields of view 130 and 140, each field of view showing vehicle 150 after movement. At a slightly later time t3, vehicle 100 can receive measurement results from a sensor with a field of view 130, which shows vehicle 150 moving further away.

[0024] In some implementations, vehicle 100 can use sensor measurements acquired at each time point from time t1 to time t3 to perform object fusion operations. For example, measurements acquired at time t2 from fields of view 130 and 140, along with position information extrapolated from the position and velocity of vehicle 150 determined at time t1, can be used together to determine quantities such as the possible position of vehicle 150 at time t2.

[0025] In some implementations, sensor measurements may arrive at the processor of vehicle 100 asynchronously but in the chronological order in which they were generated. For example, sensor measurements acquired at time t1 may arrive before those acquired at time t2, and sensor measurements acquired at time t2 may in turn arrive before those acquired at time t3. However, some sensor measurements may arrive later, not in chronological order. For example, due to any number of factors such as hysteresis, processing overhead of certain sensors, etc., the measurement of vehicle 150 acquired by the sensor with field of view 130 at time t2 may arrive after the measurement of vehicle 150 acquired by the sensor with field of view 140 at time t3. In this case, instead of simply discarding the later-arriving measurements and calculating the position of vehicle 150 at time t2 without using the later-arriving measurements acquired from field of view 130, the embodiments of this disclosure can recalculate the position of vehicle 150 at time t2 once the measurements from field of view 130 arrive. That is, once the measurement result of vehicle 150 acquired at time t2 in field of view 140 arrives, the position of vehicle 150 at time t2 can be calculated first. Next, once the measurement result of vehicle 150 acquired at time t3 in field of view 140 arrives, the position of vehicle 150 at time t3 can also be calculated. Subsequently, when the later measurement result of vehicle 150 at time t2 in field of view 130 arrives, the position of vehicle 150 at time t2 is recalculated at that point using the later measurement result. In some embodiments, the modified position can then be forward-propagated, i.e., it can be used to update the position of vehicle 150 at a slightly later time t3. This process can also be performed with any one or more detected objects to recalculate the state information of any detected object, such as the object's position, speed, orientation, size (e.g., length and width), shape, etc., such as lane markings of lane 180, pedestrian crossing markings of crosswalk 160, sidewalk 110, intersection 170, etc.

[0026] Figure 1B This is a top view of the object fusion output according to an embodiment of this disclosure, conceptually illustrating the operation of a system and method for asynchronous late fusion of measurement results. As described above, vehicle 100 can measure and track other nearby objects, such as vehicle 150, within its field of view. Figure 1B In the context, vehicle 200 can be similar to... Figure 1AVehicle 150 is one such tracked object. In some embodiments of this disclosure, vehicle 200 may have a previous trajectory state as shown, for example, determining that the object is at this location and exists at this specific time, which may be the state output of a previous set of object fusion operations. The position of vehicle 200 can then be propagated to subsequent position 210 to account for the object's movement over time. When the next measurement result is received, it can be used to calculate the actual position of vehicle 200, i.e., the first measurement result update 220. That is, the propagated estimate 210 is updated to the first measurement result update 220. Subsequently, vehicle 200 is propagated again to position 230 to account for movement over time. When a subsequent measurement result is received, the actual position of vehicle 200 is calculated again as the second measurement result update 240. After another time increment, vehicle 200 can be propagated again to position estimate 250.

[0027] Then, late-arriving measurement results can be received. For example, after calculating position 250, a measurement result acquired between the time when update 220 is generated and the time when update 240 is generated may arrive late. In this case, corresponding to the measured position of vehicle 200 at the time the late-arriving measurement result is acquired, a measurement result update is calculated between update 220 and update 240. This measurement result update is then propagated to modify position 230 and measurement result update 240. Subsequently, the modified position 230 and update 240 are used to determine the modified position estimate 250. That is, the late-arriving measurement results are used to recalculate the state information at the time the late-arriving measurement result was acquired or generated, and this recalculated state information can then be propagated to the current or most recent time, thereby improving the accuracy of the current state.

[0028] It should be noted that the processes of the embodiments of this disclosure can be combined with any sensor measurement results output by any sensor. For example, the processes of the embodiments of this disclosure can be combined with the outputs of any autonomous or other vehicle sensors. Figure 2 This is a top view of an exemplary vehicle sensor and its corresponding field of view for asynchronous late fusion of measurement results according to embodiments of the present disclosure. Here, the vehicle sensor may include a front radar (such as one or more center front radar modules CFR1, CFR2), an angle radar, an object LiDAR (such as a main LiDAR module ML), one or more ADAS or other cameras, and one or more forward-looking cameras FVC, each having the field of view shown. Embodiments of the present disclosure cover any configuration of these sensors arranged in any way to have the field of view shown or other fields of view. Additionally, the sensor may include other sensors not shown, such as GPS and inertial measurement unit (IMU) sensors, or any other desired sensors.

[0029] Figure 3 This illustrates a timeline of the reception and storage of late-arriving sensor measurement results according to embodiments of the present disclosure. As described above, in embodiments of the present disclosure, sensor measurement results can arrive at the processing unit asynchronously and in any order. Each measurement result has an associated time index at which it is obtained from or output from its sensor. Object fusion or other operations can be performed on these measurement results in the chronological order of their time indices. That is, if measurement results do not arrive in chronological order (e.g., an earlier acquired measurement result arrives after a later acquired measurement result), the later-arriving measurement results are rearranged in their appropriate chronological order.

[0030] Measurement results can be analyzed, and status information can be generated for each received measurement result. In some implementations, the updated status information can be used by other systems only at specific intervals. For example, updated status information can be transmitted at the end of each epoch e0, e1. That is, status information can be calculated on a rolling (or other) basis as measurement results arrive, but only transmitted to other systems at certain times, such as at specific intervals. Figure 3 In the example, status information can be calculated or updated for each arriving measurement, with the latest status information transmitted to other vehicle 100 systems at the end of each epoch e0, e1. In some implementations, status information can be stored along with each measurement. That is, the status information generated by each measurement can be stored along with that measurement, such that the status information is stored in the same chronological order as the measurement and updated in the same manner. The status information can be stored in the same or separate storage device or data structure.

[0031] Therefore, in operation, the processor of vehicle 100 can receive sensor measurement results from its onboard sensors at certain times, as shown in the figure. Vehicle 100 can then store and process these measurement results sequentially. Embodiments of this disclosure envision any method and structure through which sensor measurement results can be stored and subsequently processed. In some embodiments, vehicle 100 can store each measurement result in a buffer, such as a buffer in vehicle 100's onboard memory. The buffer can be constructed and operated in any suitable manner. In some embodiments, the buffer can be a first-in, first-out (FIFO) buffer, where sensor measurement results are pushed onto the buffer in chronological order according to their time index. When a later measurement result is received, its position in the buffer can be determined according to its time index, for example, the later measurement result is inserted into the buffer at the position where the time index of the stored measurement results is retained in chronological order. For example, a slightly later measurement result can be ejected from the buffer and stored in, for example, a separate buffer, onto which the later measurement result can be pushed, and the slightly later measurement result can subsequently be ejected from the separate buffer and pushed back into the FIFO buffer in chronological order. In this way, the sensor measurements that arrive later are inserted into the FIFO buffer in the correct time order according to the time index.

[0032] exist Figure 3 In the example, the first measurement result ML1, acquired at time t1, arrives from the main LiDAR module ML, followed by the first measurement result CFR1 acquired by the central front radar module CFR1 at time t2, which is greater than t1. This is followed in sequence: the second LiDAR measurement result ML2, acquired or generated at time t3; the forward-looking camera FVC measurement result FVC1, acquired at time t4; the second CFR measurement result CFR2, acquired at time t5; the third LiDAR measurement result ML3, acquired at time t6; and the third CFR measurement result CFR3, acquired at time t8. Here, t8>t6>t5>t4>t3>t2, ensuring that these measurement results arrive in the order in which they were acquired or generated. These measurement results can be processed in the order of their arrival; for example, each measurement result can be used to generate vehicle 100 status information upon its arrival. At the end of the epoch, i.e., at time e1, state information can be extrapolated from time t8 to time e1, and this extrapolated state information is transmitted or reported to the appropriate system of vehicle 100 for any desired purpose, such as navigation.

[0033] After the end of epoch e1, measurement results may arrive late. For example, a second FVC measurement result FVC2 may be obtained at time t7, where t6 < t7 < t8, but the measurement result FVC2 may not arrive until the end of epoch e1. The measurement result FVC2 can then be inserted in order of its time index t between ML3(t6) and CFR3(t8). As described above, for example, if the measurement results are stored in a buffer, the measurement result FVC2 can be inserted into the buffer between the measurement results ML3 and the measurement result CFR3. The state information can then be extrapolated from ML3 at t6 and used together with the measurement result FVC2 to determine the new state information at time t7. This new state information can then be propagated forward to modify all subsequent states, i.e., the state information at time t7 can be extrapolated to time t8 and used with CFR3 to determine the modified state information for time t8, which is then in turn extrapolated to time e1 to determine the modified epoch end e1 state. In this way, late-arriving measurement results can be placed in their appropriate time-index order and used to recalculate the state information, which can then be propagated forward to determine the modified current or most recent state. Although the modified state is determined after the end of epoch e1, the modified state information is used in subsequent calculations to achieve more accurate state information at the end of epoch e2. Additionally, the modified state information can also be used for any other desired purpose, such as using the state information for testing or debugging purposes, in systems where results that do not need to be generated in real time are used, etc.

[0034] The methods of embodiments of the present disclosure can be implemented in any system that employs late fusion of sensor measurement results. As an example, a vehicle such as an autonomous vehicle can have cameras or other sensors disposed therein or thereon to capture images and other measurements of its surroundings. The vehicle's processing circuitry or remote processing circuitry can then implement the methods and processes described above to analyze late-arriving sensor measurement results. Figure 4 A block diagram of components of a system of one such vehicle 400 in accordance with some embodiments of the present disclosure is shown. Vehicle 400 may correspond to vehicle 100 of FIG. 1. Vehicle 400 can be an automobile (e.g., a coupe, sedan, truck, SUV, bus), a motorcycle, an aircraft (e.g., a drone), a marine vessel (e.g., a boat), or any other type of vehicle.

[0035] Vehicle 400 may include control circuitry 402, which may include processor 404 and memory 406. Processor 404 may include a hardware processor, a software processor (e.g., a processor emulated using a virtual machine), or any combination thereof. In some embodiments, the combination of processor 404 and memory 406 may be referred to as control circuitry 402 of vehicle 400. In some embodiments, processor 404 alone may be referred to as control circuitry 402 of vehicle 400. Memory 406 may include hardware elements for non-transitory storage of commands or instructions that, when executed by processor 404, cause processor 404 to operate vehicle 400 according to the embodiments discussed above and below. Control circuitry 402 may be communicatively connected to components of vehicle 400 via one or more wires or via a wireless connection.

[0036] Control circuitry 402 can be communicatively connected to input interface 416 via input circuitry 408 (e.g., a steering wheel, a touchscreen on display 424, buttons, knobs, a microphone, or other audio capture devices). In some embodiments, the driver of vehicle 400 may be allowed to select certain settings in conjunction with the operation of vehicle 400 (e.g., Figure 3 (The color scheme for urgency levels, the presentation of suggested turn indicators, and when to provide suggested turn indicators, etc.). In some embodiments, control circuitry 402 may be communicatively connected to the vehicle 400's GPS system 440, whereby the driver can interact with the GPS system via input interface 416. GPS system 440 may communicate with multiple satellites to determine the driver's location and provide navigation directions to control circuitry 402.

[0037] Control circuit 402 can be communicatively connected to display 422 and speaker 424 via output circuit 410. Display 422 may be located on the dashboard of vehicle 400 (e.g., Figure 2 The dashboard 204 and / or dashboard 208) and / or the windshield of the vehicle 400 (e.g., Figure 2 The head-up display is located at the windshield (206). For example, Figure 3 The GUI can be generated for display at display 422, and display 422 may include an LCD display, an OLED display, an LED display, or any other type of display. Speaker 424 may be located anywhere within the cabin of vehicle 400, for example, on the dashboard of vehicle 400, or on the interior portion of a door. Display 422 and speaker 424 may be combined to provide the driver of vehicle 400 with steering action indicators suggesting that vehicle 400 steer to the side to avoid obstacles or impassable spaces, providing visual and audio feedback respectively.

[0038] Control circuit 402 may be communicatively connected to haptic element 426 via output circuit 410. Haptic element 426 may be a mechanical device, such as including actuators configured to vibrate to evoke tactile or tactile sensations in the driver's body. The haptic element may be located at one or more locations in various positions within vehicle 400 (e.g., on the driver's seat, passenger seat, steering wheel, brake pedal, and / or accelerator pedal) to provide tactile feedback in conjunction with providing the driver of vehicle 400 with steering action indications suggesting that vehicle 400 be turned laterally to avoid a first obstacle.

[0039] Control circuitry 402 may be communicatively connected (e.g., via sensor interface 414) to sensors (e.g., front sensor 432, rear sensor 434, left-side sensor 436, right-side sensor 438, orientation sensor 418, speed sensor 420). Orientation sensor 418 may be an inclinometer, accelerometer, slew rate sensor, any other pitch sensor, or any combination thereof, and may be configured to provide control circuitry 402 with vehicle orientation values ​​(e.g., vehicle pitch and / or vehicle roll). Speed ​​sensor 420 may be a speedometer, GPS sensor, etc., or any combination thereof, and may be configured to provide control circuitry 402 with a reading of the vehicle's current speed.

[0040] In some implementations, the front sensor 432 may be located at various positions on the vehicle 400 and may be one or more of various types, such as an image sensor, infrared sensor, ultrasonic sensor, radar sensor, LED sensor, LIDAR sensor, etc., and is configured to capture images of nearby objects such as vehicles or other location information (e.g., by outputting light or radio wave signals and measuring the time and / or intensity of the return signal to be detected, and / or performing image processing on images of the environment surrounding the vehicle 400 captured by the image sensor).

[0041] Control circuitry 402 may be communicatively connected to battery system 428, which may be configured to supply power to one or more components of vehicle 400 during operation. In some embodiments, vehicle 400 may be an electric vehicle or a hybrid electric vehicle.

[0042] Control circuit 402 may be communicatively connected to light source 430 via light source control 412. Light source 430 may be, for example, a series of LEDs and may be located in one or more of a variety of locations in vehicle 400 to provide visual feedback in conjunction with a steering action indicator that suggests turning vehicle 400 to the side to avoid a first obstacle.

[0043] It should be understood that Figure 4Only some components of vehicle 400 are shown, and it should be understood that vehicle 400 also includes other components commonly found in vehicles (e.g., electric vehicles), such as motors, brakes, wheels, wheel controls, turn signals, windows, doors, etc.

[0044] Figure 5 This is a block diagram of the logical components of a system for asynchronous late fusion of measurement results according to an embodiment of this disclosure. In some embodiments of this disclosure, Figure 5 Each component or block may represent one or more modules or a set of instructions stored, for example, in memory 406 and executed by processor 404. Figure 5 Exemplary functions of each module and the data paths between them are also described. Here, the pre-filter measurement result module 500 can filter and process the output measurement result signal from the sensor. Module 500 can pre-filter the signal in any way, such as by performing image processing, noise reduction, etc. The filtered measurement result is then transmitted to the data association module 510, which determines whether the new measurement result is associated with an existing trajectory or a detected object. Once the association is completed and the object position is correctly determined, the measurement result is sent to the occlusion detection module 520 to determine the measurement result that corresponds to an occluded area or a partially occluded area. In some embodiments, occlusion can be determined independently from the perspective of each sensor, and different processing of objects can be generated depending on whether they are determined to be occluded or not. The following is combined with Figure 7 The occlusion detection and processing are further described.

[0045] The sensor measurements and the list of detected object trajectories are then transmitted to the state filter update module 530 and the likelihood filter 550. The state filter update module 530 determines state information from the received sensor measurements and may employ any of the late fusion methods described herein to determine state information for timely and late-arriving sensor measurements. The state filter update module 530 can determine any state information from the received sensor measurements using any method or process. For example, in some embodiments, the state filter update module 530 can determine position, velocity, direction, or orientation for any detected object or trajectory. In some embodiments, module 530 can determine quantities such as object size, shape, and extent. Module 530 can calculate any quantity or attribute of any object that can be determined from any one or more input sensor measurements. State information can be calculated in any suitable manner using any method (such as estimation via an extended Kalman filter method that determines the object state and associated covariance). State propagation can also be performed in any suitable manner, such as via a motion model.

[0046] Likelihood filter 550 updates the likelihood estimate of the trajectory (for an object), i.e., determines whether the received sensor measurements correspond to the actual physical object. These determinations can be performed in any suitable manner using any suitable method (such as Bayesian filtering, etc.). The possible objects are then transmitted with the output from state filter update module 530 to module 540 for attachment and transmission to trajectory management module 560. Trajectory management module 560 creates, destroys, and / or merges trajectories or detected objects to output those objects that are likely real objects along with their possible attributes, such as position, velocity, etc. Trajectory management can be performed in any suitable manner. For example, uncorrelated measurements or measurements that produce the detection of a new object can be used to create a new trajectory. Trajectory management module 560 can also delete trajectories or objects that are no longer detected, such as any new measurements that have not been received within a predetermined time period (e.g., 100 ms). Trajectory management module 560 can also merge trajectories. For example, if two different sensors each detect objects that are sufficiently close to each other in location, module 560 can treat the two objects as the same object. That is, detected objects that are determined to be sufficiently close to each other are identified as the same object, and their trajectories are merged accordingly to remove duplicates. Trajectory merging can be performed in any suitable manner, such as by employing mean-shift clustering. The output of module 560 can therefore contain an updated list of objects, which contains updated values ​​of the most probable objects and their attributes (such as position and velocity). This list of objects can be transmitted to other systems of vehicle 100 for other purposes, such as path planning, navigation, etc. The object list can also be returned to data association module 510, where the above process can be repeated to update the trajectory list using newly received sensor measurements. In this way, vehicle 100 can continuously and repeatedly detect surrounding objects for any purpose.

[0047] Figure 6 This is a flowchart illustrating an exemplary method for asynchronous late fusion of measurement results according to an embodiment of the present disclosure. In some embodiments, Figure 6The flowchart can be implemented by state filter update 530. The process of the embodiments of this disclosure can initially receive sensor measurement results, each corresponding to a specific time index (step 600), such as measurement results from the aforementioned vehicle 100 sensors CFR, ML, FVC, and / or any other sensors supported by vehicle 100. The processor 404 can then arrange these sensor measurement results in chronological order according to their time indices (step 610) and determine any desired state information from these measurement results. At some point, the processor 404 can receive a later sensor measurement result after the arrangement of previously arrived measurement results, wherein the later-arriving measurement result has a time index earlier than at least one other previously arrived measurement result (step 620). That is, a later-arriving measurement result can be obtained before another already arrived measurement result. In this case, the later-arriving sensor measurement result is inserted into the arrangement structure in chronological order according to its time index, such that the updated arrangement structure contains sensor measurement results arranged in chronological order according to their time indices, regardless of when the measurement result is received (step 630). Then, processor 404 can determine, at least in part, the probability of the presence of at least one object from the modified or updated layout (step 640). More specifically, the state information can be updated from later-arriving measurements and propagated forward to generate updated, most recent state information.

[0048] Thus, it can be observed that the method of the embodiments of this disclosure allows for the calculation of state information from sensor measurements (transmitted and received asynchronously from any sensor type in any order). Later-arriving measurements from any sensor are simply inserted into the correct time sequence, and the state information is modified and propagated forward accordingly. Sensors can be swapped for different types or variations thereof at different timings without reprogramming or altering the methods described herein. Therefore, state information can be calculated more reliably and in a more modular manner, allowing for sensor changes without excessive reprogramming of the system of this disclosure or other modifications thereof.

[0049] The implementation of this disclosure also envisions determining the occlusion area from the object being detected. Figure 7 This is a top view conceptually illustrating the determination of an obstruction area according to an embodiment of the present disclosure. Vehicle 700 can determine the position of another vehicle 710 that may be positioned ahead, as shown. The position of vehicle 710 can be determined from sensor measurements of vehicle 700, where later arriving measurements are used to modify the calculated position of vehicle 710 as described above. Vehicle 710 obstructs the field of view from vehicle 700, blocking the line of sight between the observer or sensor and vehicle 700 and creating an obstruction area 730 as shown. Objects entering the obstruction area 730, such as vehicle 720, may not be visible from vehicle 700, and therefore their positions may not be easily determined from them.

[0050] Once the position of vehicle 710 is determined, a ray can be extended from the position of the sensor used to determine that position, passing through the outermost edge of vehicle 710, to determine the occlusion region 730. In some embodiments of this disclosure, the occlusion region can be calculated independently for each sensor of vehicle 700, since each sensor may have a different position on vehicle 700.

[0051] Objects detected outside of region 730 can be processed differently by vehicle 700 as those detected within or entering the occlusion region 730. For example, likelihood filter 550 can stop updating the likelihood estimate for occluded object 720 because sensor data may be unreliable. Similarly, trajectory management module 560 can delete trajectories of objects entering occlusion region 730, or can stop merging any trajectories located within occlusion region 730 because the state of any occluded object 720 may be considered too unreliable or undetectable with sufficient accuracy.

[0052] For purposes of explanation, the foregoing description uses specific nomenclature to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that specific details are not required to practice the methods and systems of this disclosure. Therefore, the foregoing description of specific embodiments of this disclosure is presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed. In view of the foregoing teachings, many modifications and variations are possible. For example, embodiments of this disclosure can update state information based on any sensor measurements arriving at any time in any order. The embodiments were chosen and described in order to best explain the principles of this disclosure and its practical application, thereby enabling others skilled in the art to best utilize the methods and systems of this disclosure, as well as various embodiments with various modifications suitable for the particular intended use. Furthermore, different features of the various disclosed or otherwise embodiments may be mixed and matched or otherwise combined to form other embodiments contemplated by this disclosure.

Claims

1. A method for performing measurement result fusion, the method comprising: The processing circuitry is used to calculate the object's state information for each sensor measurement in a set of sensor measurements with corresponding time indices. After the calculation, a late sensor measurement result is received, wherein the time index of the late sensor measurement result is later in time than the first time index of the first sensor measurement result in the set of sensor measurement results but earlier than the second time index of the second sensor measurement result in the set of sensor measurement results; as well as Based on the previously calculated state information for the first sensor measurement result and the second sensor measurement result, the state information of the object for the second sensor measurement result is recalculated.

2. The method of claim 1, wherein the first time index and the second time index each correspond to a first epoch, and wherein the receiving further includes receiving the late sensor measurement result during a second epoch that occurs after the first epoch.

3. The method according to claim 1, wherein each sensor measurement result in the set of sensor measurement results has a corresponding time index, and wherein the method further comprises storing the set of sensor measurement results in a buffer in chronological order according to the corresponding time index.

4. The method of claim 3, further comprising storing the late sensor measurement result in the buffer between the first sensor measurement result and the second sensor measurement result.

5. The method according to claim 1, wherein the set of sensor measurement results and the late sensor measurement results are measurement results from sensors of the vehicle.

6. The method of claim 1, wherein the set of sensor measurements and the late sensor measurements are one or more of radio detection and ranging radar measurements, optical detection and ranging LiDAR measurements, or images.

7. The method according to claim 1, wherein the state information includes object fusion output.

8. The method of claim 7, wherein the object fusion output includes the possibility that the object exists.

9. The method of claim 1, further comprising determining an occlusion region for the object.

10. A system for performing measurement result fusion, the system comprising: Storage devices; and Processing circuit, the processing circuit being configured to: For each sensor measurement result in a set of sensor measurement results with a corresponding time index, calculate the state information of the object; After calculating the state information, a late sensor measurement result is received, wherein the time index of the late sensor measurement result is later in time than the first time index of the first sensor measurement result in the set of sensor measurement results but earlier than the second time index of the second sensor measurement result in the set of sensor measurement results. as well as Based on the previously calculated state information for the first sensor measurement result and the second sensor measurement result, the state information of the object for the second sensor measurement result is recalculated.

11. The system of claim 10, wherein the first time index and the second time index each correspond to a first epoch, and wherein the receiving further includes receiving the late sensor measurement during a second epoch occurring after the first epoch.

12. The system of claim 10, wherein each of the set of sensor measurement results has a corresponding time index, and wherein the processing circuit is further configured to store the set of sensor measurement results in a buffer of the storage device in chronological order of the corresponding time index.

13. The system of claim 12, wherein the processing circuitry is further configured to store the later sensor measurement result in the buffer between the first sensor measurement result and the second sensor measurement result.

14. The system of claim 10, wherein the set of sensor measurements and the late sensor measurements are measurements from sensors of the vehicle.

15. The system of claim 10, wherein the set of sensor measurements and the late sensor measurements are one or more of radio detection and ranging radar measurements, optical detection and ranging LiDAR measurements, or images.

16. The system of claim 10, wherein the state information includes object fusion output.

17. The system of claim 16, wherein the object fusion output includes the possibility that the object exists.

18. The system of claim 10, wherein the processing circuitry is further configured to determine an occlusion region for the object.

19. A vehicle, the vehicle comprising: Body; Processing circuitry, coupled to the vehicle body and configured to: For each sensor measurement result in a set of sensor measurement results with a corresponding time index, calculate the state information of the object; After calculating the state information, a late sensor measurement result is received, wherein the time index of the late sensor measurement result is later in time than the first time index of the first sensor measurement result in the set of sensor measurement results but earlier than the second time index of the second sensor measurement result in the set of sensor measurement results. as well as Based on the previously calculated state information for the first sensor measurement result and the second sensor measurement result, the state information of the object for the second sensor measurement result is recalculated.

20. The vehicle of claim 19, wherein the set of sensor measurements and the late sensor measurements are one or more of radio detection and ranging radar measurements, light detection and ranging LiDAR measurements, or images.

Images