Dynamic occupancy grid architecture

Abstract

Techniques are provided for utilizing a dynamic occupancy grid (DoG) to track objects proximate to an autonomous or semi-autonomous vehicle. An example method for generating a list of object trajectories in a vehicle includes obtaining sensor information from one or more sensors on the vehicle; determining a first object data set based at least in part on the sensor information and the object recognition process; generating a dynamic grid based at least in part on the sensor information to based on the environment proximate the vehicle; determining a second object data set based at least in part on the dynamic grid; and outputting an object trajectory list based on a fusion of the first object data set and the second object data set.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Patent Application No. 18 / 827,951, filed September 9, 2024, entitled “DYNAMIC OCCUPANCY GRID ARCHITECTURE,” and U.S. Provisional Application No. 63 / 592,596, filed October 24, 2023, entitled “DYNAMIC OCCUPANCY GRID ARCHITECTURE,” both of which have been assigned to the assignee of this application, and the entire contents of both are hereby incorporated herein by reference for all purposes. Background Technology

[0003] As industry increasingly deploys more sophisticated self-driving technologies, vehicles are becoming more intelligent, capable of operating with little or no human input, and thus semi-autonomous or autonomous. Autonomous and semi-autonomous vehicles may be able to detect information about their location and surrounding environment (e.g., using ultrasound, radar, lidar, SPS (Satellite Positioning System), and / or odometers, and / or one or more sensors such as accelerometers, cameras, etc.). Autonomous and semi-autonomous vehicles typically include a control system to interpret information about the vehicle's environment, thereby identifying hazards and determining a navigation path to follow.

[0004] Driver assistance systems can mitigate driving risks for drivers and / or other road users of self-driving vehicles (i.e., vehicles configured to perceive their environment). Driver assistance systems may include one or more active and / or passive devices that can be used to determine the environment of the self-driving vehicle and, for semi-autonomous vehicles, may inform the driver of situations that the driver can resolve. Driver assistance systems can be configured to control various aspects of driving safety and / or driver monitoring. For example, a driver assistance system may control the speed of the self-driving vehicle to maintain at least a desired distance (or time) between the self-driving vehicle and another vehicle (e.g., as part of an active cruise control system). Driver assistance systems may monitor the surrounding environment of the self-driving vehicle, for example, to maintain situational awareness of the self-driving vehicle. This situational awareness can be used to inform the driver of problems, such as another vehicle being in the driver's blind spot, another vehicle being on the collision path with the self-driving vehicle, etc. This situational awareness may include information about the self-driving vehicle (e.g., speed, position, heading) and / or other vehicles or objects (e.g., position, speed, heading, size, object type, etc.). Summary of the Invention

[0005] An example method for generating a list of object trajectories in a vehicle according to this disclosure includes: obtaining sensor information from one or more sensors on the vehicle; determining a first object dataset based at least in part on the sensor information and an object recognition process; generating a dynamic mesh based at least in part on the sensor information and the environment in proximity to the vehicle; determining a second object dataset based at least in part on the dynamic mesh; and outputting a list of object trajectories based on the fusion of the first and second object datasets.

[0006] An example apparatus according to this disclosure includes at least one memory, one or more sensors, and at least one processor communicatively coupled to the at least one memory and the one or more sensors, and configured to: obtain sensor information from the one or more sensors; determine a first object dataset based at least in part on the sensor information and an object recognition process; generate a dynamic mesh based at least in part on the sensor information; determine a second object dataset based at least in part on the dynamic mesh; and output a list of object trajectories based on the fusion of the first object dataset and the second object dataset.

[0007] The projects and / or technologies described herein can provide one or more of the following capabilities, as well as others not mentioned. Autonomous or semi-autonomous vehicles may include one or more sensors, such as cameras, radar, and lidar. Low-level perception operations can be performed on information acquired by the sensors. A dynamic occupancy grid can be generated based on input received from the sensors. Clusters of dynamic cells within the dynamic occupancy grid can be identified. The results of the low-level perception operations and the identified dynamic cell clusters can be fused to generate a list of object trajectories. The dynamic occupancy grid can be configured to generate a static object list. The object trajectory list and the static object list can be provided to a perception planning module within the vehicle. The fusion of low-level perception object detection results with dynamic grid detection techniques enables the detection of smaller objects or objects outside the training scope of the machine learning model used in the low-level perception operations. The bandwidth and / or complexity required to provide dynamic and static object information to the perception planning module can be reduced. Other capabilities can be provided, and not every specific embodiment of this disclosure is required to provide any, let alone all, of the capabilities discussed. Attached Figure Description

[0008] Figure 1 This is a top view of an example of a vehicle.

[0009] Figure 2 yes Figure 1 The self-driving vehicle shown can be a block diagram of the components of an example device.

[0010] Figure 3 This is a block diagram of the components of an example send / receive point.

[0011] Figure 4 This is a block diagram of the server components.

[0012] Figure 5 This is a block diagram of the example device.

[0013] Figure 6 This is a map illustrating an example geographical environment.

[0014] Figure 7 It is divided into grids Figure 6 The map shows the geographical environment.

[0015] Figure 8 Is with Figure 7 An example of the occupancy map corresponding to the grid shown.

[0016] Figure 9 This is a block diagram of an example method for dynamic grid clustering.

[0017] Figure 10 yes Figure 5 The diagram shows an example functional architecture block diagram of the environment building module of the device.

[0018] Figure 11 This is a block diagram of an example process for controlling autonomous vehicles.

[0019] Figure 12 This is a block diagram of an example functional architecture for an environment building module with occlusion mesh computing.

[0020] Figure 13 It is a block diagram of an example functional architecture for building modules that include external constraints.

[0021] Figure 14 This is a flowchart of a sample method for generating a list of object trajectories in a vehicle.

[0022] Figure 15 This is a flowchart of an example method for generating motion plans for autonomous vehicles. Detailed Implementation

[0023] This paper discusses techniques for using a dynamic occupancy grid (DoG) to track objects approaching autonomous or semi-autonomous vehicles. For example, measurements from multiple sensors, including one or more radars and cameras, can be obtained and used during object recognition to determine the dynamic grid. The object recognition process can be part of a low-level perception (LLP) module and can be based on a machine learning model. The dynamic grid may include a clustering process configured to detect dynamic objects. Objects detected by the LLP can be fused with dynamic objects detected via the dynamic grid to generate an object tracking list. Static obstacles can also be detected via the dynamic grid. The object tracking list and static obstacle information can be provided to a perception and planning module within the vehicle. In the example, the dynamic grid can be configured to identify occluded cells within the grid. Map data and remote sensor information can be used to assist in the detection of dynamic objects. In the example, the dynamic grid can utilize vehicle-to-everything (V2X) signaling to improve the detection of dynamic objects. However, other techniques can be used.

[0024] Specific aspects of the subject matter described in this disclosure can be implemented to achieve one or more of the following potential advantages: The fusion of LLP object detection results with dynamic mesh detection techniques enables the detection of smaller objects or other objects outside of LLP training. The bandwidth and / or complexity required to provide dynamic and static object information to the perception planning module can be reduced. Object tracking information can be compressed and / or simplified. Object detection machine learning models can be trained based on dynamic mesh object detection results. Object classification can be based on the fusion of LLP object detection and dynamic mesh clustering results. Static obstacles can also be classified. Remote sensors from other vehicles can utilize V2X signaling to improve dynamic mesh detection results. Other advantages may also be achieved.

[0025] refer to Figure 1The self-driving vehicle 100 includes a self-driving vehicle driver assistance system 110. The driver assistance system 110 may include multiple sensors of different types mounted at appropriate locations on the self-driving vehicle 100. For example, the system 110 may include: a pair of diverging and outward-pointing radar sensors 121 mounted at corresponding front corners of the vehicle 100; a similar pair of diverging and outward-pointing radar sensors 122 mounted at corresponding rear corners of the vehicle 100; a forward-pointing LRR sensor 123 (long-range radar) centrally mounted at the front of the vehicle 100; and a pair of generally forward-pointing optical sensors 124 (cameras) forming part of an SVS 126 (stereo vision system), which may be mounted, for example, in the area of ​​the upper edge of the windshield 128 of the vehicle 100. Each of the sensors 121, 122 may include LRR and / or SRR (short-range radar). Various sensors 121 to 124 are operatively connected to a central electronic control system, which is typically provided in the form of an ECU 140 (Electronic Control Unit) installed in a convenient location within the vehicle 100. In the particular arrangement illustrated, the front sensor 121 and the rear sensor 122 are connected to the ECU 140 via one or more conventional Controller Area Network (CAN) buses 150, and the sensors of the LRR sensor 123 and the SVS 126 are connected to the ECU 140 via a serial bus 160 (e.g., a faster FlexRay serial bus).

[0026] Together, and under the control of ECU 140, various sensors 121 to 124 can be used to provide a variety of different types of driver assistance functions. For example, sensors 121 to 124 and ECU 140 can provide blind spot monitoring, adaptive cruise control, collision prevention assist, lane departure protection, and / or rear collision mitigation.

[0027] The CAN bus 150 can be regarded by the ECU 140 as a sensor that provides self-vehicle parameters to the ECU 140. For example, a GPS module can also be connected to the ECU 140 as a sensor to provide geographic location parameters to the ECU 140.

[0028] Also refer to Figure 2Device 200 (which may be a mobile device such as User Equipment (UE) or a Vehicle Equipment (VUE)) includes a computing platform containing processor 210, a memory 211 containing software (SW) 212, one or more sensors 213, a transceiver interface 214 for transceivers 215 (which includes a wireless transceiver 240 and a wired transceiver 250), a user interface 216, a satellite positioning system (SPS) receiver 217, a camera 218, and a positioning device (PD) 219. The term “user equipment” or UE (or a variant thereof) is not specific to or otherwise limited to any particular radio access technology (RAT) unless otherwise indicated. Processor 210, memory 211, sensors 213, transceiver interface 214, user interface 216, SPS receiver 217, camera 218, and positioning device 219 may be communicatively coupled to each other via bus 220 (which may be configured for, for example, optical and / or electrical communication). One or more of the devices shown in the apparatus (e.g., camera 218, positioning device 219, and / or one or more sensors in sensor 213, etc.) may be omitted from device 200. Processor 210 may include one or more hardware devices, such as a central processing unit (CPU), microcontroller, application-specific integrated circuit (ASIC), etc. Processor 210 may include multiple processors, including a general-purpose / application processor 230, a digital signal processor (DSP) 231, a modem processor 232, a video processor 233, and / or a sensor processor 234. One or more of processors 230 to 234 may include multiple devices (e.g., multiple processors). For example, sensor processor 234 may include processors for, for example, RF (radio frequency) sensing (where one or more transmitted (cellular) wireless signals and reflections are used to identify, map, and / or track objects) and / or ultrasound, etc. Modem processor 232 may support dual SIM / dual connectivity (or even more SIMs). For example, a SIM (subscriber identity module or subscriber identification module) may be used by an original equipment manufacturer (OEM), and another SIM may be used by the end user of device 200 to obtain connectivity. Memory 211 may be a non-transitory processor-readable storage medium, including random access memory (RAM), flash memory, disk storage, and / or read-only memory (ROM). Memory 211 may store software 212, which may be processor-readable, processor-executable software code containing instructions that can be configured to cause processor 210 to perform the various functions described herein when executed. Alternatively, software 212 may not be directly executable by processor 210, but may be configured to cause processor 210 to perform these functions, for example, when compiled and executed. The description herein refers to processor 210 performing functions, but includes other specific implementations, such as specific implementations of instructions by processor 210 executing software and / or firmware.The description herein may be abbreviated as processor 210 performing functions, referred to as one or more processors 230 to 234 performing functions. The description herein may be abbreviated as device 200 performing functions, referred to as one or more suitable components of device 200 performing the function. In addition to and / or instead of memory 211, processor 210 may include memory with stored instructions. The functionality of processor 210 is discussed more fully below.

[0029] Figure 2 The configuration of device 200 shown is exemplary and not intended to limit this disclosure (including the claims), and other configurations may be used. For example, an example configuration of the UE may include one or more of processors 230 to 234 in processor 210, memory 211, and wireless transceiver 240. Other example configurations may include one or more of processors 230 to 234 in processor 210, memory 211, wireless transceiver, and one or more of the following devices: sensor 213, user interface 216, SPS receiver 217, camera 218, PD 219, and / or wired transceiver.

[0030] Device 200 may include a modem processor 232 capable of performing baseband processing on signals received and downconverted by transceiver 215 and / or SPS receiver 217. Modem processor 232 may also perform baseband processing on signals to be upconverted for transmission by transceiver 215. Alternatively, baseband processing may be performed by general-purpose / application processor 230 and / or DSP 231. However, other configurations may be used to perform baseband processing.

[0031] Device 200 may include sensor 213, which may include one or more sensors of various types, such as one or more inertial sensors, one or more magnetometers, one or more environmental sensors, one or more optical sensors, one or more weight sensors, and / or one or more radio frequency (RF) sensors. An inertial measurement unit (IMU) may include, for example, one or more accelerometers (e.g., collectively responding to acceleration of device 200 in three dimensions) and / or one or more gyroscopes (e.g., three-dimensional gyroscopes). Sensor 213 may include one or more magnetometers (e.g., three-dimensional magnetometers) to determine orientation (e.g., relative to magnetic north and / or true north), which can be used for any of a variety of purposes (e.g., to support one or more compass applications). Environmental sensors may include, for example, one or more temperature sensors, one or more barometric pressure sensors, one or more ambient light sensors, one or more camera imagers, and / or one or more microphones. Sensor 213 may generate analog and / or digital signals, indications of which may be stored in memory 211 and processed by DSP 231 and / or general-purpose / application processor 230 to support one or more applications (e.g., applications involving positioning and / or navigation operations).

[0032] Sensor 213 can be used for relative position measurement, relative position determination, motion determination, etc. Information detected by sensor 213 can be used for motion detection, relative displacement, dead reckoning, sensor-based position determination, and / or sensor-assisted position determination. Sensor 213 can be used to determine whether device 200 is stationary or mobile and / or whether certain useful information related to the mobility of device 200 needs to be reported to the LMF (Location Management Function), for example. For example, based on information obtained / measured by sensor 213, device 200 can notify / report to the LMF that device 200 has detected movement or that device 200 has moved, and can report relative displacement / distance (e.g., via dead reckoning implemented by sensor 213, or sensor-based position determination, or sensor-assisted position determination). In another example, for relative positioning information, the sensor / IMU can be used to determine the angle and / or orientation of another object (e.g., another device) relative to device 200.

[0033] The IMU can be configured to provide measurements of the direction and / or velocity of motion of device 200, which can be used for relative position determination. For example, one or more accelerometers and / or one or more gyroscopes of the IMU can detect the linear acceleration and rotational velocity of device 200, respectively. The linear acceleration and rotational velocity measurements of device 200 can be integrated over time to determine the instantaneous direction of motion and displacement of device 200. The instantaneous direction of motion and displacement can be integrated to track the position of device 200. For example, a reference position of device 200 at a certain moment can be determined, for example, using SPS receiver 217 (and / or by some other means), and measurements acquired from the accelerometers and gyroscopes after that moment can be used for dead reckoning to determine the current position of device 200 based on the movement (direction and distance) of device 200 relative to that reference position.

[0034] A magnetometer can determine the strength of a magnetic field in different directions, which can be used to determine the orientation of device 200. For example, this orientation can be used to provide a digital compass for device 200. The magnetometer may include a two-dimensional magnetometer configured to detect and provide an indication of the magnetic field strength in two orthogonal dimensions. The magnetometer may also include a three-dimensional magnetometer configured to detect and provide an indication of the magnetic field strength in three orthogonal dimensions. The magnetometer may provide components for sensing the magnetic field and, for example, providing an indication of the magnetic field to processor 210.

[0035] Transceiver 215 may include a wireless transceiver 240 and a wired transceiver 250 configured to communicate with other devices via wireless and wired connections, respectively. For example, wireless transceiver 240 may include a wireless transmitter 242 and a wireless receiver 244 coupled to antenna 246 for transmitting (e.g., on one or more uplink channels and / or one or more sidelink channels) and / or receiving (e.g., on one or more downlink channels and / or one or more sidelink channels) wireless signals 248 and converting signals from wireless signals 248 into guided (e.g., wired electrical and / or optical) signals and from guided (e.g., wired electrical and / or optical) signals into wireless signals 248. Wireless transmitter 242 includes suitable components (e.g., a power amplifier and a digital-to-analog converter). Wireless receiver 244 includes suitable components (e.g., one or more amplifiers, one or more frequency filters, and an analog-to-digital converter). Wireless transmitter 242 may include multiple transmitters, which may be discrete components or combined / integrated components, and / or wireless receiver 244 may include multiple receivers, which may be discrete components or combined / integrated components. Wireless transceiver 240 may be configured to transmit signals according to a variety of radio access technologies (RATs) (e.g., with TRP and / or one or more other devices), such as 5G New Radio (NR), GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), AMPS (Advanced Mobile Telephone Systems), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), LTE Direct (LTE-D), 3GPP LTE-V2X (PC5), IEEE 802.11 (including IEEE 802.11p), and WiFi. ® Short-range wireless communication technology, WiFi ® Direct connection (WiFi) ® -D), Bluetooth ® Short-range wireless communication technology, Zigbee ®Short-range wireless communication technologies, etc. The new radio can use millimeter-wave frequencies and / or frequencies below 6 GHz. Wired transceiver 250 may include a wired transmitter 252 and a wired receiver 254 configured for wired communication, for example, a network interface that can be used to communicate with NG-RAN (Next Generation Radio Access Network) to transmit communications to and receive communications from NG-RAN. Wired transmitter 252 may include multiple transmitters that can be discrete components or combined / integrated components, and / or wired receiver 254 may include multiple receivers that can be discrete components or combined / integrated components. Wired transceiver 250 may be configured, for example, for optical and / or electrical communication. Transceiver 215 may be communicatively coupled to transceiver interface 214, for example, via optical and / or electrical connections. Transceiver interface 214 may be at least partially integrated with transceiver 215. The wireless transmitter 242, the wireless receiver 244, and / or the antenna 246 may each include multiple transmitters, multiple receivers, and / or multiple antennas for transmitting and / or receiving appropriate signals, respectively.

[0036] User interface 216 may include one or more of a number of devices, such as speakers, microphones, display devices, vibration devices, keyboards, touchscreens, etc. User interface 216 may include more than one of these devices. User interface 216 may be configured to enable a user to interact with one or more applications hosted by device 200. For example, user interface 216 may store indications of analog and / or digital signals in memory 211 in response to actions from the user, for processing by DSP 231 and / or general-purpose / application processor 230. Similarly, applications hosted on device 200 may store indications of analog and / or digital signals in memory 211 to present output signals to the user. User interface 216 may include audio input / output (I / O) devices, including, for example, speakers, microphones, digital-to-analog circuitry, analog-to-digital circuitry, amplifiers, and / or gain control circuitry (including more than one of these devices). Other configurations of the audio I / O devices may be used. Additionally or alternatively, the user interface 216 may include one or more touch sensors that respond to touch and / or pressure on, for example, the keyboard and / or touchscreen of the user interface 216.

[0037] SPS receiver 217 (e.g., a Global Positioning System (GPS) receiver) may be able to receive and acquire SPS signal 260 via SPS antenna 262. SPS antenna 262 is configured to convert SPS signal 260 from a wireless signal to a guiding signal (e.g., a wired signal or an optical signal) and may be integrated with antenna 246. SPS receiver 217 may be configured to process the acquired SPS signal 260 fully or partially for estimating the location of device 200. For example, SPS receiver 217 may be configured to determine the location of device 200 by performing trilateration using SPS signal 260. SPS receiver 217 may be used in conjunction with general-purpose / application processor 230, memory 211, DSP 231, and / or one or more dedicated processors (not shown) to process the acquired SPS signal fully or partially and / or calculate the estimated location of device 200. Memory 211 may store indications (e.g., measurements) of SPS signal 260 and / or other signals (e.g., signals acquired from wireless transceiver 240) for use in performing positioning operations. General-purpose / application processor 230, DSP 231, and / or one or more dedicated processors, and / or memory 211 may provide or support a position engine for use in processing measurements to estimate the position of device 200.

[0038] Device 200 may include a camera 218 for capturing still or moving images. Camera 218 may include, for example, an imaging sensor (e.g., a charge-coupled device or CMOS (complementary metal-oxide-semiconductor) imager), lenses, analog-to-digital circuitry, frame buffers, etc. Additional processing, conditioning, encoding, and / or compression of signals representing the captured images may be performed by general-purpose / application processor 230 and / or DSP 231. Additionally or alternatively, video processor 233 may perform conditioning, encoding, compression, and / or manipulation of signals representing the captured images. Video processor 233 may decode / decompress stored image data for presentation on a display device (not shown), for example, user interface 216.

[0039] Location device (PD) 219 may be configured to determine the location of device 200, the movement of device 200, and / or the relative location of device 200, and / or time. For example, PD 219 may communicate with SPS receiver 217 and / or include part or all of the SPS receiver. PD 219 may, where appropriate, operate in conjunction with processor 210 and memory 211 to perform at least a portion of one or more location methods, although the description herein may refer to PD 219 being configured to perform according to a location method or the PD performing according to a location method. PD 219 may additionally or alternatively be configured to use terrestrial signals (e.g., at least some of the radio signals in radio signals 248) for trilateration, assisted acquisition, and use of SPS signal 260, or both, to determine the location of device 200. PD 219 may be configured to determine the location of device 200 based on the coverage area of ​​a serving base station and / or another technology (such as E-CID). PD 219 may be configured to determine the location of device 200 using one or more images from camera 218 and image recognition combined with the known location of landmarks (e.g., natural landmarks such as mountains and / or man-made landmarks such as buildings, bridges, streets, etc.). PD 219 may be configured to determine the location of device 200 using one or more other technologies (e.g., relying on the UE's self-reported location (e.g., part of the UE's positioning beacon)), and may use a combination of these technologies (e.g., SPS and terrestrial positioning signals) to determine the location of device 200. PD 219 may include one or more sensors among sensors 213 (e.g., gyroscopes, accelerometers, magnetometers, etc.) that can sense the orientation and / or motion of device 200 and provide an indication of such orientation and / or motion, and processor 210 (e.g., general-purpose / application processor 230 and / or DSP 231) may be configured to use this indication to determine the motion of device 200 (e.g., velocity vector and / or acceleration vector). PD 219 can be configured to provide an indication of uncertainty and / or error in the determined positioning and / or motion. The functionality of PD 219 can be provided in a variety of ways and / or configurations, such as by a general-purpose / application processor 230, transceiver 215, SPS receiver 217 and / or another component of device 200, and can be provided by hardware, software, firmware or various combinations thereof.

[0040] Also refer to Figure 3Examples of TRP 300 (such as gNB (General Node B) and / or ng-eNB (Next Generation Evolved Node B) base stations) may include a computing platform containing processor 310, a memory 311 containing software (SW) 312, and a transceiver 315. Even when cited in the singular, processor 310 may include one or more processors, transceiver 315 may include one or more transceivers (e.g., one or more transmitters and / or one or more receivers), and / or memory 311 may include one or more memories. Processor 310, memory 311, and transceiver 315 may be communicatively coupled to each other via bus 320 (which may be configured for, for example, optical and / or electrical communications). One or more devices in the illustrated apparatus (e.g., wireless transceivers) may be omitted from TRP 300. Processor 310 may include one or more hardware devices, such as a central processing unit (CPU), microcontroller, application-specific integrated circuit (ASIC), etc. Processor 310 may include multiple processors (e.g., including general-purpose / application processors, DSPs, modem processors, video processors, and / or sensor processors, such as... Figure 2 (As shown). Memory 311 may be a non-transitory storage medium including random access memory (RAM), flash memory, disk storage, and / or read-only memory (ROM). Memory 311 may store software 312, which may be processor-readable, processor-executable software code containing instructions configured to cause processor 310 to perform the various functions described herein when executed. Alternatively, software 312 may not be directly executable by processor 310, but may be configured to cause processor 310 to perform these functions, for example, when compiled and executed.

[0041] The description herein refers to the functionality performed by processor 310, but this includes other specific implementations, such as specific implementations of software and / or firmware performed by processor 310. The description herein may refer to the functionality performed by processor 310 as an abbreviation for the functionality performed by one or more processors included in processor 310. The description herein may refer to the functionality performed by TRP 300 as an abbreviation for the functionality performed by one or more suitable components of TRP 300 (e.g., processor 310 and memory 311). In addition to and / or instead of memory 311, processor 310 may include memory with stored instructions. The functionality of processor 310 is discussed more fully below.

[0042] Transceiver 315 may include a wireless transceiver 340 and / or a wired transceiver 350 configured to communicate with other devices via wireless and wired connections, respectively. For example, wireless transceiver 340 may include a wireless transmitter 342 and a wireless receiver 344 coupled to one or more antennas 346 for transmitting (e.g., on one or more uplink channels and / or one or more downlink channels) and / or receiving (e.g., on one or more downlink channels and / or one or more uplink channels) wireless signals 348 and converting signals from wireless signals 348 into guided (e.g., wired electrical and / or optical) signals and from guided (e.g., wired electrical and / or optical) signals into wireless signals 348. Therefore, wireless transmitter 342 may include multiple transmitters that may be discrete components or combined / integrated components, and / or wireless receiver 344 may include multiple receivers that may be discrete components or combined / integrated components. The wireless transceiver 340 can be configured to transmit signals according to a variety of radio access technologies (RATs) (e.g., with device 200, one or more other UEs, and / or one or more other devices), such as 5G New Radio (NR), GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), AMPS (Advanced Mobile Telephone Systems), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), LTE Direct (LTE-D), 3GPP LTE-V2X (PC5), IEEE 802.11 (including IEEE 802.11p), and WiFi. ® Short-range wireless communication technology, WiFi ® Direct connection (WiFi) ® -D), Bluetooth ® Short-range wireless communication technology, Zigbee ® Short-range wireless communication technologies, etc. The wired transceiver 350 may include a wired transmitter 352 and a wired receiver 354 configured for wired communication, for example, a network interface used to communicate with NG-RAN to transmit and receive communications to, for example, LMF and / or one or more other network entities. The wired transmitter 352 may include multiple transmitters that can be discrete components or combined / integrated components, and / or the wired receiver 354 may include multiple receivers that can be discrete components or combined / integrated components. The wired transceiver 350 may be configured, for example, for optical communication and / or electrical communication.

[0043] Figure 3The configuration of TRP 300 shown is illustrative and not intended to limit this disclosure (including the claims), and other configurations may be used. For example, the description herein discusses that TRP 300 may be configured to perform several functions or that the TRP performs several functions, but one or more of these functions may be performed by LMF and / or device 200 (i.e., LMF and / or device 200 may be configured to perform one or more of these functions).

[0044] Also refer to Figure 4 Server 400 (LMF is an example thereof) may include a computing platform containing processor 410, memory 411 containing software (SW) 412, and transceiver 415. Even when cited in the singular, processor 410 may include one or more processors, transceiver 415 may include one or more transceivers (e.g., one or more transmitters and / or one or more receivers), and / or memory 411 may include one or more memories. Processor 410, memory 411, and transceiver 415 may be communicatively coupled to each other via bus 420 (which may be configured for, for example, optical communication and / or electrical communication). One or more devices in the illustrated apparatus (e.g., wireless transceivers) may be omitted from server 400. Processor 410 may include one or more hardware devices, such as a central processing unit (CPU), microcontroller, application-specific integrated circuit (ASIC), etc. Processor 410 may include multiple processors (e.g., including general-purpose / application processors, DSPs, modem processors, video processors, and / or sensor processors, such as... Figure 2 (As shown). Memory 411 may be a non-transitory storage medium including random access memory (RAM), flash memory, disk storage, and / or read-only memory (ROM). Memory 411 may store software 412, which may be processor-readable, processor-executable software code containing instructions configured to cause processor 410 to perform the various functions described herein when executed. Alternatively, software 412 may not be directly executable by processor 410, but may be configured to cause processor 410 to perform these functions, for example, when compiled and executed. The description herein may refer to processor 410 performing functions, but this includes other specific implementations, such as specific implementations of processor 410 performing software and / or firmware. The description herein may refer to the function performed by processor 410 as an abbreviation for one or more processors included in processor 410 performing functions. The description herein may refer to the function performed by server 400 as an abbreviation for one or more suitable components of server 400 performing functions. In addition to and / or instead of memory 411, processor 410 may include memory with stored instructions. The functionality of the processor 410 will be discussed more comprehensively below.

[0045] Transceiver 415 may include a wireless transceiver 440 and / or a wired transceiver 450 configured to communicate with other devices via wireless and wired connections, respectively. For example, wireless transceiver 440 may include a wireless transmitter 442 and a wireless receiver 444 coupled to one or more antennas 446 for transmitting (e.g., on one or more downlink channels) and / or receiving (e.g., on one or more uplink channels) wireless signals 448 and converting signals from wireless signals 448 to guided (e.g., wired electrical and / or optical) signals and from guided (e.g., wired electrical and / or optical) signals to wireless signals 448. Therefore, wireless transmitter 442 may include multiple transmitters that may be discrete components or combined / integrated components, and / or wireless receiver 444 may include multiple receivers that may be discrete components or combined / integrated components. The wireless transceiver 440 can be configured to transmit signals according to a variety of radio access technologies (RATs) (e.g., with device 200, one or more other UEs, and / or one or more other devices), such as 5G New Radio (NR), GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), AMPS (Advanced Mobile Telephone Systems), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), LTE Direct (LTE-D), 3GPP LTE-V2X (PC5), IEEE 802.11 (including IEEE 802.11p), and WiFi. ® Short-range wireless communication technology, WiFi ® Direct connection (WiFi) ® -D), Bluetooth ® Short-range wireless communication technology, Zigbee ® Short-range wireless communication technologies, etc. Wired transceiver 450 may include a wired transmitter 452 and a wired receiver 454 configured for wired communication, for example, a network interface used to communicate with NG-RAN to transmit and receive communications to, for example, LMF 300 and / or one or more other network entities. Wired transmitter 452 may include multiple transmitters that can be discrete components or combined / integrated components, and / or wired receiver 454 may include multiple receivers that can be discrete components or combined / integrated components. Wired transceiver 450 may be configured, for example, for optical communication and / or electrical communication.

[0046] The description herein may refer to the functionality performed by processor 410, but this includes other specific implementations, such as the specific implementation of firmware and / or software (stored in memory 411) performed by processor 410. The description herein may refer to the functionality performed by server 400 as an abbreviation for the functionality performed by one or more appropriate components of server 400 (e.g., processor 410 and memory 411).

[0047] Figure 4 The configuration of server 400 shown is exemplary and not intended to limit this disclosure (including the claims), and other configurations may be used. For example, wireless transceiver 440 may be omitted. Furthermore or alternatively, this specification discusses server 400 being configured to perform certain functions, or the server performing certain functions, but one or more of these functions may be performed by TRP 300 and / or device 200 (i.e., TRP 300 and / or device 200 may be configured to perform one or more of these functions).

[0048] refer to Figure 5 Device 500 includes a processor 510, a transceiver 520, a memory 530, and a sensor 540 communicatively coupled to each other via a bus 550. Even when cited in the singular, processor 510 may include one or more processors, transceiver 520 may include one or more transceivers (e.g., one or more transmitters and / or one or more receivers), and memory 530 may include one or more memories. Device 500 may take any of a variety of forms, such as mobile devices (e.g., vehicle UE (VUE)). Device 500 may include... Figure 5 The components shown may include one or more other components, such as Figure 2 Any of the components shown makes device 200 an example of device 500. For example, processor 510 may include one or more components of processor 210. Transceiver 520 may include one or more components of transceiver 215, such as wireless transmitter 242 and antenna 246, or wireless receiver 244 and antenna 246, or wireless transmitter 242, wireless receiver 244 and antenna 246. Additionally or alternatively, transceiver 520 may include wired transmitter 252 and / or wired receiver 254. Memory 530 may be configured similarly to memory 211, for example including software having processor-readable instructions configured to cause processor 510 to perform functions. Sensor 540 includes one or more radar sensors 542 and one or more cameras 544. Sensor 540 may include other sensors, such as lidar, Hall effect sensors, ultrasonic sensors, or other sensors configured to assist in the operation of a vehicle.

[0049] The description herein may refer to processor 510 performing functions, but this includes other specific implementations, such as specific implementations of processor 510 performing firmware and / or software (stored in memory 530). The description herein may refer to the device 500 performing functions as an abbreviation for one or more suitable components of device 500 (e.g., processor 510 and memory 530) performing functions. Processor 510 (possibly in conjunction with memory 530 and, where appropriate, transceiver 520) may include occupying grid unit 560 (which may include ADAS (Advanced Driver Assistance Systems) for VUE). Occupying grid unit 560 is discussed further herein, and the description herein may refer to occupying grid unit 560 performing one or more functions, and / or may generally refer to processor 510 or generally refer to device 500 as performing the functions of occupying grid unit 560, wherein device 500 is configured to perform these functions.

[0050] One or more functions performed by device 500 (e.g., occupied grid cell 560) may be performed by another entity. For example, sensor measurements (e.g., radar measurements, camera measurements (e.g., pixels, images)) and / or processed sensor measurements (e.g., camera images converted into bird's-eye view images) may be provided to another entity, such as server 400, and that other entity may perform one or more of the functions discussed herein with respect to occupied grid cell 560 (e.g., using machine learning to determine the currently occupied grid, and / or applying observation models, analyzing measurements from different sensors to determine the currently occupied grid, etc.).

[0051] Also refer to Figure 6The geographic environment 600 (in this example, the driving environment) includes multiple mobile wireless communication devices (here, vehicles 601, 602, 603, 604, 605, 606, 607, 608, 609), buildings 610, RSUs 612 (roadside units), and street signs 620 (e.g., stop signs). RSUs 612 may be configured similarly to TRPs 300, but may have less functionality and / or shorter range than TRPs 300 (e.g., base station-based TRPs). One or more vehicles among vehicles 601 to 609 may be configured to perform autonomous driving. A vehicle considering its perspective (e.g., for environmental assessment, autonomous driving, etc.) may be referred to as an observer vehicle or a self-vehicle. Self-vehicles such as vehicle 601 may assess the area around the self-vehicle for one or more desired purposes (e.g., to facilitate autonomous driving). Vehicle 601 may be an example of device 500. The vehicle 601 can divide the area around the vehicle into multiple sub-areas and assess whether an object occupies each sub-area, and if so, determine one or more characteristics of the object (e.g., size, shape (e.g., dimensions (possibly including height)), speed (speed and direction), object type or category (bicycle, car, truck, etc.) etc.).

[0052] Also refer to Figure 7 and Figure 8In this example, a region 700 spanning a portion of environment 600 can be evaluated to determine an occupancy grid 800 (also called an occupancy map). This occupancy grid indicates multiple probabilities for each cell of grid 800 regarding whether the cell is occupied or vacant, and whether the occupancy is static or dynamic. For example, region 700 may be divided into grids with sub-regions 710 (which may be called occupancy grids). These sub-regions may be similar (e.g., identical) in size and shape, or may have two or more sizes and / or shapes (e.g., where sub-regions are smaller near the vehicle (e.g., vehicle 601) and larger further away from the vehicle, and / or where sub-regions near the vehicle have a different shape than sub-regions further away from the vehicle). Region 700 and grid 800 can be of regular shapes (e.g., rectangles, triangles, hexagons, octagons, etc.) and / or, for convenience (e.g., to simplify calculations), can be divided into sub-regions of the same or regular shape, but other shapes of regions / grids (e.g., irregular shapes) and / or sub-regions (e.g., irregular shapes, multiple different regular shapes, or a combination of one or more irregular shapes and one or more regular shapes) can be used. For example, sub-region 710 can have a rectangular (e.g., square) shape. Region 700 can be any of a variety of sizes and can have any of a variety of granularity of sub-regions. For example, region 700 can be a rectangle (e.g., a square) with approximately 100m on each side. As another example, although region 700 is shown as having a sub-region 710 with a square of approximately 1m on each side, other sizes of sub-regions can be used, including much smaller sub-regions. For example, a square sub-region with approximately 25cm on each side can be used. In this example, region 700 is divided into M rows (here, parallel to...). Figure 8 The indicated x-axis has 24 rows, each with N columns (here, parallel to...). Figure 8 (The y-axis has 23 columns shown). As another example, the grid can include a 512×512 sub-region array. Other specific implementations that occupy the grid can still be used.

[0053] Each subregion in subregion 710 may correspond to a corresponding cell 810 in an occupancy map, and information about what (if any) occupies each subregion in subregion 710 and whether the occupancy object is static or dynamic may be obtained, so that cells 810 of occupancy grid 800 are filled with the probability that the cell is occupied (O) or free (F) (i.e., unoccupied) and the probability that the object occupying the cell is at least partially static (S) or dynamic (D). Each probability may be a floating-point value. Information about what (if any) occupies each subregion in subregion 710 may be obtained from a variety of sources. For example, occupancy information may be obtained from sensor measurements from sensor 540 of device 500. As another example, occupancy information may be obtained by one or more other devices and communicated to device 500. For example, one or more vehicles among vehicles 602 to 609 may communicate occupancy information to vehicle 601, for example, via C-V2X communication. As another example, RSU 612 may (e.g., from one or more sensors of RSU 612 and / or from communication with one or more vehicles of vehicles 602 to 609 and / or one or more other devices) collect occupancy information and communicate the collected information to vehicle 601, for example, directly and / or through one or more network entities (e.g., TRP).

[0054] like Figure 8 As shown, each cell in cell 810 may include a set 820 of occupancy information, which indicates a dynamic probability 821 (P). D ), static probability 822 (P) S ), free probability 823 (P) F ), occupancy probability 824 (P) P The probability 821 indicates the probability that an object (if any) in the corresponding sub-region 710 is dynamic. The probability 822 indicates the probability that an object (if any) in the corresponding sub-region 710 is static. The probability 823 indicates the probability that no object exists in the corresponding sub-region 710. The probability 824 indicates the probability that an object exists in (any part of) the corresponding sub-region 710. Each cell in cell 810 may include the corresponding probabilities 821 to 824 corresponding to whether cell 810 is dynamic, static, non-existent, or has an object, where the sum of the probabilities is 1. Figure 8 In the example shown, for the sake of graph simplicity and readability of occupied grid 800, no cells in occupied grid 800 are marked as being more likely to be free (empty) than occupied cells. Similarly... Figure 8As shown, cells that are more likely to be occupied than idle and more likely to be occupied by dynamic objects than static objects are marked with "D", and cells that are more likely to be occupied than idle and more likely to be occupied by static objects than dynamic objects are marked with "S". A self-driving vehicle may not be able to determine whether a cell is occupied (e.g., behind the visible surface of an object and indistinguishable based on the observed object (e.g., if the size and shape of the detected object are unknown)), and such cells can be marked as unoccupied.

[0055] Constructing a dynamic occupancy grid (an occupancy grid with dynamic occupancy types) can be helpful, or even necessary, for understanding the environment of the device (e.g., environment 600) to facilitate or even enable further processing. For example, a dynamic occupancy grid can be useful for predicting occupancy, motion planning, etc. A dynamic occupancy grid may include one or more cells of static occupancy type and / or one or more cells of dynamic occupancy type at any given time. Dynamic objects can be represented as a collection of one or more velocity vectors. For example, an occupancy grid cell may make some or all of the occupancy probabilities dynamic, and within the dynamic occupancy probabilities, there may be multiple (e.g., four) velocity vectors, each with a corresponding probability, which together sum to the dynamic occupancy probability of that cell 810. The dynamic occupancy grid can be obtained, for example, by occupancy grid cell 560 by processing information from multiple sensors (such as those from a radar system, e.g., in sensor 540). Adding data from one or more cameras to determine the dynamic occupancy grid can provide significant improvements to the grid, such as the accuracy of probabilities and / or velocities in the grid cells.

[0056] Also refer to Figure 9The dynamic grid clustering method 900 includes the stages shown. However, method 900 is illustrative and not limiting. Method 900 can be modified, for example, by adding, removing, rearranging, combining, performing one or more stages concurrently, and / or splitting one or more stages into multiple stages. At stage 910, an intermittent interruption 901 occurs (e.g., at a periodic interval 905, such as every 40 ms in this example), and is checked by processor 510 to determine whether sensor data (e.g., vehicle positioning information such as speed) is available. In this example, a check corresponding to intermittent interruption 906 is performed. If sensor data is available, the internal attitude, velocity, and yaw rate variables are updated based on the sensor data. At stage 920, an extraction fusion interval function is executed, where sensor data is collected over time. At stage 930, processor 510 determines whether one or more rules corresponding to success are met, such as at least a threshold number of radar measurements have been performed within a specified time interval. If successful (the rules have been met), method 900 proceeds to stage 940; otherwise, it returns to stage 910 to check the data for updating. At stage 940, a measurement grid is determined for each sensor for which sensor data is available. The values ​​of the grid points for each sensor (e.g., radar, camera, etc.) can be determined. For example, an M×N array (e.g., a 512×512 array) for each sensor can be determined and the arrays combined into a single M×N array of grid points. At stage 950, a dynamic grid update is performed. Processor 510 can determine a predictive grid (e.g., using a particle filter) and can fuse the measurement grid from stage 940 with the predictive grid. At stage 960, a clustering function is performed; for example, processor 510 identifies and groups grid cells with similar velocities and orientations (i.e., cells that are close to each other with similar velocities). At stage 970, a particle cluster ID update function is performed to track the cluster IDs within particles (from stage 960).

[0057] Also refer to Figure 10The functional architecture of the environment building module 1000 of device 500 includes functional blocks, including LLP 1010 (Low-Level Perception Functional Block), object tracker functional block 1020, dynamic mesh functional block 1030, clustering functional block 1040, and static extraction functional block 1050. LLP 1010 and dynamic mesh functional block 1030 can receive input data 1060 from radar 542 of sensor 540, camera 544 of sensor 540, and the attitude of device 500 (e.g., orientation of the vehicle relative to a reference axis, e.g., yaw). LLP 1010 can be configured to apply machine learning to at least some of the input data 1060 to identify dynamic objects (e.g., vehicles) corresponding to the environment of device 500 (e.g., environment 600), and output indications of the identified objects to object tracker 1020. Dynamic mesh function block 1030 can be configured to determine a dynamic mesh, including, for example, occupancy probability, static / dynamic probability, and velocity, using at least some of the input data in input data 1060. The dynamic mesh determined by dynamic mesh function block 1030 may be smaller than the dynamic mesh determined by LLP 1010. Dynamic mesh function block 1030 may use more conventional (non-machine learning) techniques that can identify some objects not identified by LLP 1010 (e.g., objects with odd shapes and / or set at odd angles relative to device 500). In the example, dynamic mesh function block 1030 may utilize machine learning labels (e.g., outputs) to generate category information associated with detected objects (e.g., cars, trucks, road edges, etc.). Dynamic mesh function block 1030 may provide the dynamic mesh to clustering function block 1040 and static extraction function block 1050. Clustering function block 1040 can be configured to identify clusters of dynamic grid cells with similar properties (e.g., similar object classifications and / or similar velocities) and provide object tracker 1020 with indications of the clustering of grid cells of dynamic objects. In the example, clustering may be based at least in part on category information. Static extraction function block 1050 may also utilize category information to provide segmentation output in addition to static object information. Object tracker 1020 can be configured to track objects using indications of identified objects from LLP 1010 and indications of clustering of dynamic grid cells from clustering function block 1040, for example, using a Kalman filter (and / or other algorithms). Object tracker 1020 can be configured to merge the identified objects from LLP 1010 with the dynamic objects (corresponding to cell clusters) determined by clustering function block 1040 and output a list 1070 of object trajectories indicating the tracked objects. The object trajectory list 1070 may include the position, velocity, length, and width (and possibly other information) of each object in the object trajectory list 1070.The object trajectory list 1070 may include the shape representing each object, such as a closed polygon or other shape (e.g., an ellipse (e.g., indicated by the values ​​of the major and minor axes)), where different objects are represented by the same shape or different shapes. The static extraction function block 1050 may be configured to determine static objects (e.g., road boundaries, traffic signs, etc.) in the dynamic mesh provided by the dynamic mesh function block 1030, and provide a static object list 1080 indicating the determined static objects.

[0058] refer to Figure 11 The diagram illustrates an example process 1100 for controlling an autonomous vehicle. However, process 1100 is an example and not a limitation. Process 1100 can be modified, for example, by adding, removing, rearranging, combining, performing one or more stages concurrently, and / or splitting one or more stages into multiple stages.

[0059] At stage 1102, the process includes obtaining input from one or more sensors. Sensors may include radar 542 of sensor 540, camera 544 of sensor 540, and the input may include the attitude of device 500 (e.g., the orientation of the autonomous vehicle relative to a reference axis (e.g., yaw)). The signals obtained from the sensors may be provided to a sensing module, such as... Figure 10 The environment described in the text is module 1000.

[0060] At stage 1104, the process includes processing sensor input using an environment building module 1000. Typically, the environment building module is configured to determine the contextual understanding of the environment, such as detecting and classifying nearby vehicles or objects (e.g., cars, trucks, bicycles, pedestrians, etc.), determining the location of nearby obstacles, detecting road signs, and classifying data based on semantic definitions. The environment building module 1000 may be configured to analyze sensor data using one or more object recognition processes. For example, LLP 1010 may be configured to use one or more machine learning models to identify objects detected by sensor 540. LLP 1010 may be configured to perform per-frame analysis of the sensor data and provide indications (e.g., output labels) from the machine learning models to object tracker 1020. In this example, the output labels may be parameter values ​​(e.g., location coordinates, vehicle, size, lane, etc.). Object tracker 1020 is configured to track indications (e.g., objects) over time and output a list of object trajectories 1070. Dynamic mesh function block 1030 and clustering function block 1040 can be configured to provide object tracker 1020 with non-parametric object information (e.g., a set 820 of occupancy information) associated with dynamic objects. Object tracker 1020 can be configured to perform a fusion of LLP objects (e.g., parametric objects) and clustered objects (e.g., non-parametric objects) to output object trajectory list 1070, which includes, for example, shapes representing objects, such as closed polygons or other shapes. Static extraction function block 1050 can be configured to determine static objects (e.g., road boundaries, traffic signs, etc.) in the dynamic mesh provided by dynamic mesh function block 1030 and provide static objects 1080 indicating the determined static objects. Compared to directly providing dynamic mesh information and LLP objects, the object trajectory list 1070 and static objects 1080 can be provided to other perception modules with lower bandwidth and a less complex interface.

[0061] At stage 1106, the process includes providing the object trajectory list 1070 and static objects 1080 from the environment building module 1000 to the perception planning block. In this example, the perception planning block may be configured to perform task, behavior, and / or motion planning functions. Task planning functions may include analysis of traversable road segments based on map information. Behavior planning may include operating vehicles according to local road rules. Motion planning functions may include generating motion paths to avoid collisions with obstacles, such as those provided in the object trajectory list 1070 and static objects 1080. Other decision structures for obstacle avoidance may also utilize the object trajectory list 1070 and static objects 1080. The perception planning block may be configured to output one or more target actions to one or more control modules.

[0062] At stage 1108, the process includes providing a target action to the control module. The control module is configured to execute the target action provided by the perception planning block. For example, the control module may be configured to provide commands, such as steering, acceleration, and braking, to actuators to control the movement of the vehicle. The controller may also be configured to perform trajectory and path tracking and provide feedback to one or more perception planning blocks.

[0063] refer to Figure 12 For further reference Figure 6 and Figure 10 An example functional architecture of an environment building block 1200 with occlusion mesh calculation 1210 is shown. Typically, occlusion mesh calculation 1210 is configured to receive input data 1060 from radar 542 of sensor 540 and camera 544 of sensor 540, and determine whether a sub-region 710 in the dynamic mesh is occluded (e.g., if a cell is not visible to the vehicle itself, for example, occluded by another object (e.g., a cell in the field of view of vehicle 601 is occluded by vehicle 602)) and output an occlusion mesh 1220. In the example, dynamic mesh function block 1030 may be configured to utilize Dempster-Shafer evidence theory to determine whether a cell is occluded. For example, refer to... Figure 8 If subregion 710 was previously in the field of view and is now blocked outside the field of view, the occupancy information can indicate the occlusion state (e.g., P). oc Occupancy information can indicate multiple occupant types; for example, if sub-region 710 was previously identified as occupied by a moved object but sub-region 710 is now occluded, then the movement is occluded. In example occupancy grid 800, Figure 6 The vehicle 601 shown is a self-contained vehicle, and the occupancy information of all cells 810 has been determined. In the occupancy grid 800, unmarked cells 810 (for clarity of the diagram) can be free space, unknown, or occluded. In the example, prediction, update, and / or conflict rules within the dynamic grid function block 1030 can be specified at least in part based on the occlusion grid 1220.

[0064] refer to Figure 13 For further reference Figure 10A block diagram illustrating an example functional architecture of an environment building module 1300, including external constraints, is shown. The environment building module 1300 includes a map data module 1310 and a CV2X module 1320 communicatively coupled to a dynamic grid functional block 1030. In this example, the map data module 1310 may be a navigation system operating on device 500 and / or device 200. The dynamic grid functional block 1030 may be configured to receive map information to enhance the determination of dynamic grid cells. For example, when a car is traveling in the leftmost lane of a three-lane road and is obscured by a truck traveling in the middle lane while the car is traveling in the right lane, the map information can provide indications of whether the road is straight or curved or whether an intersection exists. The map data can be used to enhance the probability of cells (e.g., dynamic grid cells) associated with the obscured vehicle. Knowledge of the map information can be used to provide initial probabilities for grid calculations. For example, the dynamic grid function block 1030 can be configured to initialize the static quality of grid cells (e.g., where there are no roads) and the free / dynamic quality where roads exist using map information, rather than initializing the solution with all grid cells having unknown states. Further outputs of the dynamic grid function block 1030 can be fused with map information to enhance the accuracy of the resulting dynamic grid.

[0065] CV2X module 1320 can be configured to provide map information and / or additional sensor information to dynamic mesh functional block 1030. For example, other radio nodes such as UEs, vehicles, RSUs, and base stations (e.g., gNBs) can be configured to provide their respective sensor readings to device 500. In this example, other radio nodes can provide their representation of the dynamic mesh of the area. Dynamic mesh functional block 1030 can utilize the additional mesh representation to improve the accuracy of the dynamic mesh. Additional dynamic mesh information received from external sources (e.g., other radio nodes) can be used to improve the tracking of occluded cells. The locations of other radio nodes (e.g., other vehicles) can also be used to improve the estimation of dynamic objects in the mesh by associating the dynamic mesh quality with the locations of other vehicles. RSUs can be configured to provide traffic light and traffic state information via V2X links (e.g., Uu, PC5), which can enhance dynamic mesh determination. For example, geographic knowledge of intersections and indications of red traffic lights can increase the probability that a detected object is a stopped vehicle. Other associations based on traffic states at intersections can be used.

[0066] In the example, the fusion of dynamic objects with indications of identified objects from LLP 1010 in object tracker 1020 can generate results for objects not identified by LLP 1010. That is, clusters identified by clustering function block 1040 may correspond to objects not in the training data of the machine learning model in LLP 1010. Object tracker 1020 can be configured to output Active Learning (AL) trigger 1330 to update the training in LLP 1010 based on objects identified in the dynamic grid. In the example, object tracker 1020 can be configured to compare a list of clusters (e.g., clusters identified by clustering function block 1040) with objects from LLP 1010 and determine whether a particular set of one or more frames (e.g., data from radar / camera / sensor within a time window at or near that time (e.g., 0.5, 1.0, 1.5 seconds, etc.) can be selected for further training. The selected frames can be stored in storage for offline training or fed for online training. Frame determination can be based on comparing the differences between two outputs. Frames with higher differences (e.g., above a threshold) can be selected by AL trigger 1330. Typically, the selection algorithm can be based on a direct comparison between the LLP 1010 and the dynamic mesh 1030 outputs. For example, if AL trigger 1330 indicates that the scene is more biased towards intersection types (e.g., based on static quality layout), this classification can be an aspect used to select the corresponding frame for further training.

[0067] refer to Figure 14 For further reference Figures 1 to 13 Method 1400 for generating a list of object trajectories in a vehicle includes the stages shown. However, method 1400 is illustrative and not restrictive. Method 1400 can be modified, for example, by adding, removing, rearranging, combining, performing one or more stages concurrently, and / or splitting one or more stages into multiple stages.

[0068] At stage 1402, the method includes obtaining sensor information from one or more sensors on the vehicle. Device 500, including processor 510 and sensor 540, is a component for obtaining sensor information. In the example, reference... Figure 10 The environment building module 1000 can be configured to acquire sensor information. The LLP 1010 and dynamic mesh 1030 function blocks can receive input data 1060 from the radar 542 of the sensor 540, the camera 544 of the sensor 540, and the attitude of the device 500 (e.g., the orientation of the autonomous vehicle relative to a reference axis (e.g., yaw)).

[0069] At stage 1404, the method includes determining a first object dataset based at least in part on sensor information and an object recognition process. Device 500, including processor 510 and environment building module 1000, is the component used to determine the first object dataset. In the example, LLP 1010 may be configured to implement the object recognition process via one or more machine learning models configured to receive at least some of the input data from input data 1060 to identify dynamic objects (e.g., vehicles) corresponding to the environment of device 500 (e.g., environment 600). The machine learning models may be based on deep learning techniques, such as, for example, camera deep learning (DL) detection models, low-level (LL) fusion of object models, and radar detection models. The machine learning models may be trained to output object data based on input received from sensor 540.

[0070] At stage 1406, the method includes generating a dynamic mesh based at least in part on sensor information to be based on the environment in proximity to the vehicle. Device 500, including processor 510 and environment building module 1000, is the component for generating the dynamic mesh. Dynamic mesh function block 1030 can be configured to determine a dynamic mesh including occupancy probability, static / dynamic probability, and velocity using at least some of the input data in input data 1060. In the example, dynamic mesh function block 1030 can use more conventional (non-machine learning) techniques that can identify some objects not identified by LLP 1010 (e.g., objects with odd shapes and / or set at odd angles relative to device 500). In the example, dynamic mesh function block 1030 can leverage machine learning labels (e.g., outputs) to generate category information associated with detected objects (e.g., cars, trucks, road edges, etc.). For example, dynamic mesh function block 1030 can include one or more machine learning models, such as a camera drivable space model, a camera-based semantic segmentation (camera SemSeg) model, a radar point cloud model, and a low-level bird's-eye view (BEV) segmentation and occupancy flow model. Other models can also be used to generate dynamic meshes.

[0071] At stage 1408, the method includes determining a second object dataset based at least in part on a dynamic grid. Device 500, including processor 510 and environment building module 1000, is the component for determining the second object dataset. In the example, dynamic grid function block 1030 can provide a dynamic grid to clustering function block 1040 and static extraction function block 1050. Clustering function block 1040 can be configured to generate the second object dataset based on clusters of dynamic grid cells that identify similar properties (e.g., similar object classifications and / or similar velocities). In the example, clustering can be based at least in part on category information. In the example, static extraction function block 1050 can be configured to generate a third object dataset based on static objects (e.g., road boundaries, traffic signs, etc.) in the dynamic grid information provided by dynamic grid function block 1030.

[0072] At stage 1410, the method includes outputting a list of object trajectories based on the fusion of a first object dataset and a second object dataset. Device 500, including processor 510 and environment building module 1000, is the component for outputting the list of object trajectories. In the example, object tracker 1020 may be configured to track objects using a first object dataset (e.g., indications of identified objects from LLP 1010) and a second object dataset (e.g., indications of clustering of dynamic grid cells from clustering function block 1040), for example, using a Kalman filter (and / or other algorithms). Object tracker 1020 may be configured to fuse the identified objects from LLP 1010 with the dynamic objects (corresponding to cell clusters) determined by clustering function block 1040 and output a list 1070 of object trajectories indicating the tracked objects. In the example, object trajectory list 1070 may include the position, velocity, length, and width (and possibly other information) of each object in object trajectory list 1070. The object trajectory list 1070 may include a shape representing each object, such as a closed polygon or other shape (e.g., an ellipse (e.g., indicated by the values ​​of the major and minor axes)).

[0073] refer to Figure 15 For further reference Figures 1 to 13 The method 1500 for generating motion planning for autonomous vehicles includes the stages shown. However, method 1500 is illustrative and not restrictive. Method 1500 can be modified, for example, by adding, removing, rearranging, combining, performing one or more stages concurrently, and / or splitting one or more stages into multiple stages. For example, outputting control commands at stage 1506 is optional.

[0074] At stage 1502, the method includes receiving object information including a list of object trajectories and static obstacle information, wherein the object trajectory list is based on a fusion of the output of the object recognition process and clustering analysis of the dynamic occupancy grid. A device 500, including a processor 510 and a sensing planning block, is a component for receiving the object information. In the example, the sensing planning block may receive an object trajectory list 1070 and static objects 1080 from an environment building module 1000. The object trajectory list represents a simplification of the dynamic grid because objects are provided in list form, rather than the more complex raw output from the dynamic grid function block 1030. The reduced complexity and relative compression of the object data also reduces the bandwidth required to provide object information to the sensing planning block and / or other planning control elements.

[0075] At stage 1504, the method includes generating motion planning based at least in part on object information. Device 500, including processor 510 and a perceptual planning block, is the component for generating motion planning. The perceptual planning block can be configured to perform tasks, behaviors, and / or motion planning functions. Motion planning functions may include generating motion paths to avoid collisions with obstacles, such as those provided in object trajectory list 1070 and static objects 1080. Other decision structures for avoiding obstacles may also utilize object trajectory list 1070 and static objects 1080.

[0076] At stage 1506, the method optionally includes outputting control commands based at least in part on motion planning. The device 500, including processor 510 and a sensing planning block, is the component for outputting control commands. The sensing planning block can be configured to output one or more target actions to one or more control modules. In the example, the control modules can be configured to receive control commands and provide commands, such as steering, acceleration, and braking, to actuators to control the movement of the vehicle based on the control commands.

[0077] Other examples and specific implementations are within the scope of this disclosure and the appended claims. For example, due to the nature of software and computers, the functions described above can be implemented using software, hardware, firmware, hardwiring, or any combination thereof executed by a processor. Features implementing the functions can also be physically located in various locations, including various portions distributed such that the functions are implemented in different physical locations.

[0078] As used herein, the singular forms “a,” “an,” and “the” also include the plural forms, unless the context clearly indicates otherwise. Thus, references to a device in the singular form included in the claims (e.g., “device,” “the / said device”) include at least one of such devices (i.e., one or more) (e.g., “processor” includes at least one processor (e.g., one processor, two processors, etc.), “the / said processor” includes at least one processor, “memory” includes at least one memory, “the / said memory” includes at least one memory, etc.). The phrases “at least one” and “one or more” are used interchangeably, and such that the object referred to by “at least one” and the object referred to by “one or more” include embodiments having one referred object and embodiments having multiple referred objects. For example, “at least one processor” and “one or more processors” each include embodiments having one processor and embodiments having multiple processors.

[0079] Furthermore, as used herein, the "or" (possibly followed by "at least one of" or "one or more of") used in the item enumeration indicates a disjunctive enumeration such that an enumeration of, for example, "at least one of A, B, or C," or an enumeration of "one or more of A, B, or C," or an enumeration of "A or B or C" represents A or B or C or AB (A and B) or AC (A and C) or BC (B and C) or ABC (i.e., A and B and C), or a combination having more than one feature (e.g., AA, AAB, ABBC, etc.). Therefore, a statement that an item (e.g., a processor) is configured to perform a function relating to at least one of A or B, or a statement that an item is configured to perform function A or function B, indicates that the item can be configured to perform a function relating to A, or can be configured to perform a function relating to B, or can be configured to perform a function relating to both A and B. For example, the phrase "a processor configured to measure at least one of A or B" or "a processor configured to measure A or measure B" means that the processor can be configured to measure A (and may or may not be configured to measure B), or can be configured to measure B (and may or may not be configured to measure A), or can be configured to measure both A and B (and can be configured to select which of A and B or measure both). Similarly, a description of a component for measuring at least one of A or B includes: a component for measuring A (which may or may not be able to measure B), or a component for measuring B (which may or may not be configured to measure A), or a component for measuring A and B (which may be able to select which of A and B or measure both). As another example, a description of an item (e.g., a processor) being configured to perform at least one of function X or function Y means that the item can be configured to perform function X, or can be configured to perform function Y, or can be configured to perform both functions X and Y. For example, the phrase "processor configured to measure at least one of X or Y" means that the processor can be configured to measure X (and may or may not be configured to measure Y), or can be configured to measure Y (and may or may not be configured to measure X), or can be configured to measure both X and Y (and can be configured to select which of X and Y or measure both).

[0080] As used herein, unless otherwise stated, a description of a function or operation as “based on” an item or condition means that the function or operation is based on the described item or condition and may be based on one or more items and / or conditions other than the described item or condition.

[0081] Substantial changes can be made depending on specific requirements. For example, custom hardware may be used, and / or specific elements may be implemented in the hardware, in software executed by the processor (including portable software such as applets), or both. Furthermore, connections to other computing devices, such as network input / output devices, may be employed. Unless otherwise specified, components shown in the figures and / or discussed herein that are connected or communicate with each other (functionally or otherwise) are communicatively coupled. That is, these components may be connected directly or indirectly to enable communication between them.

[0082] The systems and devices discussed above are examples. Various configurations may appropriately omit, substitute, or add various processes or components. For example, features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of configurations may be combined in a similar manner. Furthermore, technology is constantly evolving, and therefore many elements are examples and do not limit the scope of this disclosure or the claims.

[0083] A wireless communication system is a system in which communication is transmitted wirelessly between wireless communication devices, i.e., by electromagnetic waves and / or sound waves propagating through the atmosphere rather than through wires or other physical connections. A wireless communication system (also called a wireless communication system or wireless communication network) may not transmit all communication wirelessly, but is configured to transmit at least some communication wirelessly. Furthermore, the term "wireless communication device" or similar terms do not require the device to be functionally exclusive or even primarily used for communication, do not require that communication using the wireless communication device be exclusive or even primarily wireless, and do not require that the device be a mobile device, but rather indicate that the device includes wireless communication capabilities (one-way or two-way), for example, including at least one radio component (each radio component being part of a transmitter, receiver, or transceiver) for wireless communication.

[0084] Specific details are provided in this description to offer a thorough understanding of the example configurations, including specific implementations. However, the configurations can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary detail to avoid obscuring these configurations. The description herein provides example configurations and does not limit the scope, applicability, or configuration of the claims. Rather, the preceding description of the configurations provides a description for implementing the described techniques. Various changes can be made to the function and arrangement of the elements.

[0085] As used herein, the terms “processor-readable medium,” “machine-readable medium,” and “computer-readable medium” refer to any medium that participates in providing data that enables a machine to operate in a particular manner. Using a computing platform, various processor-readable media may involve providing instructions / code to a processor for execution, and / or may be used to store and / or carry such instructions / code (e.g., as signals). In many specific implementations, processor-readable media are physical and / or tangible storage media. Such media can take many forms, including but not limited to non-volatile and volatile media. Non-volatile media include, for example, optical discs and / or magnetic disks. Volatile media include, but are not limited to, dynamic memory.

[0086] Having described several example configurations, various modifications, alternative constructions, and equivalents can be used. For example, the above elements can be components of a larger system, where other rules may take precedence over or otherwise modify the application of this disclosure. Furthermore, several operations may be performed before, during, or after considering the above elements. Accordingly, the above description does not limit the scope of the claims.

[0087] Unless otherwise indicated, the terms "about" and / or "approximately" as used herein when referring to measurable values ​​(such as quantities, durations of time, etc.) cover variations of ±20%, ±10%, ±5%, or ±0.1% from the specified value, as appropriate in the context of the systems, devices, circuits, methods, and other specific embodiments described herein. Similarly, unless otherwise indicated, the term "substantially" as used herein when referring to measurable values ​​(such as quantities, durations of time, physical properties (such as frequencies), etc.) also covers variations of ±20%, ±10%, ±5%, or ±0.1% from the specified value, as appropriate in the context of the systems, devices, circuits, methods, and other specific embodiments described herein.

[0088] A statement that a value exceeds (or is greater than or higher than) a first threshold is equivalent to a statement that a value meets or exceeds a second threshold slightly greater than the first threshold. For example, in the resolution of the computing system, the second threshold is one value higher than the first threshold. A statement that a value is less than the first threshold (or within or below the first threshold) is equivalent to a statement that a value is less than or equal to a second threshold slightly lower than the first threshold. For example, in the resolution of the computing system, the second threshold is one value lower than the first threshold.

[0089] Specific implementation examples are described in the following numbered clauses:

[0090] Clause 1. A method for generating a list of object trajectories in a vehicle, the method comprising: obtaining sensor information from one or more sensors on the vehicle; determining a first object dataset based at least in part on the sensor information and an object recognition process; generating a dynamic mesh based at least in part on the sensor information and on an environment close to the vehicle; determining a second object dataset based at least in part on the dynamic mesh; and outputting the list of object trajectories based on a fusion of the first object dataset and the second object dataset.

[0091] Clause 2. The method according to Clause 1, wherein the one or more sensors include at least a camera and a radar sensor.

[0092] Clause 3. The method according to Clause 1, wherein determining the second object dataset includes clustering that identifies dynamic grid cells in the dynamic grid.

[0093] Clause 4. The method described in Clause 3, wherein the clustering of the dynamic grid cells has a similar speed.

[0094] Clause 5. The method described in Clause 3, wherein the clustering of the dynamic grid cells has similar object classifications.

[0095] Clause 6. The method of Clause 1, further comprising generating an occlusion grid including occluded grid cells, wherein the determination of the second object dataset is based at least in part on the occlusion grid.

[0096] Clause 7. The method according to Clause 1, wherein the object recognition process includes one or more machine learning models configured to output an indication of an identified object based at least in part on the sensor information.

[0097] Clause 8. The method according to Clause 7, the method further comprising outputting an active learning (AL) trigger at least in part based on a comparison of the first object dataset with the second object dataset, wherein the AL trigger is configured to cause retraining of the one or more machine learning models.

[0098] Clause 9. The method according to Clause 1, wherein the object trajectory list includes shape information for representing the detected objects.

[0099] Clause 10. The method according to Clause 1, wherein the list of object trajectories includes at least the position and velocity of the detected objects.

[0100] Clause 11. The method according to Clause 1, the method further comprising receiving map information, wherein the generation of the dynamic grid is based at least in part on the map information.

[0101] Clause 12. The method according to Clause 1, the method further comprising receiving remote sensor information via CV2X network communication, wherein the generation of the dynamic mesh is based at least in part on the remote sensor information.

[0102] Clause 13. The method described in Clause 12, wherein the remote sensor information is provided by a roadside unit (RSU).

[0103] Clause 14. An apparatus comprising: at least one memory; one or more sensors; at least one processor communicatively coupled to the at least one memory and the one or more sensors, and configured to: obtain sensor information from the one or more sensors; determine a first object dataset based at least in part on the sensor information and an object recognition process; generate a dynamic mesh based at least in part on the sensor information; determine a second object dataset based at least in part on the dynamic mesh; and output a list of object trajectories based on a fusion of the first object dataset and the second object dataset.

[0104] Clause 15. The apparatus according to Clause 14, wherein the one or more sensors include at least a camera and a radar sensor.

[0105] Clause 16. The apparatus of Clause 14, wherein the at least one processor is further configured to identify clusters of dynamic grid cells in the dynamic grid.

[0106] Clause 17. The apparatus of Clause 16, wherein the clustering of the dynamic grid cells has a similar speed.

[0107] Clause 18. The apparatus of Clause 16, wherein the clustering of the dynamic grid cells has similar object classifications.

[0108] Clause 19. The apparatus of Clause 14, wherein the at least one processor is further configured to: generate an occlusion mesh including occluded grid cells, and determine the second object dataset based at least in part on the occlusion mesh.

[0109] Clause 20. The apparatus of Clause 14, wherein the at least one memory includes one or more machine learning models, and the at least one processor is further configured to output an indication of an identified object based at least in part on the sensor information and the one or more machine learning models.

[0110] Clause 21. The apparatus of Clause 20, wherein the at least one processor is further configured to: output an active learning (AL) trigger based at least in part on a comparison of the first object dataset with the second object dataset, and to retrain the one or more machine learning models in response to the AL trigger.

[0111] Clause 22. The apparatus of Clause 14, wherein the object trajectory list includes shape information for representing detected objects.

[0112] Clause 23. The apparatus according to Clause 14, wherein the list of object trajectories includes at least the position and velocity of the detected objects.

[0113] Clause 24. The apparatus according to Clause 14, wherein the at least one processor is further configured to: receive map information and generate the dynamic grid based at least in part on the map information.

[0114] Clause 25. The apparatus according to Clause 14, wherein the at least one processor is further configured to: receive remote sensor information via CV2X network communication, and generate the dynamic mesh based at least in part on the remote sensor information.

[0115] Clause 26. The apparatus described in Clause 25, wherein the remote sensor information is provided by a roadside unit (RSU).

[0116] Clause 27. An apparatus for generating a list of object trajectories in a vehicle, the apparatus comprising: components for obtaining sensor information from one or more sensors on the vehicle; components for determining a first object dataset based at least in part on the sensor information and an object recognition process; components for generating a dynamic mesh based at least in part on the sensor information and on an environment close to the vehicle; components for determining a second object dataset based at least in part on the dynamic mesh; and components for outputting the list of object trajectories based on a fusion of the first object dataset and the second object dataset.

[0117] Clause 28. A non-transitory processor-readable storage medium including processor-readable instructions configured to cause one or more processors to generate a list of object trajectories in a vehicle, the processor-readable instructions including code for: obtaining sensor information from one or more sensors on the vehicle; determining a first object dataset based at least in part on the sensor information and an object recognition process; generating a dynamic mesh based at least in part on the sensor information and on an environment close to the vehicle; determining a second object dataset based at least in part on the dynamic mesh; and outputting the list of object trajectories based on a fusion of the first object dataset and the second object dataset.

Claims

1. A method for generating a list of object trajectories in a vehicle, the method comprising: Sensor information is obtained from one or more sensors on the vehicle; The first object dataset is determined at least in part based on the sensor information and the object recognition process; A dynamic mesh is generated, at least in part, based on the sensor information and the environment in proximity to the vehicle. The second object dataset is determined at least in part based on the dynamic grid; as well as The object trajectory list is output based on the fusion of the first object dataset and the second object dataset.

2. The method of claim 1, wherein the one or more sensors include at least a camera and a radar sensor.

3. The method of claim 1, wherein determining the second object dataset includes clustering that identifies dynamic grid cells in the dynamic grid.

4. The method of claim 3, wherein the clustering of the dynamic grid cells has a similar speed.

5. The method of claim 3, wherein the clustering of the dynamic grid cells has similar object classifications.

6. The method of claim 1, further comprising generating an occlusion grid including occluded grid cells, wherein the determination of the second object dataset is at least partially based on the occlusion grid.

7. An apparatus comprising: At least one memory; One or more sensors; At least one processor, communicatively coupled to the at least one memory and the one or more sensors, and configured to: Obtain sensor information from the one or more sensors; The first object dataset is determined at least in part based on the sensor information and the object recognition process; The dynamic mesh is generated at least in part based on the sensor information; The second object dataset is determined at least in part based on the dynamic grid; as well as The object trajectory list is output based on the fusion of the first object dataset and the second object dataset.

8. The apparatus of claim 7, wherein the one or more sensors include at least a camera and a radar sensor.

9. The apparatus of claim 7, wherein the at least one processor is further configured to identify clusters of dynamic grid cells in the dynamic grid.

10. The apparatus of claim 9, wherein the clustering of the dynamic grid cells has a similar speed.

11. The apparatus of claim 9, wherein the clustering of the dynamic grid cells has similar object classifications.

12. The apparatus of claim 7, wherein the at least one processor is further configured to: generate an occlusion grid including occluded grid cells, and determine the second object dataset based at least in part on the occlusion grid.

13. The apparatus of claim 7, wherein the at least one memory includes one or more machine learning models, and the at least one processor is further configured to output an indication of an identified object based at least in part on the sensor information and the one or more machine learning models.

14. The apparatus of claim 13, wherein the at least one processor is further configured to: output an active learning (AL) trigger at least in part based on a comparison of the first object dataset with the second object dataset, and to retrain the one or more machine learning models in response to the AL trigger.

15. The apparatus of claim 7, wherein the object trajectory list includes shape information for representing detected objects.

16. The apparatus of claim 7, wherein the list of object trajectories includes at least the position and velocity of the detected objects.

17. The apparatus of claim 7, wherein the at least one processor is further configured to: receive map information and generate the dynamic grid based at least in part on the map information.

18. The apparatus of claim 7, wherein the at least one processor is further configured to: receive remote sensor information via CV2X network communication, and generate the dynamic mesh based at least in part on the remote sensor information.

19. The apparatus of claim 18, wherein the remote sensor information is provided by a roadside unit (RSU).

20. An apparatus for generating a list of object trajectories in a vehicle, the apparatus comprising: Components for obtaining sensor information from one or more sensors on the vehicle; Components for determining a first object dataset based at least in part on the sensor information and the object recognition process; A component for generating a dynamic mesh based at least in part on the sensor information and on the environment in proximity to the vehicle; Components for determining a second object dataset based at least in part on the dynamic grid; and A component for outputting the list of object trajectories based on the fusion of the first object dataset and the second object dataset.

Images