Occupancy grid determination

Abstract

An occupancy grid determination method includes determining a predicted occupancy grid based on a previous occupancy grid, the predicted occupancy grid including a plurality of first cells corresponding to sub-regions of a region, each of the plurality of first cells includes a plurality of predicted probability indications each indicating a probability of a prediction of a respective possible type of an occupant of the respective first cell; determining an observed occupancy grid using machine learning and based on the first sensor measurements, the observed occupancy grid comprising a plurality of second cells corresponding to the sub-regions of the region; and determining an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. application serial number 18 / 477,893, filed on September 29, 2023, entitled “OCCUPANCY GRID DETERMINATION,” which claims the benefit of U.S. provisional application serial number 63 / 380,978, filed on October 26, 2022, entitled “OCCUPANCY GRID DETERMINATION,” which is assigned to the assignee of this application, and the entire contents of both applications are hereby incorporated herein by reference for all purposes. Background Art

[0003] Vehicles are becoming more intelligent as industry moves toward deploying increasingly sophisticated self-driving technologies that are capable of operating the vehicle with little or no human input and are therefore semi-autonomous or autonomous. Autonomous and semi-autonomous vehicles may be able to detect information about their location and surrounding environment (e.g., using ultrasound, radar, lidar, SPS (satellite positioning system), and / or odometers, and / or one or more sensors such as accelerometers, cameras, etc.). Autonomous and semi-autonomous vehicles typically include a control system to interpret information about the environment in which the vehicle is located, thereby identifying hazards and determining a navigation path to follow.

[0004] The driver assistance system can mitigate the driving risk of the driver and / or other road users of the self-vehicle (i.e., a vehicle configured to perceive the environment of the vehicle). The driver assistance system may include one or more active devices and / or one or more passive devices that can be used to determine the environment of the self-vehicle, and for semi-autonomous vehicles, may notify the driver of situations that the driver may be able to resolve. The driver assistance system may be configured to control various aspects of driving safety and / or driver monitoring. For example, the driver assistance system may control the speed of the self-vehicle to maintain at least the desired interval (in distance or time) between the self-vehicle and another vehicle (e.g., as part of an active cruise control system). The driver assistance system may monitor the surrounding environment of the self-vehicle, for example, to maintain the situation awareness of the self-vehicle. The situation awareness can be used to notify the driver of problems, for example, another vehicle is in the driver's blind spot, another vehicle is located on a collision path with the self-vehicle, etc. The situation awareness may include information about the self-vehicle (e.g., speed, position, heading) and / or other vehicles or objects (e.g., position, speed, heading, size, object type, etc.).

[0005] The state of the ego vehicle may be used as input to a number of driver assistance functionalities such as an advanced driver assistance system (ADAS). Downstream driver assistance such as ADAS may be safety critical and / or may give the driver information about the vehicle and / or control the vehicle in some manner. Summary of the invention

[0006] An example apparatus includes: a memory; and a processor communicatively coupled to the memory and configured to: determine a predicted occupancy grid based on a previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to sub-regions of an area, each of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell; determine an observed occupancy grid using machine learning and based on first sensor measurements, the observed occupancy grid comprising a plurality of second cells corresponding to the sub-regions of the area; and determine an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid.

[0007] An example occupancy grid determination method includes: determining a predicted occupancy grid based on a previous occupancy grid, the predicted occupancy grid including a plurality of first cells corresponding to sub-regions of a region, each first cell of the plurality of first cells including a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell; determining an observed occupancy grid using machine learning and based on first sensor measurements, the observed occupancy grid including a plurality of second cells corresponding to the sub-regions of the region; and determining an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid.

[0008] Another example apparatus includes: means for determining a predicted occupancy grid based on a previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to sub-regions of an area, each first cell of the plurality of first cells comprising a plurality of predicted probability indications that each indicate a predicted probability of a respective possible type of an occupant of the respective first cell; means for determining an observed occupancy grid using machine learning and based on first sensor measurements, the observed occupancy grid comprising a plurality of second cells corresponding to the sub-regions of the area; and means for determining an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid.

[0009] An example non-transitory processor-readable storage medium includes processor-readable instructions that cause a processor to: determine a predicted occupancy grid based on a previous occupancy grid, the predicted occupancy grid including a plurality of first cells corresponding to sub-regions of an area, each first cell of the plurality of first cells including a plurality of predicted probability indications that each indicate a predicted probability of a respective possible type of an occupant of the respective first cell; determine an observed occupancy grid using machine learning and based on first sensor measurements, the observed occupancy grid including a plurality of second cells corresponding to the sub-regions of the area; and determine an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a top view of an example ego vehicle.

[0011] Figure 2 yes Figure 1 The illustrated block diagram is of components of an example device of which an ego-vehicle may be an example.

[0012] Figure 3 is a block diagram of the components of an example send / receive point.

[0013] Figure 4 is a block diagram of the components of a server.

[0014] Figure 5 is a block diagram of an example device.

[0015] Figure 6 is a diagram of an example geographic environment.

[0016] Figure 7 It is divided into a grid Figure 6 Illustration of the geographical environment shown.

[0017] Figure 8 is with Figure 7 An example of an occupancy map corresponding to the grid shown.

[0018] Fig. 9 is a block diagram of an example functional architecture for Bayesian filtering.

[0019] Fig.10 is a block diagram of an example functional architecture for Bayesian filtering using measurements from multiple sensors.

[0020] Fig.11 is another block diagram of an example functional architecture for Bayesian filtering using measurements from multiple sensors.

[0021] Fig.12 is another block diagram of an example functional architecture for Bayesian filtering using measurements from multiple sensors.

[0022] Fig.13 is another block diagram of an example functional architecture for Bayesian filtering using measurements from multiple sensors.

[0023] Fig.14 is another block diagram of an example functional architecture for Bayesian filtering using measurements from multiple sensors.

[0024] Fig.15 is a flowchart for developing the image to occupancy grid transformation.

[0025] Fig.16 is a flowchart of an example occupancy grid determination method. DETAILED DESCRIPTION

[0026] Techniques for determining and using occupancy grids are discussed herein. For example, measurements from multiple sensors may be obtained, and measurements from at least one of these sensors may be applied to an observation matrix determined using machine learning. Machine learning may be used to select which sensor measurement(s) to use for a particular cell of an occupancy grid of an observation determined from the sensor measurements, possibly using a combination of measurements from different sensors for the same observed occupancy grid cell. Machine learning may be used to select which occupancy grid cell to use for a particular cell of a current occupancy grid from one or more occupancy grids each corresponding to a different sensor. The current occupancy grid may be used to update a predicted occupancy grid determined from a previous occupancy grid. Machine learning may be used to derive an image to occupancy grid transform to transform a camera image or a set of information arrays determined from a camera image into an occupancy grid. However, other techniques may be used.

[0027] Items and / or techniques described herein may provide one or more of the following capabilities, as well as other capabilities not mentioned. Occupancy grid accuracy and / or reliability may be improved. Occupancy grids may be determined without losing much, if any, information from camera images. Probability, belief, and / or plausibility of occupancy grids for dynamically occupied grid cells may be better predicted. Other capabilities may be provided, and not every implementation according to the present disclosure necessarily provides any, much less all, of the capabilities discussed.

[0028] refer to Figure 1, the ego vehicle 100 includes an ego vehicle driver assistance system 110. The driver assistance system 110 may include a plurality of different types of sensors mounted at appropriate locations on the ego vehicle 100. For example, the system 110 may include: a pair of divergent and outwardly directed radar sensors 121 mounted at respective front corners of the vehicle 100, a similar pair of divergent and outwardly directed radar sensors 122 mounted at respective rear corners of the vehicle, a forwardly directed LRR sensor 123 (long range radar) mounted centrally at the front of the vehicle 100, and a pair of generally forwardly directed optical sensors 124 (cameras) forming part of a SVS 126 (stereo vision system) which may be mounted, for example, in the region of an upper edge of a windshield 128 of the vehicle 100. Each of the sensors 121 may include an LRR and / or an SRR (short range radar). The various sensors 121 to 124 are operably connected to a central electronic control system, which is typically provided in the form of an ECU 140 (electronic control unit) mounted at a convenient location within the vehicle 100. In the particular arrangement illustrated, the front and rear sensors 121, 122 are connected to the ECU 140 via one or more conventional controller area network (CAN) buses 150, and the LRR sensors 123 and SVS 126 sensors are connected to the ECU 140 via a serial bus 160 (e.g., a faster FlexRay serial bus).

[0029] Collectively, and under the control of ECU 140, the various sensors 121-124 may be used to provide a variety of different types of driver assistance functionality. For example, sensors 121-124 and ECU 140 may provide blind spot monitoring, adaptive cruise control, collision prevention assistance, lane departure protection, and / or rear collision mitigation.

[0030] The CAN bus 150 may be viewed by the ECU 140 as a sensor that provides ego vehicle parameters to the ECU 140. For example, a GPS module may also be connected to the ECU 140 as a sensor, thereby providing the ECU 140 with geographic location parameters.

[0031] Also refer to Figure 2, the device 200 (which may be a mobile device such as a user equipment (UE) such as a vehicle (VUE)) includes a computing platform including a processor 210, a memory 211 including software (SW) 212, one or more sensors 213, a transceiver interface 214 for a transceiver 215 (which includes a wireless transceiver 240 and a wired transceiver 250), a user interface 216, a satellite positioning system (SPS) receiver 217, a camera 218, and a positioning device (PD) 219. The processor 210, the memory 211, the sensor 213, the transceiver interface 214, the user interface 216, the SPS receiver 217, the camera 218, and the positioning device 219 may be communicatively coupled to each other via a bus 220 (which may be configured, for example, for optical communication and / or electrical communication). One or more of the devices shown (e.g., the camera 218, the positioning device 219, and / or one or more sensors in the sensor 213, etc.) may be omitted from the device 200. The processor 210 may include one or more hardware devices, such as a central processing unit (CPU), a microcontroller, an application specific integrated circuit (ASIC), etc. The processor 210 may include multiple processors, including a general / application processor 230, a digital signal processor (DSP) 231, a modem processor 232, a video processor 233, and / or a sensor processor 234. One or more of the processors 230 to 234 may include multiple devices (e.g., multiple processors). For example, the sensor processor 234 may include, for example, a processor for RF (radio frequency) sensing (wherein one or more (cellular) wireless signals and reflections transmitted are used to identify, map and / or track objects) and / or ultrasound, etc. The modem processor 232 may support dual SIM / dual connections (or even more SIMs). For example, a SIM (subscriber identity module or subscriber identity module) may be used by an original equipment manufacturer (OEM), and another SIM may be used by an end user of the device 200 to obtain a connection. The memory 211 is a non-transitory storage medium that may include a random access memory (RAM), a flash memory, a disk memory, and / or a read-only memory (ROM), etc. Memory 211 stores software 212, which may be processor-readable, processor-executable software code containing instructions that are configured to cause processor 210 to perform various functions described herein when executed. Alternatively, software 212 may not be directly executable by processor 210, but may be configured to cause processor 210 to perform these functions, for example, when compiled and executed. This specification may refer to processor 210 performing a function, but this includes other specific implementations, such as specific implementations in which processor 210 executes software and / or firmware. This specification may refer to processor 210 performing a function as shorthand for one or more of processors 230-234 performing the function.This specification may refer to device 200 performing a function as shorthand for one or more appropriate components of device 200 performing that function. Processor 210 may include a memory with stored instructions in addition to and / or in lieu of memory 211. The functionality of processor 210 is discussed more fully below.

[0032] Figure 2 The configuration of the device 200 shown is an example and not a limitation of the present disclosure (including the claims), and other configurations may be used. For example, an example configuration of the UE may include one or more of the processors 230 to 234 in the processor 210, the memory 211, and the wireless transceiver 240. Other example configurations may include one or more of the processors 230 to 234 in the processor 210, the memory 211, the wireless transceiver, and one or more of the following: sensor 213, user interface 216, SPS receiver 217, camera 218, PD 219, and / or wired transceiver.

[0033] The device 200 may include a modem processor 232 that may be capable of performing baseband processing of signals received and down-converted by the transceiver 215 and / or the SPS receiver 217. The modem processor 232 may perform baseband processing of signals to be up-converted for transmission by the transceiver 215. Additionally or alternatively, the baseband processing may be performed by the general / application processor 230 and / or the DSP 231. However, other configurations may be used to perform the baseband processing.

[0034] The device 200 may include a sensor 213, which may include, for example, one or more sensors of various types of sensors, such as one or more inertial sensors, one or more magnetometers, one or more environmental sensors, one or more optical sensors, one or more weight sensors, and / or one or more radio frequency (RF) sensors, etc. An inertial measurement unit (IMU) may include, for example, one or more accelerometers (e.g., collectively responsive to acceleration of the device 200 in three dimensions) and / or one or more gyroscopes (e.g., three-dimensional gyroscopes). The sensor 213 may include one or more magnetometers (e.g., three-dimensional magnetometers) to determine orientation (e.g., relative to magnetic north and / or true north), which may be used for any of a variety of purposes (e.g., to support one or more compass applications). Environmental sensors may include, for example, one or more temperature sensors, one or more barometric pressure sensors, one or more ambient light sensors, one or more camera imagers, and / or one or more microphones, etc. The sensor 213 may generate analog and / or digital signals, indications of which may be stored in the memory 211 and processed by the DSP 231 and / or the general / application processor 230 to support one or more applications (such as, for example, applications involving positioning and / or navigation operations).

[0035] Sensor 213 may be used for relative position measurement, relative position determination, motion determination, etc. The information detected by sensor 213 may be used for motion detection, relative displacement, dead reckoning, sensor-based position determination, and / or sensor-assisted position determination. Sensor 213 may be used to determine whether device 200 is fixed (stationary) or mobile and / or whether to report certain useful information related to the mobility of device 200, for example, to LMF (location management function). For example, based on the information obtained / measured by sensor 213, device 200 may notify / report to LMF that device 200 has detected movement or that device 200 has moved, and report relative displacement / distance (e.g., via dead reckoning implemented by sensor 213, or sensor-based position determination, or sensor-assisted position determination). In another example, for relative positioning information, sensor / IMU may be used to determine the angle and / or orientation of another device relative to device 200, etc.

[0036] The IMU may be configured to provide measurements of the direction of motion and / or velocity of motion of the device 200, which may be used for relative position determination. For example, one or more accelerometers and / or one or more gyroscopes of the IMU may detect the linear acceleration and rotational velocity of the device 200, respectively. The linear acceleration measurements and rotational velocity measurements of the device 200 may be integrated over time to determine the instantaneous direction of motion and displacement of the device 200. The instantaneous direction of motion and displacement may be integrated to track the position of the device 200. For example, a reference position of the device 200 at a certain moment may be determined, for example, using the SPS receiver 217 (and / or by some other means), and measurements obtained from the accelerometers and gyroscopes after that moment may be used for dead reckoning to determine the current position of the device 200 based on the movement (direction and distance) of the device 200 relative to the reference position.

[0037] The magnetometer can determine the magnetic field strength in different directions, which can be used to determine the orientation of the device 200. For example, the orientation can be used to provide a digital compass for the device 200. The magnetometer may include a two-dimensional magnetometer, which is configured to detect and provide an indication of the magnetic field strength in two orthogonal dimensions. The magnetometer may include a three-dimensional magnetometer, which is configured to detect and provide an indication of the magnetic field strength in three orthogonal dimensions. The magnetometer may provide a component for sensing a magnetic field and providing an indication of the magnetic field, for example, to the processor 210.

[0038] The transceiver 215 may include a wireless transceiver 240 and a wired transceiver 250 configured to communicate with other devices via wireless connections and wired connections, respectively. For example, the wireless transceiver 240 may include a wireless transmitter 242 and a wireless receiver 244 coupled to an antenna 246 for transmitting (e.g., on one or more uplink channels and / or one or more sidelink channels) and / or receiving (e.g., on one or more downlink channels and / or one or more sidelink channels) wireless signals 248 and converting signals from the wireless signals 248 to wired (e.g., electrical and / or optical) signals and from wired (e.g., electrical and / or optical) signals to the wireless signals 248. The wireless transmitter 242 includes appropriate components (e.g., a power amplifier and a digital-to-analog converter). The wireless receiver 244 includes appropriate components (e.g., one or more amplifiers, one or more frequency filters, and an analog-to-digital converter). The wireless transmitter 242 may include multiple transmitters that may be discrete components or combined / integrated components, and / or the wireless receiver 244 may include multiple receivers that may be discrete components or combined / integrated components. The wireless transceiver 240 may be configured to communicate signals (e.g., with the TRP and / or one or more other devices) according to a variety of radio access technologies (RATs), such as 5G New Radio (NR), GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), AMPS (Advanced Mobile Phone System), CDMA (Code Division Multiple Access), WcDMA (Wideband CDMA), LTE (Long Term Evolution), LTE Direct (LTE-D), 3GPP LTE-V2X (PC5), IEEE 802.11 (including IEEE 802.11p), Short-range wireless communication technology, Direct connection Short-range wireless communication technology, Short-range wireless communication technology, etc. The new radio may use millimeter wave frequencies and / or frequencies below 6 GHz. The wired transceiver 250 may include a wired transmitter 252 and a wired receiver 254 configured for wired communication, for example, a network interface that can be used to communicate with an NG-RAN (Next Generation - Radio Access Network) to transmit communications to the NG-RAN and receive communications from it. The wired transmitter 252 may include a plurality of transmitters that may be discrete components or combined / integrated components, and / or the wired receiver 254 may include a plurality of receivers that may be discrete components or combined / integrated components. The wired transceiver 250 may be configured, for example, for optical communication and / or electrical communication. The transceiver 215 may be communicatively coupled to the transceiver interface 214, for example, via an optical connection and / or an electrical connection. The transceiver interface 214 may be at least partially integrated with the transceiver 215. The wireless transmitter 242, the wireless receiver 244 and / or the antenna 246 may include multiple transmitters, multiple receivers and / or multiple antennas, respectively, for transmitting and / or receiving appropriate signals, respectively.

[0039] The user interface 216 may include one or more devices of several devices (such as, for example, a speaker, a microphone, a display device, a vibration device, a keyboard, a touch screen, etc.). The user interface 216 may include more than one of these devices. The user interface 216 may be configured to enable a user to interact with one or more applications hosted by the device 200. For example, the user interface 216 may store indications of analog and / or digital signals in the memory 211 in response to actions from the user to be processed by the DSP 231 and / or the general / application processor 230. Similarly, the application hosted on the device 200 may store indications of analog and / or digital signals in the memory 211 to present output signals to the user. The user interface 216 may include an audio input / output (I / O) device, which includes, for example, a speaker, a microphone, a digital-to-analog circuit, an analog-to-digital circuit, an amplifier and / or a gain control circuit (including more than one of these devices). Other configurations of audio I / O devices may be used. Additionally or alternatively, the user interface 216 may include one or more touch sensors that respond to touch and / or pressure on, for example, a keyboard and / or a touch screen of the user interface 216 .

[0040] The SPS receiver 217 (e.g., a global positioning system (GPS) receiver) may be capable of receiving and acquiring the SPS signal 260 via the SPS antenna 262. The SPS antenna 262 is configured to convert the SPS signal 260 from a wireless signal to a wired signal (e.g., an electrical signal or an optical signal) and may be integrated with the antenna 246. The SPS receiver 217 may be configured to process the acquired SPS signal 260 in whole or in part for estimating the location of the device 200. For example, the SPS receiver 217 may be configured to determine the location of the device 200 by performing trilateration using the SPS signal 260. The general / application processor 230, the memory 211, the DSP 231, and / or one or more special-purpose processors (not shown) may be used in conjunction with the SPS receiver 217 to process the acquired SPS signal in whole or in part and / or calculate the estimated location of the device 200. The memory 211 may store indications (e.g., measurements) of the SPS signals 260 and / or other signals (e.g., signals obtained from the wireless transceiver 240) for use in performing positioning operations. The general / application processor 230, the DSP 231, and / or one or more special-purpose processors, and / or the memory 211 may provide or support a location engine for use in processing measurements to estimate the location of the device 200.

[0041] The device 200 may include a camera 218 for capturing still or moving images. The camera 218 may include, for example, an imaging sensor (e.g., a charge coupled device or a CMOS (complementary metal oxide semiconductor) imager), a lens, analog-to-digital circuits, a frame buffer, etc. Additional processing, conditioning, encoding, and / or compression of signals representing captured images may be performed by the general / application processor 230 and / or the DSP 231. Additionally or alternatively, a video processor 233 may perform conditioning, encoding, compression, and / or manipulation of signals representing captured images. The video processor 233 may decode / decompress stored image data for presentation on a display device (not shown) (e.g., of the user interface 216).

[0042] Positioning device (PD) 219 may be configured to determine the position of device 200, the movement of device 200, and / or the relative position of device 200, and / or time. For example, PD 219 may communicate with SPS receiver 217 and / or include a portion or all of an SPS receiver. PD 219 may work appropriately with processor 210 and memory 211 to perform at least a portion of one or more positioning methods, although this specification may refer to PD 219 being configured to perform according to a positioning method or the PD performing according to a positioning method. PD 219 may be additionally or alternatively configured to determine the position of device 200 using ground-based signals (e.g., at least some of wireless signals 248) for trilateration, assisted acquisition, and use of SPS signals 260, or both. PD 219 may be configured to determine the position of device 200 based on the cell of a serving base station (e.g., a cell center) and / or another technology (such as E-CID). PD 219 may be configured to determine the location of device 200 using one or more images from camera 218 and image recognition combined with known locations of landmarks (e.g., natural landmarks (such as mountains) and / or artificial landmarks (such as buildings, bridges, streets), etc.). PD 219 may be configured to determine the location of device 200 using one or more other techniques (e.g., relying on the self-reported location of the UE (e.g., part of the UE's location beacon)), and may use a combination of techniques (e.g., SPS and ground positioning signals) to determine the location of device 200. PD 219 may include one or more of sensors 213 (e.g., gyroscopes, accelerometers, magnetometers, etc.), which may sense the orientation and / or movement of device 200 and provide an indication of the orientation and / or movement, and processor 210 (e.g., general / application processor 230 and / or DSP 231) may be configured to use the indication to determine the movement of device 200 (e.g., velocity vector and / or acceleration vector). PD 219 may be configured to provide an indication of uncertainty and / or error in the determined position and / or motion. The functionality of PD 219 may be provided in a variety of ways and / or configurations, such as by general / application processor 230, transceiver 215, SPS receiver 217, and / or another component of device 200, and may be provided by hardware, software, firmware, or various combinations thereof.

[0043] Also refer to Figure 3An example of a TRP 300 (of a base station such as a gNB (generic Node B) and / or ng-eNB (next generation evolved Node B)) may include a computing platform including a processor 310, a memory 311 including software (SW) 312, and a transceiver 315. The processor 310, the memory 311, and the transceiver 315 may be communicatively coupled to each other via a bus 320 (which may be configured, for example, for optical communication and / or electrical communication). One or more of the illustrated devices (e.g., a wireless transceiver) may be omitted from the TRP 300. The processor 310 may include one or more hardware devices, such as a central processing unit (CPU), a microcontroller, an application specific integrated circuit (ASIC), etc. The processor 310 may include a plurality of processors (e.g., including a general / application processor, a DSP, a modem processor, a video processor, and / or a sensor processor, such as Figure 2 ). The memory 311 may be a non-transitory storage medium that may include random access memory (RAM), flash memory, disk memory, and / or read-only memory (ROM), etc. The memory 311 may store software 312, which may be a processor-readable, processor-executable software code containing instructions that are configured to cause the processor 310 to perform various functions described herein when executed. Alternatively, the software 312 may not be directly executable by the processor 310, but may be configured to cause the processor 310 to perform these functions, for example, when compiled and executed.

[0044] The description herein may refer to processor 310 performing a function, but this includes other specific implementations, such as a specific implementation in which processor 310 executes software and / or firmware. The description herein may refer to processor 310 performing a function as shorthand for one or more of the processors included in processor 310 performing the function. This specification may refer to TRP 300 performing a function as shorthand for one or more appropriate components of TRP 300 (e.g., processor 310 and memory 311) performing the function. Processor 310 may include a memory with stored instructions in addition to and / or in lieu of memory 311. The functionality of processor 310 is discussed more fully below.

[0045] The transceiver 315 may include a wireless transceiver 340 and / or a wired transceiver 350 configured to communicate with other devices via wireless connections and wired connections, respectively. For example, the wireless transceiver 340 may include a wireless transmitter 342 and a wireless receiver 344 coupled to one or more antennas 346 for transmitting (e.g., on one or more uplink channels and / or one or more downlink channels) and / or receiving (e.g., on one or more downlink channels and / or one or more uplink channels) wireless signals 348 and converting signals from the wireless signals 348 to wired (e.g., electrical and / or optical) signals and from wired (e.g., electrical and / or optical) signals to the wireless signals 348. Thus, the wireless transmitter 342 may include multiple transmitters that may be discrete components or combined / integrated components, and / or the wireless receiver 344 may include multiple receivers that may be discrete components or combined / integrated components. The wireless transceiver 340 may be configured to communicate signals (e.g., with the device 200, one or more other UEs, and / or one or more other devices) according to a variety of radio access technologies (RATs), such as 5G New Radio (NR), GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), AMPS (Advanced Mobile Phone System), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), LTE Direct (LTE-D), 3GPP LTE-V2X (PC5), IEEE 802.11 (including IEEE 802.11p), Short-range wireless communication technology, Direct connection Short-range wireless communication technology, Short-range wireless communication technology, etc. The wired transceiver 350 may include a wired transmitter 352 and a wired receiver 354 configured for wired communication, for example, a network interface that can be used to communicate with the NG-RAN to transmit communications to, for example, the LMF and / or one or more other network entities and receive communications from them. The wired transmitter 352 may include multiple transmitters that may be discrete components or combined / integrated components, and / or the wired receiver 354 may include multiple receivers that may be discrete components or combined / integrated components. The wired transceiver 350 may be configured, for example, for optical communication and / or electrical communication.

[0046] Figure 3 The configuration of the TRP 300 shown is an example and not a limitation of the present disclosure (including the claims), and other configurations may be used. For example, the description herein discusses that the TRP 300 may be configured to perform several functions or that the TRP performs several functions, but one or more of these functions may be performed by the LMF and / or the device 200 (i.e., the LMF and / or the device 200 may be configured to perform one or more of these functions).

[0047] Also refer to Figure 4 , the server 400 (LMF is an example thereof) may include a computing platform including a processor 410, a memory 411 including software (SW) 412, and a transceiver 415. The processor 410, the memory 411, and the transceiver 415 may be communicatively coupled to each other via a bus 420 (the bus may be configured, for example, for optical communication and / or electrical communication). One or more of the devices shown (e.g., a wireless transceiver) may be omitted from the server 400. The processor 410 may include one or more hardware devices, such as a central processing unit (CPU), a microcontroller, an application specific integrated circuit (ASIC), etc. The processor 410 may include a plurality of processors (e.g., including a general / application processor, a DSP, a modem processor, a video processor, and / or a sensor processor, such as Figure 2 410). The memory 411 may be a non-transitory storage medium that may include random access memory (RAM), flash memory, disk storage, and / or read-only memory (ROM), etc. The memory 411 may store software 412, which may be a processor-readable, processor-executable software code containing instructions that are configured to cause the processor 410 to perform various functions described herein when executed. Alternatively, the software 412 may not be directly executable by the processor 410, but may be configured to cause the processor 410 to perform these functions, for example, when compiled and executed. The description herein may refer to the processor 410 performing functions, but this includes other specific implementations, such as specific implementations of the processor 410 executing software and / or firmware. The description herein may refer to the processor 410 performing functions as a shorthand for one or more processors included in the processor 410 performing the function. The description herein may refer to the server 400 performing functions as a shorthand for one or more appropriate components of the server 400 performing the function. The processor 410 may include a memory with stored instructions as a supplement and / or replacement for the memory 411. The functionality of processor 410 is discussed more fully below.

[0048] The transceiver 415 may include a wireless transceiver 440 and / or a wired transceiver 450 configured to communicate with other devices via wireless connections and wired connections, respectively. For example, the wireless transceiver 440 may include a wireless transmitter 442 and a wireless receiver 444 coupled to one or more antennas 446 for transmitting (e.g., on one or more downlink channels) and / or receiving (e.g., on one or more uplink channels) wireless signals 448 and converting signals from the wireless signals 448 to wired (e.g., electrical and / or optical) signals and from wired (e.g., electrical and / or optical) signals to the wireless signals 448. Thus, the wireless transmitter 442 may include multiple transmitters that may be discrete components or combined / integrated components, and / or the wireless receiver 444 may include multiple receivers that may be discrete components or combined / integrated components. The wireless transceiver 440 may be configured to communicate signals (e.g., with the device 200, one or more other UEs, and / or one or more other devices) according to a variety of radio access technologies (RATs), such as 5G New Radio (NR), GSM (Global System for Mobile Communications), UMTS (Universal Mobile Telecommunications System), AMPS (Advanced Mobile Phone System), CDMA (Code Division Multiple Access), WCDMA (Wideband CDMA), LTE (Long Term Evolution), LTE Direct (LTE-D), 3GPP LTE-V2X (PC5), IEEE 802.11 (including IEEE 802.11p), Short-range wireless communication technology, Direct connection Short-range wireless communication technology, Short-range wireless communication technology, etc. The wired transceiver 450 may include a wired transmitter 452 and a wired receiver 454 configured for wired communication, for example, a network interface that can be used to communicate with the NG-RAN to transmit and receive communications to, for example, the LMF 300 and / or one or more other network entities. The wired transmitter 452 may include multiple transmitters that may be discrete components or combined / integrated components, and / or the wired receiver 454 may include multiple receivers that may be discrete components or combined / integrated components. The wired transceiver 450 may be configured, for example, for optical communication and / or electrical communication.

[0049] This specification may refer to processor 410 performing a function, but this includes other implementations, such as implementations in which processor 410 executes software (stored in memory 411) and / or firmware. This specification may refer to server 400 performing a function as shorthand for one or more appropriate components of server 400 (e.g., processor 410 and memory 411) performing that function.

[0050] Figure 4The configuration of the server 400 shown is an example and not a limitation of the present disclosure (including the claims), and other configurations may be used. For example, the wireless transceiver 440 may be omitted. Additionally or alternatively, this specification discusses that the server 400 is configured to perform several functions or that the server performs several functions, but one or more of these functions may be performed by the TRP 300 and / or the device 200 (i.e., the TRP 300 and / or the device 200 may be configured to perform one or more of these functions).

[0051] refer to Figure 5 , device 500 includes a processor 510, a transceiver 520, a memory 530, and a sensor 540 that are communicatively coupled to each other via a bus 550. Even when referred to in the singular, processor 510 may include one or more processors, transceiver 520 may include one or more transceivers (e.g., one or more transmitters and / or one or more receivers), and memory 530 may include one or more memories. Device 500 may take any of a variety of forms, such as a mobile device, such as a vehicle UE (VUE). Device 500 may include Figure 5 The components shown, and may include one or more other components, such as Figure 2 Any of those components shown, such that device 200 may be an example of device 500. For example, processor 510 may include one or more of the components of processor 210. Transceiver 520 may include one or more of the components of transceiver 215, for example, wireless transmitter 242 and antenna 246, or wireless receiver 244 and antenna 246, or wireless transmitter 242, wireless receiver 244, and antenna 246. Additionally or alternatively, transceiver 520 may include wired transmitter 252 and / or wired receiver 254. Memory 530 may be configured similarly to memory 211, for example, including software having processor-readable instructions configured to cause processor 510 to perform functions.

[0052] This specification may refer to the processor 510 performing a function, but this includes other specific implementations, such as a specific implementation in which the processor 510 executes software (stored in the memory 530) and / or firmware. This specification may refer to the device 500 performing a function as shorthand for one or more appropriate components of the device 500 (e.g., the processor 510 and the memory 530) performing that function. The processor 510 (possibly in conjunction with the memory 530 and, where appropriate, the transceiver 520) may include an occupancy information unit 560 (which may include an ADAS (Advanced Driver Assistance System) for the VUE). The occupancy information unit 560 is discussed further below, and this specification may refer to the occupancy information unit 560 performing one or more functions, and / or may generally refer to the processor 510 or generally the device 500 as performing any of the functions of the occupancy information unit 560, where the device 500 is configured to perform those functions.

[0053] One or more functions performed by the device 500 (e.g., the occupancy information unit 560) may be performed by another entity. For example, sensor measurements (e.g., radar measurements, camera measurements (e.g., pixels, images)) and / or processed sensor measurements (e.g., camera images converted into bird's-eye view images) may be provided to another entity, such as the server 400, and the other entity may perform one or more functions discussed herein with respect to the occupancy information unit 560 (e.g., using machine learning to determine and / or apply an observation model, analyzing measurements from different sensors to determine a current occupancy grid, etc.).

[0054] Also refer to Figure 6, a geographic environment 600 (a driving environment in this example) includes a plurality of mobile wireless communication devices (here vehicles 601, 602, 603, 604, 605, 606, 607, 608, 609), a building 610, an RSU 612 (roadside unit), and a street sign 620 (e.g., a stop sign). The RSU 612 may be configured similarly to the TRP 300, but may have less functionality and / or a shorter range than the TRP 300 (e.g., a base station-based TRP). One or more of the vehicles 601 to 609 may be configured to perform autonomous driving. A vehicle that considers its perspective (e.g., for environmental assessment, autonomous driving, etc.) may be referred to as an observer vehicle or an ego vehicle. An ego vehicle such as the vehicle 601 may assess the area around the ego vehicle for one or more desired purposes (e.g., to facilitate autonomous driving). The vehicle 601 may be an example of the device 500. Vehicle 601 can divide the area around the self-vehicle into multiple sub-areas and evaluate whether an object occupies each sub-area, and if so, determine one or more characteristics of the object (e.g., size, shape (e.g., dimensions (possibly including height)), speed (speed and direction), object type (bicycle, car, truck, etc.), etc.).

[0055] Also refer to Figure 7 and Figure 8, an area 700, which in this example spans a portion of the environment 600, may be evaluated to determine an occupancy grid 800 (also referred to as an occupancy map) that indicates the type of occupants of each of a plurality of sub-areas of the area 700. For example, the area 700 may be divided into a grid (which may be referred to as an occupancy grid) having sub-areas 710 that may be of similar (e.g., identical) size and shape, or may have two or more sizes and / or shapes (e.g., where the sub-areas are smaller near an ego-vehicle (e.g., vehicle 601) and larger farther away from the ego-vehicle, and / or where the sub-areas near the ego-vehicle have a different shape than the shape of the sub-areas farther away from the ego-vehicle). Region 700 and grid 800 may be of regular shape (e.g., rectangle, triangle, hexagon, octagon, etc.) and / or may be divided into sub-regions of the same shape, regular shape for convenience (e.g., to simplify calculations), but regions / grids of other shapes (e.g., irregular shapes) and / or sub-regions (e.g., irregular shapes, multiple different regular shapes, or a combination of one or more irregular shapes and one or more regular shapes) may be used. For example, sub-region 710 may have a rectangular (e.g., square) shape. Region 700 may be any of a variety of sizes and have any of a variety of sub-regions of granularity. For example, region 700 may be a rectangle (e.g., square) with each side being approximately 100 m. As another example, although region 700 is shown as having sub-regions 710 that are squares of approximately 1 m on each side, sub-regions of other sizes may be used, including much smaller sub-regions. For example, square sub-regions of approximately 25 cm on each side may be used. In this example, region 700 is divided into M rows (here, parallel to Figure 8 24 rows of the x-axis indicated by N columns per row (here, parallel to Figure 8 23 columns on the y-axis shown in ).

[0056] Each of the sub-areas 710 may correspond to a corresponding cell 810 of the occupancy map, and information about what, if anything, occupies each of the sub-areas 710 may be obtained so as to populate the cells 810 of the occupancy map 800 with occupancy indications indicating the type of occupant of the sub-area corresponding to the cell. The information about what, if anything, occupies each of the sub-areas 710 may be obtained from one or more of a variety of sources. For example, the occupancy information may be obtained from one or more sensor measurements from one or more of the sensors 540 of the device 500. As another example, the occupancy information may be obtained by one or more other devices and communicated to the device 500. For example, one or more of the vehicles 602 to 609 may communicate the occupancy information to the vehicle 601, for example, via C-V2X communication. As another example, RSU 612 may collect occupancy information (e.g., from one or more sensors of RSU 612 and / or from communications with one or more of vehicles 602 to 609 and / or one or more other devices) and communicate the collected information to vehicle 601 (e.g., directly and / or via one or more network entities (e.g., TRPs), for example.

[0057] like Figure 8 As shown, each of the cells 810 may include occupancy information indicating the type of occupant of the sub-area 710 corresponding to the cell 810. As examples, the occupancy information may indicate that the corresponding sub-area 710 is occupied by a static object (S), or may indicate that the corresponding sub-area 710 is occupied by a moving or movable dynamic object (D), or may indicate that the corresponding sub-area 710 is occupied by free space and is therefore empty (E) or unoccupied, or may indicate that the occupancy of the corresponding sub-area is unknown (U), for example, where there is no information about possible occupants of the corresponding sub-area 710. Each of the cells 810 may include a corresponding probability that the cell 810 is static, dynamic, empty, or unknown, where the sum of the probabilities is 1. Figure 8 In the example shown, empty cells are not marked in the occupancy grid 800 for simplicity of the figure and readability of the occupancy grid 800 .

[0058] Constructing a dynamic occupancy grid (an occupancy grid with a dynamic occupancy type) may be helpful, or even necessary, for understanding the environment of the device (e.g., environment 600) to facilitate or even enable further processing. For example, the dynamic occupancy grid may be useful for predicting occupancy, motion planning, etc. The dynamic occupancy grid may include one or more cells of static occupancy type and / or one or more cells of dynamic occupancy type at any time. Dynamic objects may be represented as a collection of velocity vectors. For example, an occupancy grid cell may make some or all of the occupancy probabilities dynamic, and within the dynamic occupancy probabilities, there may be multiple (e.g., four) velocity vectors, each with a corresponding probability, which together sum up to the dynamic occupancy probability of the cell 810. The dynamic occupancy grid may be obtained, for example, by the occupancy information unit 560 by processing information from multiple sensors (e.g., in the sensor 540), such as from a radar system, a camera, etc.

[0059] Also refer to Fig. 9 , the occupancy information unit 560 may be configured to implement a Bayesian filter approach to predict an occupancy grid based on an observation model and to update the occupancy grid. Functional architecture 900 illustrates Bayesian filtering. Sensor measurements 910 (e.g., radar measurements) may be used by an observation model function 920 (also referred to as an ISM (Explanatory Structure Model) function) that uses the conditional probabilities of the radar measurements and the occupancy grid to determine a current occupancy grid 930 (also referred to as an observed occupancy grid). The occupancy information unit 560 may use the current occupancy grid 930 and the predicted occupancy grid 990 to perform an update function 940 of the predicted occupancy grid 990 to produce an updated occupancy grid 950, which the occupancy information unit 560 may perform a resampling function 960 on the updated occupancy grid to produce what then becomes a previous occupancy grid 970, which may be provided to any appropriate user of the updated occupancy grid (e.g., an autonomous driving application, a motion planner, etc.) and used for prediction of the next occupancy grid. The occupancy information unit 560 may use the previous occupancy grid 970 in a prediction function 980 to determine a predicted occupancy grid 990. The occupancy information unit 560 may perform the prediction function 980 according to the following formula:

[0060]

[0061] Among them G k is an NxN occupancy grid at time k (ie, the current occupancy grid 930) and is a dynamic occupancy grid (DOGMa (dynamic occupancy grid map)), and can be implemented as a particle filter, G k-1 is the occupied grid at time k-1 (i.e., the previously occupied grid 970), u k is the motion data, dG kis a differential element, bel(G k-1 ) is an update of the previous occupancy grid, and p indicates a probability. The occupancy information unit 560 may perform an update function 940 for the predicted occupancy grid 990 according to the following formula:

[0062]

[0063] Where p(R k |G k ) is the sensor measurement at time k (in this example, the radar measurement R at time k k ) is the observation model, and η is a normalizing constant.

[0064] Also refer to Fig.10 , the occupancy information unit 560 may be configured to implement a Bayesian filter method to predict an occupancy grid and update the occupancy grid based on an observation model that may use measurements from one or more of the multiple sensors. The functional architecture 1000 illustrates a Bayesian filter method implemented by the occupancy information unit 560 for sensor measurements from multiple sensors. The occupancy information unit 560 may execute an update function 1040, a resampling function 1060, and a prediction function 1080 similar to the update function 940, the resampling function 960, and the prediction function 980 described above. In some embodiments, the prediction function 1080 and the update function 1040 may be replaced with an RNN (recurrent neural network) / LSTM (long short-term memory) / transformer architecture. Sensor measurements 1011, 1012 from multiple sensors (e.g., radar measurements, camera measurements (pixel measurements)) may be used in the observation model function 1020 implemented by the occupancy information unit 560 to determine the current occupancy grid 1030. The observation model function 1020 may include machine learning (e.g., may include a neural network (e.g., a CNN (convolutional neural network))) to develop an observation model and apply the observation model to the sensor measurements 1011, 1012 to determine the current occupancy grid 1030. The occupancy information unit 560 may implement the neural network with respect to some sensor measurements but not for other sensor measurements, for example, with respect to camera measurements but not with respect to radar measurements (using classical methods for radar measurements), or vice versa. The occupancy information unit 560 may determine the current occupancy grid 1030 as p(R k , C k |G k ), and various architectures may be implemented to determine the current occupied grid 1030. For example, the occupancy information unit 560 may determine the current occupied grid 1030 as p(R k , C k |C k ):

[0065] p(R k , C k |G k )=p(R k |G k )p(C k |R k , G k ) (3)

[0066] p(R k , C k |G k )=p(C k |G k )p(R k |C k , G k ) (4)

[0067] p(R k , C k |G k )=p(R k |G k )p(C k |G k ) (5)

[0068] Where R k is the radar frame at time k, C k is the camera image at time k. A radar frame at time k may consist of detection pings, where each ping may have attributes such as location, velocity, RCS (radar cross section), SNR (signal-to-noise ratio), confidence level, etc. Each camera frame may be a grid (e.g., a rectangular grid) of pixels representing RGB (red / green / blue) information (e.g., intensity). For equation (5), assume that G k is a sufficient statistic. In another embodiment, the occupancy information unit 560 may evaluate measurements from multiple sensors and selectively use measurements from one sensor or another, or a combination of these measurements. For example, if the radar measurement indicates a strong probability that an object is at a particular location (e.g., 90%), but the camera measurement indicates a weak probability that the object is at that location (e.g., 10%), then the camera measurement may be discarded. In another example, if both the radar measurement and the camera measurement indicate a significant probability that an object is at a location (e.g., 40% and 60%), then the occupancy information unit 560 may combine these measurements in some manner, such as a weighted combination of these measurements.

[0069] Also refer to Fig.11, equation (3) may be implemented using functional architecture 1100 for multi-sensor measurement occupancy grid development and use. A specific implementation of equation (3) may provide for joint processing of measurements from different sensors. In this example and other examples discussed herein, radar points and camera images are used as examples of sensor measurements and radar systems and cameras are used as examples of sensors, but the discussion is applicable to one or more other sensors and corresponding sensor measurements. Additionally, in this example and other examples discussed herein, two sensors and corresponding measurements are used, but more than two sensors may be used. For example, one or more additional observation model functions may be implemented, for example, to take into account other than Fig.11 Other sensor measurements beyond those shown in the observation model function, and / or other combinations of sensor measurements. For example, the observation model function may consider measurements from a third sensor, the observation model may consider measurements from the camera and the third sensor, and / or the observation model may consider measurements from all available sensors, etc.

[0070] For the functional architecture 1100, the occupancy information unit 560 can be configured to implement an observation model function 1110 to apply an observation model to the radar points 1101 to determine a single sensor occupancy grid 1115 (here, a radar-based occupancy grid). The occupancy information unit 560 can also be configured to implement an observation model function 1120 that can use machine learning to develop p(C k |R k , G k ) and applies it to radar points 1101 and camera images 1102 to determine the multi-sensor occupancy grid 1125. The expression p(C k |R k , G k ) indicates the radar frame R at a given observation k and the grid state G k The camera image C is captured under the condition of kThe observation model of the probability of the camera image varies based on the grid state and the radar frame. For example, if all cells in the grid are empty, the probability of observing a camera image including a vehicle will be very low, and vice versa. The occupancy information unit 560 may combine the single sensor occupancy grid 1115 with the multi-sensor occupancy grid 1125, for example, by multiplying the single sensor occupancy grid 1115 with the multi-sensor occupancy grid 1125. As another example, the occupancy information unit 560 may selectively use one or more portions of the single sensor occupancy grid 1115 and / or selectively use one or more portions of the multi-sensor occupancy grid 1125 to determine the current occupancy grid for use in the update function 1140. For example, one or more portions of the single sensor occupancy grid 1115 and one or more portions of the multi-sensor occupancy grid 1125 may be used to populate the current occupancy grid, wherein each cell of the current occupancy grid is from one of the occupancy grids 1115, 1125. As another example, one or more cells in the cells of the current occupancy grid may each be determined using a corresponding cell of the single sensor occupancy grid 1115 and a corresponding cell of the multi-sensor occupancy grid 1125 (e.g., by multiplying the probabilities of the corresponding cells). The current occupancy grid and the predicted occupancy grid may be applied to an update function 1140, which may be similar to the update function 940, for example, the current occupancy grid may be multiplied by the predicted occupancy grid. The resampling function 1160 and the prediction function 1180 may be similar to the resampling function 960 and the prediction function 980.

[0071] The occupancy information unit 560 may be configured to perform a non-parametric camera image to BEV (bird's eye view) conversion. For example, the occupancy information unit 560 may be configured to perform a non-parametric camera image to BEV conversion using IPM (inverse perspective mapping) or using a flat road assumption. As another example, the occupancy information unit 560 may be configured to implement a data-assisted and parameterized (e.g., downlink-based) camera image to BEV conversion, such as by developing a BEV conversion model (e.g., using machine learning) using camera image data collected while driving on a road.

[0072] The functional architecture 1100 can be robust to sensor failures. For example, where the occupancy information unit 560 is configured to implement the update function 1140 to selectively use the single-sensor occupancy grid and / or the multi-sensor occupancy grid 1125, or to selectively use one or more portions of the grid 1115 and / or one or more portions of the grid 1125, the functional architecture 1100 can adapt in accordance with sensor failures. For example, the occupancy information unit 560 can avoid using measurements corresponding to a faulty sensor and / or information derived from the faulty sensor.

[0073] Also refer to Fig.12 , equation (4) may be implemented using functional architecture 1200 for multi-sensor measurement occupancy grid development and use. For functional architecture 1200, occupancy information unit 560 may be configured to implement observation model function 1210 to apply an observation model to camera image 1201 to determine a single sensor occupancy grid 1215 (here, a camera-based occupancy grid). Occupancy information unit 560 may also be configured to implement observation model function 1220, which may use machine learning to develop p(R k |C k , G k ) and applies it to the camera image 1201 and the radar point 1202 to determine the multi-sensor occupancy grid 1225. The occupancy information unit 560 can combine the single sensor occupancy grid 1215 with the multi-sensor occupancy grid 1225, for example, as discussed with respect to the single sensor occupancy grid 1115 and the multi-sensor occupancy grid 1125. The occupancy information unit 560 can implement an update function 1240 similar to the update function 1140 or the update function 940. The resampling function 1260 and the prediction function 1280 can be similar to the resampling function 960 and the prediction function 980. Like the functional architecture 1100, the functional architecture 1200 can be robust to sensor failures.

[0074] Also refer to Fig.13 , the functional architecture 1300 may be used to implement equation (5) by performing a camera image to BEV conversion for multi-sensor measurement occupancy grid development and use. For the functional architecture 1300, the occupancy information unit 560 may implement an observation model function 1310 similar to the observation model function 1110 to operate on the radar points 1301 to determine a radar-based occupancy grid 1315, and may implement a resampling function 1360 and a prediction function 1380 similar to the resampling function 1160 and the prediction function 1180, respectively. Also for the functional architecture 1300, the occupancy information unit 560 may be configured to implement a BEV function 1320 to convert the camera image 1302 into a bird's-eye view depiction of the environment captured by the camera. For example, the occupancy information unit 560 may be configured to segment the camera image 1302 into segmented images and apply a probabilistic projection to the segmented images to derive a BEV. The occupancy information unit 560 may implement a DNN (deep neural network) to perform an observation model function 1322 to determine the observation model p(C k |G k ), where C k is the BEV transformed image. The occupancy information unit 560 may apply the observation model function 1322 to the BEV to determine the camera-based occupancy grid 1325. The occupancy information unit 560 may implement an update function 1340, for example, multiplying the radar-based occupancy grid 1315 and the camera-based occupancy grid 1325.

[0075] Also refer to Fig.14 , the functional architecture 1400 can be used to implement equation (5) for multi-sensor measurement occupancy grid development and use by utilizing a grid-to-image conversion. For the functional architecture 1400, the occupancy information unit 560 can implement an observation model function 1410 similar to the observation model function 1110 to operate on the radar point 1401 to determine a radar-based occupancy grid 1415, and can implement a resampling function 1460 and a prediction function 1480 similar to the resampling function 1160 and the prediction function 1180, respectively. Also for the functional architecture 1400, the occupancy information unit 560 can be configured to implement the observation model function 1420 by implementing a DNN to determine a camera-based occupancy grid 1415 based on a grid-to-image conversion. The occupancy information unit 560 can implement an update function 1440, for example, multiplying the radar-based occupancy grid 1415 and the camera-based occupancy grid 1425.

[0076] Various architectures may be used for the observation model function 1420. For example, the occupancy information unit 560 may learn intrinsic camera characteristics (i.e., camera characteristics (e.g., lens quality, lens shape, light sensor quality, light sensor density, etc.) that affect the captured image (e.g., the quality of the captured image). The occupancy information unit 560 may, for example, apply a CNN to the captured image to perform the loss calculation. The CNN may implicitly transform the image into a grid frame. As another example, the occupancy information unit 560 may apply a CNN to the captured image and apply the transformation to the grid (e.g., through a VPN (view parser network)) to determine the loss calculation. As another example, the occupancy information unit 560 may apply a CNN encoder to the captured image, then apply the transformation to the grid, and then apply a CNN decoder to determine the loss calculation. For example, a PYVA (focused view projection) function may use a transformer to perform the transformation to the grid. As another example, the occupancy information unit 560 may use knowledge of intrinsic camera characteristics and extrinsic features (i.e., features that are not intrinsic to the camera that may affect the captured image (e.g., the shape of the glass (e.g., windshield) by which the camera captures the image)). For example, the occupancy information unit 560 may apply an IPM to the camera image and then apply a CNN including applying a weighted head to determine the loss calculation. A CAM2BEV conversion may be performed that pairs the IPM with a transformer (which may improve the accuracy of this technique). As another example, with knowledge of the intrinsic and extrinsic features, the occupancy information unit 560 may apply a CNN to the camera image and apply a weighted head (discussed further below) to determine the loss calculation using a grid-to-image frame transformation.

[0077] Also refer to Fig.15, the occupancy information unit 560 may be configured to determine the camera-based occupancy grid based on the grid-to-image conversion. The occupancy information unit 560 may be configured to calculate the observation model p(C k |G k，i ),in

[0078] p(C k |G k，i )=p(C k |TG2I(G k，i )) (6)

[0079] The condition is that the mapping from grid to image is reversible. Fig.15 As shown, the observation model training method 1500 begins with the occupancy information unit 560 applying the camera image 1510 to the CNN 1520 to determine a set 1530 of arrays 1535 1 - 1535n, the set comprising modified images corresponding to the camera image 1510 (and hence the camera (sensor) measurements). Array 1535 1 Each array in -1535n may have a lower resolution than camera image 1510. For example, camera image 1510 may include a 1024x512x3 pixel array that includes a 1024x512 array of three pixels each for red, green, and blue, and array 1535 1 Each array in -1535n may include a reduced resolution array of 128x62 cells. Array 1535 1 Each array in -1535n may correspond to a different mechanism for deriving the corresponding array from the camera image 1510. For example, different arrays may be determined using different frequency filters, e.g., one array is determined using an LPF (low pass filter) and another array is determined using an HPF (high pass filter), or a combination thereof. The arrays may be determined using other differentiation techniques. Each cell in each array will have a corresponding probability value. The occupancy information unit 560 may use the known occupancy grid 1540 corresponding to the camera image 1510 to perform a head training function 1550 to train heads, e.g., heads 1551, 1552, for use in deriving the corresponding array from the camera image 1510. 1 -1535n is converted to an expected occupancy grid 1560. The probability of the known occupancy grid 1540 will be 1 or 0 because the true value is known, for example, from LiDAR and / or one or more other technologies. The head is a weight vector of dimension 1xn that is part of a neural network implemented by the occupancy information unit 560 (e.g., part of CNN 1520). The head is for array 1535 1 -1535 n Each array in provides a weight.

[0080] The occupancy information unit 560 may execute the head training function 1550 to determine the value of the head so that when the head is applied to the array 1535 1 -1535 n , it is expected that the occupancy grid 1560 will substantially match the known occupancy grid 1540. To perform the head training function 1550, the occupancy information unit 560 may determine the grid-to-image transformation, and then determine the image-to-grid transformation as the inverse of the grid-to-image transformation. The occupancy information unit 560 may determine the grid-to-image transformation from the known occupancy grid 1540 to the array 1535. 1 -1535 n and the inverse of the transformation is determined as the value for transforming the array 1535 1 -1535 n An image-to-grid transformation (corresponding to camera image 1510) to expected occupancy grid 1560. The probabilities of cells of expected occupancy grid 1560 can be array 1535 1 -1535 n The pixels in camera image 1510 may be selected based on the transformation performed by CNN 1520.

[0081] Headers may be non-uniformly mapped to array 1535 1 -1535 n (and thus mapped to pixels of camera image 1510) and / or mapped to expected occupancy grid 1560. For example, array 1535 corresponding to nearby objects (and multiple pixels in camera image 1510) 1 -1535 n Multiple cells in each array may be mapped to a single cell of the intended occupancy grid, and / or array 1535 1 -1535 n A single cell of each array in (or even a single pixel of camera image 1510) may be mapped to multiple cells of expected occupancy grid 1560. Thus, a single header may be applied to array 1535. 1 -1535 n The plurality of cells in each array, and / or the header may be array 1535 1 -1535 n Each single cell of the array is mapped to multiple cells of the expected occupation grid 1560.

[0082] The head can be determined to map directly from the camera image 1510 to the expected occupancy grid 1560. Compared to mapping directly from the camera image 1510 to the expected occupancy grid 1560, using the data from the array 1535 1-1535 n A head mapped to the expected occupancy grid may retain more information from the camera image 1510 .

[0083] During the inference phase, the occupancy information unit 560 determines the array 1535 1 -1535 n , and applies the head determined during training to the array 1535 1 -1535 n To determine the expected occupancy grid 1560 which will be based on the camera's occupancy grid 1425 .

[0084] Reference again Fig.14 , the occupancy grid may be updated using the camera-based occupancy grid 1425 based on the grid-to-image conversion. The prediction function 1480 may perform a prediction of the grid state to provide a predicted occupancy grid 1490 to the update function 1440. Each grid cell may include multiple state values, for example, four state values ​​corresponding to static, dynamic, empty, and unknown, where the dynamic state may have multiple sub-states (e.g., probabilities of different velocity vectors). The occupancy information unit 560 may calculate the p(C) of each grid cell by applying the observation model function 1320. k |G k，i ) (e.g., four values ​​for each grid cell) to infer the camera image 1402. The occupancy information unit 560 may generate an updated occupancy grid by multiplying the predicted occupancy grid 1490 by the camera-based occupancy grid 1425 and the radar-based occupancy grid 1415 to calculate the point-by-point product for each grid cell. The occupancy information unit 560 may normalize the probability of each grid cell of the updated occupancy grid so that the sum of the probabilities for each grid cell is equal to 1. The updated occupancy grid may be used to predict the next predicted occupancy grid, and so on.

[0085] refer to Fig.16 , and further reference Figures 1 to 15 , the occupancy grid determination method 1600 includes the stages shown. However, the method 1600 is an example and not a limitation. The method 1600 can be modified, for example, by adding, removing, rearranging, combining, performing one or more stages concurrently, and / or splitting one or more stages into multiple stages.

[0086] At stage 1610, method 1600 includes determining a predicted occupancy grid based on the previous occupancy grid, the predicted occupancy grid including a plurality of first cells corresponding to a sub-region of the region, each of the plurality of first cells including a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell. For example, occupancy information unit 560 (or another entity such as server 400) may execute any of prediction functions 1080, 1180, 1280, 1380, 1480 to determine a predicted occupancy grid (e.g., occupancy map 800). Processor 510 (possibly in combination with memory 530) or processor 410 (possibly in combination with memory 411) may include components for determining a predicted occupancy grid.

[0087] At stage 1620, method 1600 includes determining an observed occupancy grid using machine learning and based on the first sensor measurements, the observed occupancy grid including a plurality of second cells corresponding to the sub-regions of the region. For example, occupancy information unit 560 (or another entity) may execute any of observation model functions 1020, 1120, 1220, 1322, 1420 to determine an observed occupancy grid, such as any of corresponding occupancy grids 1030, 1125, 1225, 1325, 1425. Occupancy information unit 560 may also determine another observed occupancy grid, such as any of corresponding occupancy grids 1115, 1215, 1315, 1415, without using machine learning (e.g., using classical methods). Processor 510 (possibly in combination with memory 530) or processor 410 (possibly in combination with memory 411) may include components for determining an observed occupancy grid.

[0088] At stage 1630, method 1600 includes determining an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid. For example, occupancy information unit 560 (or other entity) may perform any of update functions 1040, 1140, 1240, 1340, 1440 based on occupancy grid 1030, or occupancy grid 1125 (and possibly occupancy grid 1115), or occupancy grid 1225 (and possibly occupancy grid 1215), or occupancy grid 1325 (and possibly occupancy grid 1315), or occupancy grid 1425 (and possibly occupancy grid 1415). Processor 510 (possibly in combination with memory 530) or processor 410 (possibly in combination with memory 411) may include means for determining an updated occupancy grid.

[0089] Implementations of method 1600 may include one or more of the following features. In an example implementation, method 1600 includes: obtaining a first sensor measurement from a first sensor; and obtaining a second sensor measurement from a second sensor, wherein determining the observed occupancy grid includes using, for each second cell of the plurality of second cells, a corresponding first portion of first information corresponding to the first sensor measurement, a corresponding second portion of second information corresponding to the second sensor measurement, or a combination thereof. The first information may be a sensor measurement (e.g., a camera measurement for an image) or information derived from these sensor measurements (e.g., a BEV). The occupancy information unit 560 (or other entity) may obtain the first sensor measurement and the second sensor measurement, e.g., sensor measurements 1011, 1012 (e.g., radar points and camera images, respectively). The occupancy information unit 560 (or other entity) may analyze the sensor measurements and not use measurements from one sensor and therefore only use measurements from another sensor, or use a combination of measurements from sensors (e.g., using measurements from one sensor or another sensor for a given cell of the observed occupancy grid, or combining measurements from different sensors to determine a given cell of the observed occupancy grid). The processor 510 (possibly in combination with the memory 530, in combination with the sensor 540) or the processor 410 (possibly in combination with the memory 411 and in combination with the wired receiver 454 and / or the wireless receiver 444 and the antenna 446) may include means for obtaining first sensor measurements and means for obtaining second sensor measurements. In a further example implementation, the first information includes the first sensor measurements and the second information includes the second sensor measurements, and wherein determining the observed occupancy grid includes using at least a first sensor measurement of the first sensor measurements, at least a second sensor measurement of the second sensor measurements, or a combination thereof for each second cell of the plurality of second cells. For example, for the observation model function 1020, the occupancy information unit 560 may select one or more sensor measurements of the sensor measurements 1011, one or more sensor measurements of the sensor measurements 1012, or a combination of at least one sensor measurement of the sensor measurements 1011 and at least one sensor measurement of the sensor measurements 1012 for determining a given occupied grid cell. In another further example implementation, the method 1600 includes deriving the first information from the first sensor measurements and deriving the second information from the second sensor measurements. For example, in a further example implementation, the first information includes a bird's-eye view of the area.In another further example implementation, the first information includes multiple first probability indications, each indicating a first probability of a first corresponding possible type of occupant of a corresponding one of the sub-areas, and the second information includes multiple second probability indications, each indicating a second probability of a second corresponding possible type of occupant of a corresponding one of the sub-areas. For example, the first information may include one of the occupancy grids 1125, 1225, 1325, 1425, and the second information may include one of the occupancy grids 1115, 1215, 1315, 1415, and the update function 1140, 1240, 1340, 1440 may use one or more cells of the occupancy grids 1115, 1215, 1315, 1415, or one or more cells of the occupancy grids 1125, 1225, 1325, 1425, or one or more cells of the occupancy grids 1115, 1215, 1315, 1415 and one or more cells of the occupancy grids 1125, 1225, 1325, 1425 (e.g., multiplying the corresponding cells) for any given cell of the updated occupancy grid. In another further example implementation, method 1600 includes: determining an occupancy grid to image transformation by machine learning; determining an image to occupancy grid transformation based on the occupancy grid to image transformation; and determining the first information by applying the image to occupancy grid transformation to third information corresponding to the images measured by the first sensor, the first sensor including a camera. For example, as with respect to. Fig.15 As discussed, the occupancy information unit 560 (or other entity) may determine a transformation of the occupancy grid to the image based on the known occupancy grid 1540 in order to generate the array 1535 with acceptable accuracy. 1 -1535 n. The inverse of the occupancy grid to image transform may be determined as an image to occupancy grid transform, and the first information (e.g., occupancy grid 1425) may be determined by applying the image to occupancy grid transform to third information (e.g., a new set of arrays derived from the new camera image). Alternatively, the transform may go directly to and from the camera image 1510, such that the first information may be determined by applying the image to occupancy grid transform to the camera image. The processor 510 (possibly in combination with the memory 530) or the processor 410 (possibly in combination with the memory 411) may include means for determining an occupancy grid to image transform, means for determining an image to occupancy grid transform, and means for determining the first information. In a further example implementation, the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupied grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and wherein: the occupancy grid to image transform maps at least two of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or the occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions; or the image to occupancy grid transform maps at least two third information regions of the plurality of third information regions to a single occupied grid cell of the plurality of occupied grid cells; or the image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupied grid cells of the plurality of occupied grid cells; or a combination of two or more thereof. Thus, there is a non-uniform mapping between the occupancy grid and the third information. For example, as regards Fig.15 As discussed, in the known occupied grid 1540 and array 1535 1 -1535 n There may be a non-uniform mapping (many-to-one mapping (of cells and / or pixels) or one-to-many mapping (of cells and / or pixels)) between the camera images 1510, and / or there may be a non-uniform mapping (many-to-one ... 1 -1535 n There may be a non-uniform mapping between the (or camera image 1510 ) and the expected occupancy grid 1560 .

[0090] Additionally or alternatively, a specific implementation of method 1600 may include one or more of the following features. In an example implementation, the multiple predicted probability indications each indicate the plausibility of an occupant of the corresponding possible type of the corresponding first cell actually occupying the corresponding first cell. For example, the predicted probability indication may indicate the probability of a cell being empty, unknown, occupied by a static object, or occupied by a dynamic object (and possibly sub-probabilities of different dynamic characteristics (e.g., different velocity vectors (of different directions and / or speeds))).

[0091] Specific implementation examples

[0092] Specific implementation examples are provided in the following numbered clauses.

[0093] Clause 1. A device, comprising:

[0094] Memory; and

[0095] a processor communicatively coupled to the memory and configured to:

[0096] determining a predicted occupancy grid based on the previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell;

[0097] determining, using machine learning and based on the first sensor measurements, an observed occupancy grid comprising a plurality of second cells corresponding to the sub-region of the region; and

[0098] An updated occupancy grid is determined based on the observed occupancy grid and the predicted occupancy grid.

[0099] Clause 2. The apparatus of clause 1, further comprising:

[0100] a first sensor configured to obtain the first sensor measurement;

[0101] and

[0102] a second sensor configured to obtain a second sensor measurement;

[0103] wherein the processor is communicatively coupled to the first sensor and the second sensor, and wherein to determine the observed occupancy grid, the processor is configured to use, for each second cell of the plurality of second cells, a corresponding first portion of first information corresponding to the first sensor measurement, a corresponding second portion of second information corresponding to the second sensor measurement, or a combination thereof.

[0104] Clause 3. An apparatus according to clause 2, wherein the first information includes the first sensor measurement and the second information includes the second sensor measurement, and wherein in order to determine the observed occupancy grid, the processor is configured to use at least a first sensor measurement of the first sensor measurement, at least a second sensor measurement of the second sensor measurement, or a combination thereof for each second cell of the plurality of second cells.

[0105] Clause 4. The apparatus of Clause 2, wherein the first information is derived from the first sensor measurements and the second information is derived from the second sensor measurements.

[0106] Clause 5. The apparatus of clause 4, wherein the first information comprises a bird's-eye view of the area.

[0107] Clause 6. An apparatus according to clause 4, wherein the first information includes a plurality of first probability indications, each indicating a first probability of a first corresponding possible type of occupant of a corresponding one of the sub-areas, and the second information includes a plurality of second probability indications, each indicating a second probability of a second corresponding possible type of occupant of a corresponding one of the sub-areas.

[0108] Clause 7. The apparatus of clause 2, wherein the processor is further configured to:

[0109] Determine the occupancy grid-to-image transformation through machine learning;

[0110] determining an image-to-occupancy grid transform based on the occupancy grid-to-image transform; and determining the first information by applying the image-to-occupancy grid transform to third information corresponding to an image corresponding to measurements by the first sensor, the first sensor comprising a camera.

[0111] Clause 8. The apparatus of clause 7, wherein the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupancy grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and wherein:

[0112] The occupancy grid to image transform maps at least two occupied grid cells of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or

[0113] The occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions

[0114] domain; or

[0115] The image to occupancy grid transform maps at least two of the plurality of third information regions to a single occupancy grid cell of the plurality of occupancy grid cells.

[0116] or

[0117] The image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupancy grid cells of the plurality of occupancy grid cells

[0118] or

[0119] a combination of two or more of them;

[0120] Thereby there is a non-uniform mapping between the occupancy grid and the third information.

[0121] Clause 9. The apparatus of clause 1, wherein the plurality of predicted probability indications each indicate a plausibility of an occupant of the respective possible type of the respective first cell actually occupying the respective first cell.

[0122] Clause 10. A method for determining an occupied grid, the method comprising:

[0123] determining a predicted occupancy grid based on the previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell;

[0124] Determining an observed occupancy grid using machine learning and based on the first sensor measurements, the observed occupancy grid comprising a plurality of second

[0125] Cells; and

[0126] An updated occupancy grid is determined based on the observed occupancy grid and the predicted occupancy grid.

[0127] Clause 11. The method for determining an occupied grid according to clause 10, further comprising:

[0128] obtaining the first sensor measurement from a first sensor; and

[0129] obtaining a second sensor measurement from a second sensor;

[0130] Wherein determining the observed occupancy grid comprises using, for each second cell of the plurality of second cells, a respective first portion of first information corresponding to the first sensor measurement, a respective second portion of second information corresponding to the second sensor measurement, or a combination thereof.

[0131] Clause 12. An occupancy grid determination method according to clause 11, wherein the first information includes the first sensor measurement and the second information includes the second sensor measurement, and wherein determining the observed occupancy grid includes using at least a first sensor measurement of the first sensor measurement, at least a second sensor measurement of the second sensor measurement, or a combination thereof for each second cell of the plurality of second cells.

[0132] Clause 13. The occupancy grid determination method of Clause 11, further comprising deriving the first information from the first sensor measurements and deriving the second information from the second sensor measurements.

[0133] Clause 14. The occupancy grid determination method of clause 13, wherein the first information comprises a bird's eye view of the area.

[0134] Clause 15. An occupancy grid determination method according to Clause 13, wherein the first information includes multiple first probability indications, each indicating a first probability of a first corresponding possible type of occupant of a corresponding one of the sub-areas, and the second information includes multiple second probability indications, each indicating a second probability of a second corresponding possible type of occupant of a corresponding one of the sub-areas.

[0135] Clause 16. The method for determining an occupied grid according to clause 11, further comprising:

[0136] Determine the occupancy grid-to-image transformation through machine learning;

[0137] determining an image-to-occupancy grid transform based on the occupancy grid-to-image transform; and determining the first information by applying the image-to-occupancy grid transform to third information corresponding to an image corresponding to measurements by the first sensor, the first sensor comprising a camera.

[0138] Clause 17. An occupancy grid determination method according to clause 16, wherein the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupancy grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and wherein:

[0139] The occupancy grid to image transform maps at least two occupied grid cells of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or

[0140] The occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions

[0141] domain; or

[0142] The image to occupancy grid transform maps at least two of the plurality of third information regions to a single occupancy grid cell of the plurality of occupancy grid cells.

[0143] or

[0144] The image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupancy grid cells of the plurality of occupancy grid cells

[0145] or

[0146] a combination of two or more of them;

[0147] Thereby there is a non-uniform mapping between the occupancy grid and the third information.

[0148] Clause 18. The occupancy grid determination method of Clause 10, wherein the plurality of predicted probability indications each indicate plausibility of an occupant of the respective possible type of the respective first cell actually occupying the respective first cell.

[0149] Clause 19. An apparatus, comprising:

[0150] means for determining a predicted occupancy grid based on a previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell;

[0151] means for determining, using machine learning and based on first sensor measurements, an observed occupancy grid comprising a plurality of second cells corresponding to the sub-region of the region; and

[0152] Means for determining an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid.

[0153] Clause 20. The apparatus of clause 19, further comprising:

[0154] means for obtaining said first sensor measurement from a first sensor; and

[0155] means for obtaining a second sensor measurement from a second sensor;

[0156] Wherein the means for determining the observed occupancy grid comprises means for using, for each second cell of the plurality of second cells, a respective first portion of first information corresponding to the first sensor measurement, a respective second portion of second information corresponding to the second sensor measurement, or a combination thereof.

[0157] Clause 21. An apparatus according to clause 20, wherein the first information includes the first sensor measurements and the second information includes the second sensor measurements, and wherein the means for determining the observed occupancy grid includes means for using at least a first sensor measurement of the first sensor measurements, at least a second sensor measurement of the second sensor measurements, or a combination thereof for each second cell of the plurality of second cells.

[0158] Clause 22. The apparatus of clause 20, further comprising means for deriving the first information from the first sensor measurements and means for deriving the second information from the second sensor measurements.

[0159] Clause 23. The apparatus of clause 22, wherein the first information comprises a bird's-eye view of the area.

[0160] Clause 24. An apparatus according to clause 22, wherein the first information comprises a plurality of first probability indications each indicating a first probability of a first corresponding possible type of occupant of a corresponding one of the sub-areas, and the second information comprises a plurality of second probability indications each indicating a second probability of a second corresponding possible type of occupant of a corresponding one of the sub-areas.

[0161] Clause 25. The apparatus of clause 20, further comprising:

[0162] a component for determining an occupancy grid-to-image transformation by machine learning;

[0163] means for determining an image to occupancy grid transform based on said occupancy grid to image transform; and

[0164] Means for determining the first information by applying the image-to-occupancy grid transform to third information corresponding to an image corresponding to measurements of the first sensor, the first sensor comprising a camera.

[0165] Clause 26. The apparatus of clause 25, wherein the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupancy grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and wherein:

[0166] The occupancy grid to image transform maps at least two occupied grid cells of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or

[0167] The occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions

[0168] domain; or

[0169] The image to occupancy grid transform maps at least two of the plurality of third information regions to a single occupancy grid cell of the plurality of occupancy grid cells.

[0170] or

[0171] The image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupancy grid cells of the plurality of occupancy grid cells

[0172] or

[0173] a combination of two or more of them;

[0174] Thereby there is a non-uniform mapping between the occupancy grid and the third information.

[0175] Clause 27. The apparatus of clause 19, wherein the plurality of predicted probability indications each indicate a plausibility of an occupant of the respective possible type of the respective first cell actually occupying the respective first cell.

[0176] Clause 28. A non-transitory processor-readable storage medium comprising processor-readable instructions, the processor-readable instructions causing a processor to:

[0177] determining a predicted occupancy grid based on the previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell;

[0178] Determining an observed occupancy grid using machine learning and based on the first sensor measurements, the observed occupancy grid comprising a plurality of second

[0179] Cells; and

[0180] An updated occupancy grid is determined based on the observed occupancy grid and the predicted occupancy grid.

[0181] Clause 29. The non-transitory processor-readable storage medium of clause 28, further comprising processor-readable instructions that cause the processor to:

[0182] obtaining the first sensor measurement from a first sensor; and

[0183] obtaining a second sensor measurement from a second sensor;

[0184] Wherein the processor-readable instructions that cause the processor to determine the observed occupancy grid include processor-readable instructions that cause the processor to use, for each second cell of the plurality of second cells, a corresponding first portion of first information corresponding to the first sensor measurement, a corresponding second portion of second information corresponding to the second sensor measurement, or a combination thereof.

[0185] Clause 30. A non-transitory processor-readable storage medium according to clause 29, wherein the first information includes the first sensor measurement and the second information includes the second sensor measurement, and wherein the processor-readable instructions that cause the processor to determine the observed occupancy grid include processor-readable instructions that cause the processor to use at least a first sensor measurement of the first sensor measurement, at least a second sensor measurement of the second sensor measurement, or a combination thereof for each second cell of the plurality of second cells.

[0186] Clause 31. The non-transitory processor-readable storage medium of clause 29, further comprising processor-readable instructions that cause the processor to:

[0187] deriving the first information from the first sensor measurements; and

[0188] The second information is derived from the second sensor measurement.

[0189] Clause 32. The non-transitory processor-readable storage medium of Clause 31, wherein the first information comprises a bird's-eye view of the area.

[0190] Clause 33. A non-transitory processor-readable storage medium according to clause 31, wherein the first information includes a plurality of first probability indications, each indicating a first probability of a first corresponding possible type of occupant of a corresponding one of the sub-areas, and the second information includes a plurality of second probability indications, each indicating a second probability of a second corresponding possible type of occupant of a corresponding one of the sub-areas.

[0191] Clause 34. The non-transitory processor-readable storage medium of Clause 29, further comprising processor-readable instructions that cause the processor to:

[0192] Determine the occupancy grid-to-image transformation through machine learning;

[0193] determining an image-to-occupancy grid transform based on the occupancy grid-to-image transform; and determining the first information by applying the image-to-occupancy grid transform to third information corresponding to an image corresponding to measurements of the first sensor, the first sensor

[0194] Including camera.

[0195] Clause 35. The non-transitory processor-readable storage medium of clause 34, wherein the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupancy grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and wherein:

[0196] The occupancy grid to image transform maps at least two occupied grid cells of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or

[0197] The occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions

[0198] domain; or

[0199] The image to occupancy grid transform maps at least two of the plurality of third information regions to a single occupancy grid cell of the plurality of occupancy grid cells.

[0200] or

[0201] The image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupancy grid cells of the plurality of occupancy grid cells

[0202] or

[0203] a combination of two or more of them;

[0204] Thereby there is a non-uniform mapping between the occupancy grid and the third information.

[0205] Clause 36. The non-transitory processor-readable storage medium of Clause 28, wherein the plurality of predicted probability indications each indicate a plausibility of an occupant of the respective possible type of the respective first cell actually occupying the respective first cell.

[0206] Other considerations

[0207] Other examples and implementations are within the scope of this disclosure and the appended claims. For example, due to the nature of software and computers, the above functions can be implemented using software executed by a processor, hardware, firmware, hard wiring, or any combination thereof. Features that implement the functions can also be physically located at different locations, including being distributed so that parts of the functions are implemented at different physical locations.

[0208] As used herein, the singular forms "a", "an", and "the" also include the plural forms unless the context clearly indicates otherwise. Thus, references to devices in the singular (e.g., "device", "the / the device") included in the claims include at least one (i.e., one or more) of such devices (e.g., "processor" includes at least one processor (e.g., one processor, two processors, etc.), "the / the processor" includes at least one processor, "memory" includes at least one memory, "the / the memory" includes at least one memory, etc.). The phrases "at least one" and "one or more" are used interchangeably, and such that "at least one" referent and "one or more" referents include both specific implementations having one referent and specific implementations having multiple referents. For example, "at least one processor" and "one or more processors" each include both specific implementations having one processor and specific implementations having multiple processors.

[0209] Similarly, as used herein, "or" used in a list of items (possibly followed by "at least one of" or "one or more of") indicates a disjunctive list, so that, for example, a list of "at least one of A, B, or C", or a list of "one or more of A, B, or C", or a list of "A or B or C" means A or B or C or AB (A and B) or AC (A and C) or BC (B and C) or ABC (i.e., A and B and C), or a combination having more than one feature (e.g., AA, AAB, ABBC, etc.). Thus, a statement that an item (e.g., a processor) is configured to perform a function with respect to at least one of A or B, or a statement that an item is configured to perform function A or function B, means that the item can be configured to perform a function with respect to A, or can be configured to perform a function with respect to B, or can be configured to perform functions with respect to both A and B. For example, the phrase "a processor configured to measure at least one of A or B" or "a processor configured to measure A or measure B" means that the processor may be configured to measure A (and may or may not be configured to measure B), or may be configured to measure B (and may or may not be configured to measure A), or may be configured to measure A and measure B (and may be configured to select which of A and B to measure or both). Similarly, a statement about a component for measuring at least one of A or B includes: a component for measuring A (which may or may not be able to measure B), or a component for measuring B (and may or may not be configured to measure A), or a component for measuring A and B (which may be able to select which of A and B to measure or both). As another example, a statement that an item (e.g., a processor) is configured to perform at least one of function X or to perform function Y means that the item may be configured to perform function X, or may be configured to perform function Y, or may be configured to perform function X and perform function Y. For example, the phrase "a processor configured to measure at least one of X or Y" means that the processor may be configured to measure X (and may or may not be configured to measure Y), or may be configured to measure Y (and may or may not be configured to measure X), or may be configured to measure X and measure Y (and may be configured to select which or both of X and Y to measure).

[0210] As used herein, unless otherwise stated, a recitation that a function or operation is "based on" an item or condition means that the function or operation is based on the recited item or condition, and may be based on one or more items and / or conditions other than the recited item or condition.

[0211] Substantial changes may be made according to specific requirements. For example, customized hardware may also be used, and / or specific elements may be implemented in hardware, in software (including portable software, such as applets, etc.) executed by a processor, or in both. In addition, connections with other computing devices such as network input / output devices may be employed. Unless otherwise indicated, components (functional or otherwise) shown in the figures and / or discussed herein as being connected or communicating with each other are communicatively coupled. That is, these components may be connected directly or indirectly to enable communication between them.

[0212] The systems and devices discussed above are examples. Various configurations may appropriately omit, replace, or add various procedures or components. For example, features described with respect to certain configurations may be combined in various other configurations. Different aspects and elements of a configuration may be combined in a similar manner. In addition, technology is constantly evolving, and therefore many elements are examples and do not limit the scope of the present disclosure or claims.

[0213] A wireless communication system is a system in which communications are transmitted wirelessly between wireless communication devices, i.e., by propagation of electromagnetic waves and / or sound waves through atmospheric space rather than through wires or other physical connections. A wireless communication system (also referred to as a wireless communication system, a wireless communication network, or a wireless communication network) may not cause all communications to be sent wirelessly, but may cause at least some communications to be sent wirelessly. Furthermore, the term "wireless communication device" or similar terms does not require that the functionality of the device be used exclusively or even primarily for communication, that communications using the wireless communication device be exclusively or even primarily wireless, or that the device be a mobile device, but rather indicates that the device includes wireless communication capabilities (unidirectional or bidirectional), for example, including at least one radio component (each radio component being part of a transmitter, receiver, or transceiver) for wireless communication.

[0214] Specific details are given in the description of this article to provide a thorough understanding of example configurations (including specific implementations). However, configurations can be practiced without these specific details. For example, well-known circuits, processes, algorithms, structures, and techniques have been shown without unnecessary details to avoid confusing these configurations. The description of this article provides example configurations without limiting the scope, applicability, or configuration of the claims. On the contrary, the previous description of the configuration provides a description for implementing the described technology. Various changes can be made to the function and arrangement of the elements.

[0215] As used herein, the terms "processor-readable medium," "machine-readable medium," and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in a specific manner. Using a computing platform, various processor-readable media may be involved in providing instructions / code to a processor for execution, and / or may be used to store and / or carry such instructions / code (e.g., as signals). In many specific implementations, processor-readable media are physical and / or tangible storage media. Such media may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical disks and / or magnetic disks. Volatile media include, but are not limited to, dynamic memory.

[0216] After describing several example configurations, various modifications, alternative constructions and equivalents can be used. For example, the above elements can be components of a larger system, in which other rules can take precedence over the application of the present disclosure or otherwise modify the application of the present disclosure. In addition, several operations can be taken before, during or after considering the above elements. Accordingly, the above description does not limit the scope of the claims.

[0217] Unless otherwise indicated, "about" and / or "approximately" as used herein in reference to a measurable value (such as an amount, a duration of time, etc.) encompasses variations of ±20% or ±10%, ±5%, or ±0.1% from the specified value, as appropriate in the context of the systems, devices, circuits, methods, and other specific implementations described herein. Unless otherwise indicated, "substantially" as used herein in reference to a measurable value (such as an amount, a duration of time, a physical property (such as frequency), etc.) also encompasses variations of ±20% or ±10%, ±5%, or ±0.1% from the specified value, as appropriate in the context of the systems, devices, circuits, methods, and other specific implementations described herein.

[0218] A statement that a value exceeds (or is greater than or above) a first threshold is equivalent to a statement that the value meets or exceeds a second threshold that is slightly greater than the first threshold, e.g., the second threshold is one value higher than the first threshold at the resolution of the computing system. A statement that a value is less than (or within or below) a first threshold is equivalent to a statement that the value is less than or equal to a second threshold that is slightly less than the first threshold, e.g., the second threshold is one value lower than the first threshold at the resolution of the computing system.

Claims

1. A device, the device include: Memory; and a processor communicatively coupled to the memory and configured to: determining a predicted occupancy grid based on the previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell; determining, using machine learning and based on the first sensor measurements, an observed occupancy grid comprising a plurality of second cells corresponding to the sub-region of the region; as well as An updated occupancy grid is determined based on the observed occupancy grid and the predicted occupancy grid.

2. The device according to claim 1, further comprising: include: a first sensor configured to obtain the first sensor measurement; and a second sensor configured to obtain a second sensor measurement; wherein the processor is communicatively coupled to the first sensor and the second sensor, and wherein to determine the observed occupancy grid, the processor is configured to use, for each second cell of the plurality of second cells, a corresponding first portion of first information corresponding to the first sensor measurement, a corresponding second portion of second information corresponding to the second sensor measurement, or a combination thereof.

3. The apparatus of claim 2 , wherein the first information comprises the first sensor measurements and the second information comprises the second sensor measurements, and wherein to determine the observed occupancy grid, the processor is configured to use, for each second cell of the plurality of second cells, at least a first sensor measurement of the first sensor measurements, at least a second sensor measurement of the second sensor measurements, or a combination thereof. 4 . The apparatus of claim 2 , wherein the first information is derived from the first sensor measurements and the second information is derived from the second sensor measurements. The apparatus of claim 4 , wherein the first information comprises a bird's-eye view of the area.

6. An apparatus according to claim 4, wherein the first information includes multiple first probability indications, each indicating a first probability of a first corresponding possible type of an occupant of a corresponding one of the sub-areas, and the second information includes multiple second probability indications, each indicating a second probability of a second corresponding possible type of an occupant of a corresponding one of the sub-areas.

7. The apparatus of claim 2, wherein the processor is further configured to: Determine the occupancy grid-to-image transformation through machine learning; determining an image to occupancy grid transform based on the occupancy grid to image transform; and The first information is determined by applying the image-to-occupancy grid transform to third information corresponding to an image corresponding to measurements by the first sensor, the first sensor comprising a camera.

8. The apparatus of claim 7, wherein the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupancy grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and in: The occupancy grid to image transform maps at least two occupied grid cells of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or The occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions; or The image to occupancy grid transform maps at least two of the plurality of third information regions to a single occupied grid cell of the plurality of occupied grid cells; or The image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupied grid cells of the plurality of occupied grid cells; or a combination of two or more of them; Thereby there is a non-uniform mapping between the occupancy grid and the third information.

9. The apparatus of claim 1, wherein the plurality of predicted probability indications each indicate plausibility of an occupant of the respective possible type of the respective first cell actually occupying the respective first cell.

10. A method for determining an occupied grid, the method comprising: include: determining a predicted occupancy grid based on the previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell; determining, using machine learning and based on the first sensor measurements, an observed occupancy grid comprising a plurality of second cells corresponding to the sub-region of the region; as well as An updated occupancy grid is determined based on the observed occupancy grid and the predicted occupancy grid.

11. The method for determining an occupied grid according to claim 10, further comprising: include: obtaining the first sensor measurement from a first sensor; as well as obtaining a second sensor measurement from a second sensor; Wherein determining the observed occupancy grid comprises using, for each second cell of the plurality of second cells, a respective first portion of first information corresponding to the first sensor measurement, a respective second portion of second information corresponding to the second sensor measurement, or a combination thereof.

12. The occupancy grid determination method of claim 11, wherein the first information comprises the first sensor measurements and the second information comprises the second sensor measurements, and wherein determining the observed occupancy grid comprises using at least a first sensor measurement of the first sensor measurements, at least a second sensor measurement of the second sensor measurements, or a combination thereof for each second cell of the plurality of second cells. 13 . The occupancy grid determination method of claim 11 , further comprising deriving the first information from the first sensor measurements and deriving the second information from the second sensor measurements. The occupancy grid determination method according to claim 13 , wherein the first information comprises a bird's eye view of the area.

15. The occupancy grid determination method according to claim 13, wherein the first information includes a plurality of first probability indications each indicating a first probability of a first corresponding possible type of an occupant of a corresponding one of the sub-areas, and the second information includes a plurality of second probability indications each indicating a second probability of a second corresponding possible type of an occupant of a corresponding one of the sub-areas.

16. The method for determining an occupied grid according to claim 11, further comprising: include: Determine the occupancy grid-to-image transformation through machine learning; determining an image to occupancy grid transform based on the occupancy grid to image transform; as well as The first information is determined by applying the image-to-occupancy grid transform to third information corresponding to an image corresponding to measurements by the first sensor, the first sensor comprising a camera.

17. The occupancy grid determination method of claim 16, wherein the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupancy grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and in: The occupancy grid to image transform maps at least two occupied grid cells of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or The occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions; or The image to occupancy grid transform maps at least two of the plurality of third information regions to a single occupied grid cell of the plurality of occupied grid cells; or The image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupied grid cells of the plurality of occupied grid cells; or a combination of two or more of them; Thereby there is a non-uniform mapping between the occupancy grid and the third information.

18. The occupancy grid determination method according to claim 10, wherein the plurality of predicted probability indications each indicate plausibility of an occupant of the respective possible type of the respective first cell actually occupying the respective first cell.

19. A device, wherein the device include: means for determining a predicted occupancy grid based on a previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell; means for determining, using machine learning and based on first sensor measurements, an observed occupancy grid comprising a plurality of second cells corresponding to the sub-region of the region; and Means for determining an updated occupancy grid based on the observed occupancy grid and the predicted occupancy grid.

20. The device according to claim 19, further comprising: include: means for obtaining said first sensor measurement from a first sensor; and means for obtaining a second sensor measurement from a second sensor; Wherein the means for determining the observed occupancy grid comprises means for using, for each second cell of the plurality of second cells, a respective first portion of first information corresponding to the first sensor measurement, a respective second portion of second information corresponding to the second sensor measurement, or a combination thereof.

21. The apparatus of claim 20, wherein the first information comprises the first sensor measurements and the second information comprises the second sensor measurements, and wherein the means for determining the observed occupancy grid comprises means for using, for each second cell in the plurality of second cells, at least a first sensor measurement of the first sensor measurements, at least a second sensor measurement of the second sensor measurements, or a combination thereof.

22. The apparatus of claim 20, further comprising means for deriving the first information from the first sensor measurements and means for deriving the second information from the second sensor measurements.

23. The apparatus of claim 22, wherein the first information comprises a bird's-eye view of the area.

24. An apparatus according to claim 22, wherein the first information includes multiple first probability indications, each indicating a first probability of a first corresponding possible type of an occupant of a corresponding one of the sub-areas, and the second information includes multiple second probability indications, each indicating a second probability of a second corresponding possible type of an occupant of a corresponding one of the sub-areas.

25. The device according to claim 20, further comprising: include: a component for determining an occupancy grid-to-image transformation by machine learning; means for determining an image to occupancy grid transform based on said occupancy grid to image transform; and Means for determining the first information by applying the image-to-occupancy grid transform to third information corresponding to an image corresponding to measurements of the first sensor, the first sensor comprising a camera.

26. The apparatus of claim 25, wherein the occupancy grid to image transform maps between an occupancy grid comprising a plurality of occupancy grid cells and the third information comprising a plurality of third information regions, and the image to occupancy grid transform maps between the third information and the occupancy grid, and in: The occupancy grid to image transform maps at least two occupied grid cells of the plurality of occupied grid cells to a single pixel of the plurality of third information regions; or The occupancy grid to image transform maps a single occupied grid cell of the plurality of occupied grid cells to at least two third information regions of the plurality of third information regions; or The image to occupancy grid transform maps at least two of the plurality of third information regions to a single occupied grid cell of the plurality of occupied grid cells; or The image to occupancy grid transform maps a single third information region of the plurality of third information regions to at least two occupied grid cells of the plurality of occupied grid cells; or a combination of two or more of them; Thereby there is a non-uniform mapping between the occupancy grid and the third information.

27. The apparatus of claim 19, wherein the plurality of predicted probability indications each indicate a plausibility of an occupant of the respective possible type of the respective first cell actually occupying the respective first cell.

28. A non-transitory processor-readable storage medium comprising processor-readable instructions, the processor-readable instructions causing a processor to: determining a predicted occupancy grid based on the previous occupancy grid, the predicted occupancy grid comprising a plurality of first cells corresponding to a sub-region of the region, each first cell of the plurality of first cells comprising a plurality of predicted probability indications each indicating a predicted probability of a respective possible type of an occupant of the respective first cell; determining, using machine learning and based on the first sensor measurements, an observed occupancy grid comprising a plurality of second cells corresponding to the sub-region of the region; as well as An updated occupancy grid is determined based on the observed occupancy grid and the predicted occupancy grid.

29. The non-transitory processor-readable storage medium of claim 28, further comprising processor-readable instructions, the processor-readable instructions causing the processor to: obtaining the first sensor measurement from a first sensor; and obtaining a second sensor measurement from a second sensor; Wherein the processor-readable instructions that cause the processor to determine the observed occupancy grid include processor-readable instructions that cause the processor to use, for each second cell of the plurality of second cells, a corresponding first portion of first information corresponding to the first sensor measurement, a corresponding second portion of second information corresponding to the second sensor measurement, or a combination thereof.

30. A non-transitory processor-readable storage medium according to claim 29, wherein the first information includes the first sensor measurement and the second information includes the second sensor measurement, and wherein the processor-readable instructions that cause the processor to determine the observed occupancy grid include processor-readable instructions that cause the processor to use at least a first sensor measurement of the first sensor measurements, at least a second sensor measurement of the second sensor measurements, or a combination thereof for each second cell of the plurality of second cells.

Images