System and method for generating amodal cuboids

Abstract

A system and method for operating an autonomous vehicle. The method includes: obtaining, by a computing device, loosely fitting cuboids overlaid on a 3D graphic, such that each contains a LiDAR data point associated with a given object; defining, by the computing device, an amodal cuboid based on the loosely fitting cuboids; using the amodal cuboids to train a machine learning algorithm, by the computing device, to detect a given class of objects using sensor data generated by a sensor of the autonomous vehicle or another vehicle; and causing, by the computing device, the machine learning algorithm to be used to control the operation of the autonomous vehicle.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. patent application No. 17 / 241,637, filed on April 27, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to autonomous vehicles and more particularly to systems and methods for implementing amodal cuboid generation from ground truth data. Background Art

[0004] Modern vehicles have at least one onboard computer and have an internet / satellite connection. Software running on these onboard computers monitors and / or controls the operation of the vehicle. The vehicle also includes a LiDAR detector for detecting objects in its vicinity. The LiDAR detector generates a LiDAR dataset that measures the distance from the vehicle to the object at multiple different times. These distance measurements can be used to track the movement of the object, make predictions about the trajectory of the object, and plan the vehicle's driving path based on the predicted object trajectory. Summary of the invention

[0005] The present disclosure relates to implementing a system and method for operating an autonomous vehicle. The method includes performing the following operations by a computing device: obtaining loosely fitted cuboids overlaid on a 3D graphic, each containing LiDAR data points associated with a given object; defining an amodal cuboid based on the loosely fitted cuboid; using the amodal cuboid to train a machine learning algorithm to detect a given class of objects using sensor data generated by a sensor of the autonomous vehicle or another vehicle; and using the machine learning algorithm to control the operation of the autonomous vehicle.

[0006] The computing device may also determine whether the given object is less than a first distance (e.g., 30 meters) from the LiDAR sensor. When it is determined that the given object is less than the first distance from the LiDAR sensor, the non-modal cuboid is used to train the machine learning algorithm. When it is determined that the distance between the given object and the LiDAR sensor is greater than the first distance, at least one other cuboid is used. The other cuboid may include, but is not limited to, a loosely fitted cuboid or a weighted cuboid generated based on the loosely fitted cuboid and the non-modal cuboid.

[0007] The non-modal cuboid may be defined by: identifying LiDAR data points within one of the loosely fitting cuboids; removing data points associated with the ground from the LiDAR data points; defining a tightly fitting cuboid of tightly fitting LiDAR data points; accumulating LiDAR data points in the loosely fitting cuboid; defining a cumulatively fitting cuboid of accumulated LiDAR data points that tightly fit the loosely fitting cuboid; and / or determining the non-modal extent using the cumulatively fitting cuboid and a previous non-modal extent of the object. It should be noted that when determining or otherwise defining the geometry of the cuboid, a tolerance threshold may need to be met. For example, 95% of all LiDAR data points of an object may need to be included in a given cuboid. The present solution is not limited to the details of this example.

[0008] The modeless cuboid has dimensions specified by the modeless extent. The modeless extent can be set equal to the maximum previous dimension of the previous modeless extent when the dimension of the cumulative fit cuboid is greater than the maximum previous dimension, equal to the minimum previous dimension of the previous modeless extent when the dimension of the cumulative fit cuboid is less than the minimum previous dimension, or equal to the extent of the cumulative fit cuboid when the dimension of the modeless extent falls between the maximum previous dimension and the minimum previous dimension of the previous modeless extent.

[0009] The coordinates of a center of the non-modal cuboid are determined based on at least one of the tightly fitting cuboid or the loosely fitting cuboid. The center of the non-modal cuboid may be determined by: (i) aligning the corners of the non-modal cuboid with the closest visible corners of the tightly fitting cuboid; and (ii) aligning the corners of the non-modal cuboid such that the boundary accuracy between the non-modal cuboid and the loosely fitting cuboid is maximized.

[0010] The implementation system may include: a processor; and a non-transitory computer-readable storage medium including programming instructions, the programming instructions configured to cause the processor to implement a method for operating a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The present solution will be described with reference to the following drawings, wherein like reference numerals refer to like items throughout the drawings.

[0012] Figure 1 is a diagram of an illustrative system.

[0013] Figure 2 is a diagram of an illustrative architecture of a vehicle.

[0014] Figure 3 Is Figure 2 Illustration of an illustrative architecture of a LiDAR system employed by the vehicle shown in .

[0015] Figure 4 is a diagram of an illustrative computing device.

[0016] Figure 5 A block diagram of an illustrative vehicle trajectory planning process is provided.

[0017] Figure 6A-6B (collectively referred to herein as "FIG. 6") provides a flow chart of an illustrative method for operating a vehicle.

[0018] Figure 7 A plot including the LiDAR dataset and a loosely fitted cuboid is provided.

[0019] Figure 8 A diagram is provided that can be used to understand the object's orbit.

[0020] Fig. 9 A flow chart of an illustrative method for generating or producing a modeless cuboid is provided.

[0021] Fig.10 A plot including the LiDAR dataset and a tightly fitted cuboid is provided.

[0022] Fig.11 A flow chart of an illustrative method for determining amodal range is provided. DETAILED DESCRIPTION

[0023] As used in this document, the singular forms "a", "an", and "the" include plural references unless the context clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by one of ordinary skill in the art. As used in this document, the term "including" means "including but not limited to". Definitions of additional terms related to this document are included at the end of this detailed description.

[0024] An "electronic device" or "computing device" refers to a device that includes a processor and memory. Each device may have its own processor and / or memory, or the processor and / or memory may be shared with other devices, such as in a virtual machine or container arrangement. The memory will contain or receive programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations in accordance with the programming instructions.

[0025] The terms "memory," "memory device," "data storage," "data storage facility," and the like each refer to a non-transitory device on which computer-readable data, programming instructions, or both are stored. Unless specifically stated otherwise, the terms "memory," "memory device," "data storage," "data storage facility," and the like are intended to include single device embodiments, embodiments in which multiple memory devices together or collectively store a set of data or instructions, and individual sectors within such devices.

[0026] The terms "processor" and "processing device" refer to a hardware component of an electronic device that is configured to execute programmed instructions. Unless otherwise specifically stated, the singular term "processor" or "processing device" is intended to include both single processing device embodiments and embodiments in which multiple processing devices together or collectively execute a process.

[0027] The term "vehicle" refers to any mobile form of transportation capable of carrying one or more human occupants and / or cargo and powered by any form of energy. The term "vehicle" includes, but is not limited to, cars, trucks, vans, trains, autonomous vehicles, airplanes, aerial drones, etc. An "autonomous vehicle" is a vehicle having a processor, programming instructions, and drive train components that can be controlled by the processor without the need for a human operator. An autonomous vehicle can be fully autonomous in that it does not require a human operator for most or all driving conditions and functions, or it can be semi-autonomous in that a human operator may be required under certain conditions or for certain operations, or a human operator can override the vehicle's autonomous systems and can control the vehicle.

[0028] In this document, when terms such as "first" and "second" are used to modify nouns, such usage is intended only to distinguish one item from another and is not intended to require a sequential order unless otherwise specified. Additionally, relative position terms such as "vertical" and "horizontal" or "front" and "rear" when used are intended to be relative to each other and need not be absolute, and refer only to one possible position of the device associated with these terms, depending on the orientation of the device.

[0029] To evaluate the performance of the perception pipeline, the system relies on user-labeled cuboids and true range cuboids (amodal cuboids). The labeler labels the cuboids in LiDAR space, which provides dimensions up to the visible range and loosely fits the actual shape of the object. Therefore, the labeled cuboids do not represent the true range of the object.

[0030] This document describes systems and methods for implementing solutions to the above problems and / or other problems. Accordingly, the present solution is directed to systems and methods for generating a modeless cuboid and / or controlling a vehicle using the same. The cuboid of an object (e.g., a vehicle) is a 3D oriented bounding box that represents (i) the orientation of the object, and (ii) the complete extent of the object.

[0031] The method generally involves performing the following operations by a computing device: obtaining loose-fit cuboids overlaid on a 3D graphic, such that each contains a LiDAR data point associated with a given object; defining an amodal cuboid based on the loose-fit cuboids; using the amodal cuboids to train a machine learning algorithm, for example, to detect objects, determine a classification of objects, and / or determine motion of objects using sensor data generated by sensors of the autonomous vehicle or another vehicle; and enabling use of the machine learning algorithm to control operation of the autonomous vehicle.

[0032] The computing device may also determine whether the given object is less than a first distance (e.g., 30 meters) from the LiDAR sensor. When it is determined that the given object is less than the first distance from the LiDAR sensor, the non-modal cuboid is used to train the machine learning algorithm. When it is determined that the distance between the given object and the LiDAR sensor is greater than the first distance, at least one other cuboid is used. The other cuboid may include, but is not limited to, a loosely fitted cuboid or a weighted cuboid generated based on the loosely fitted cuboid and the non-modal cuboid.

[0033] The non-modal cuboid may be defined by: identifying LiDAR data points within a loosely fitting cuboid within one of the loosely fitting cuboids; removing data points associated with the ground from the LiDAR data points; defining a tightly fitting cuboid of tightly fitting LiDAR data points; accumulating LiDAR data points within the loosely fitting cuboid; defining a cumulatively fitting cuboid of accumulated LiDAR data points that tightly fit the loosely fitting cuboid; and / or determining the non-modal extent using the cumulatively fitting cuboid and a previous non-modal extent of the object. The non-modal cuboid has dimensions specified by the non-modal extent. The non-modal extent may be: set equal to a maximum previous dimension of a previous non-modal extent when the size of the cumulatively fitting cuboid is greater than the maximum previous dimension; set equal to a minimum previous dimension when the size of the cumulatively fitting cuboid is less than a minimum previous dimension of a previous non-modal extent; or set equal to an extent of the cumulatively fitting cuboid when the size of the non-modal extent falls between a maximum previous dimension and a minimum previous dimension of a previous non-modal extent.

[0034] The coordinates of the center of the non-modal cuboid are determined based on at least one of the tightly fitted cuboid or the loosely fitted cuboid. The center of the non-modal cuboid may be determined by: (i) aligning the corners of the non-modal cuboid with the closest visible corners of the tightly fitted cuboid; and (ii) aligning the corners of the non-modal cuboid such that the boundary accuracy between the non-modal cuboid and the loosely fitted cuboid is maximized.

[0035] The implementation system may include: a processor; and a non-transitory computer-readable storage medium including programming instructions, the programming instructions being configured to cause the processor to implement a method for operating a vehicle.

[0036] It is worth noting that this article describes the present solution in the context of an autonomous vehicle. The present solution is not limited to autonomous vehicle applications. The present solution can be used in other applications, such as robotic applications.

[0037] Descriptive System

[0038] Reference now Figure 1 , provides an illustration of an illustrative system 100. The system 100 includes a vehicle 1021 that travels along a road in a semi-autonomous or autonomous manner. The vehicle 1021 is also referred to herein as an AV. The AV 1021 may include, but is not limited to, a land vehicle (e.g., Figure 1 shown), aircraft, vessel or spacecraft.

[0039] AV 1021 is typically configured to detect objects in its vicinity. Objects may include, but are not limited to, vehicles 1022, cyclists (not shown) (cyclists such as bicycles, electric scooters, motorcycles, etc.) and / or pedestrians (not shown). Object detection can be achieved using a machine learning algorithm trained with a non-modal cuboid determined according to the present solution. As the discussion proceeds, the manner in which the non-modal cuboid is determined or otherwise generated will become apparent. In addition, it should be understood that the non-modal cuboid is determined / generated using a LiDAR dataset generated by an onboard LiDAR detector of the AV and / or another vehicle. The LiDAR detector typically measures the distance to the object 1022 by illuminating the object 1022 with light 104 (e.g., a laser) and measuring the reflected light 106 with an onboard sensor. The LiDAR detector generates a LiDAR dataset at multiple times t, t+1, t+2, ..., t+n. The LiDAR dataset is processed by an onboard computing device of the AV 1021 and / or by a remote computing device 110 to generate an amodal cuboid of an object given the LiDAR dataset. In this regard, the LiDAR dataset may be transmitted from the AV 1021 to the remote computing device 110 via a wired and / or wireless connection over a network 108 (e.g., the Internet). The LiDAR dataset may also be stored in a memory of the AV 1021, which may be manually removed from the AV 1021 and connected to the remote computing device 110. The LiDAR dataset may additionally be stored in a remote data store 112 (e.g., a database). The amodal cuboid of the object is then used to train a machine learning algorithm for object detection and / or object trajectory prediction / likelihood.

[0040] When performing such object detection, the AV 1021 performs operations to: generate one or more possible object tracks for the detected object; analyze the generated possible object tracks to determine a track for the AV 1021; and cause the AV 1021 to follow the track.

[0041] Reference now Figure 2 , a diagram of an illustrative system architecture 200 for a vehicle is provided. Figure 1 The carrier 1021 and / or 1022 may have Figure 2 Therefore, the following discussion of system architecture 200 is sufficient to understand the system architecture 200. Figure 1 Vehicles 1021, 1022.

[0042] like Figure 2 As shown, the vehicle 200 includes an engine or motor 202 and various sensors 204-218 for measuring various parameters of the vehicle. In a gas powered or hybrid vehicle having a fuel powered engine, the sensors may include, for example, an engine temperature sensor 204, a battery voltage sensor 206, an engine revolutions per minute (RPM) sensor 208, and a throttle position sensor 210. If the vehicle is an electric vehicle or a hybrid vehicle, the vehicle may have an electric motor and therefore will have sensors such as a battery monitoring system 212 (to measure the current, voltage and / or temperature of the battery), a motor current sensor 214 and a voltage sensor 216, and a motor position sensor such as a resolver and encoder 218.

[0043] Operating parameter sensors common to both types of vehicles include, for example: a position sensor 236, such as an accelerometer, gyroscope, and / or inertial measurement unit; a speed sensor 238; and an odometer sensor 240. The vehicle may also have a clock 242, which the system uses to determine vehicle time during operation. The clock 242 may be encoded into the vehicle onboard computing device, it may be a separate device, or multiple clocks may be available.

[0044] The vehicle will also include various sensors that operate to collect information about the environment in which the vehicle is traveling. These sensors may include, for example: a position sensor 260 (e.g., a global positioning system (GPS) device); an object detection sensor, such as one or more cameras 262; a LiDAR sensor system 264; and / or a radar and / or sonar system 266. The sensors may also include environmental sensors 268, such as a precipitation sensor and / or an ambient temperature sensor. The object detection sensors may enable the vehicle to detect objects within a given distance range of the vehicle 200 in any direction, while the environmental sensors collect data about the environmental conditions within the vehicle's travel area.

[0045] During operation, information is transmitted from the sensors to the onboard computing device 220. The onboard computing device 220 analyzes the data captured by the sensors and, optionally based on the results of the analysis, controls the operation of the vehicle. For example, the onboard computing device 220 may control: braking via a brake controller 232; direction via a steering controller 224; speed and acceleration via a throttle controller 226 (in a gas powered vehicle) or a motor speed controller 228 (such as a current level controller in an electric vehicle); a differential gear controller 230 (in a vehicle with a transmission); and / or other controllers.

[0046] The geographic location information may be transmitted from the location sensor 260 to the onboard computing device 220, which may then access a map of the environment corresponding to the location information to determine known fixed features of the environment, such as streets, buildings, stop signs, and / or stop / go signs. Images captured from the camera 262 and / or object detection information captured from sensors such as LiDAR 264 are transmitted from those sensors to the onboard computing device 220. The object detection information and / or captured images are processed by the onboard computing device 220 to detect objects in the vicinity of the vehicle 200. Any known or to-be-known technique for object detection based on sensor data and / or captured images may be used in the embodiments disclosed in this document.

[0047] When the onboard computing device 220 detects a moving object, the onboard computing device 220 generates one or more possible object trajectories for the detected object and uses the possible object trajectories to determine a vehicle trajectory for the AV. The onboard computing device 220 then performs operations to cause the AV to follow the defined vehicle trajectory. For example, the onboard computing device 220 uses the object trajectory information to determine what space the object has occupied and then generates a vehicle trajectory where the AV is not planning to travel to that space.

[0048] Reference now Figure 3 , a diagram of an illustrative LiDAR system 300 is provided. Figure 2 The LiDAR system 264 may be the same as or substantially similar to the LiDAR system 300. Therefore, the discussion of the LiDAR system 300 is sufficient to understand Figure 2 LiDAR system 264.

[0049] like Figure 3 As shown, the LiDAR system 300 includes a housing 306 that can rotate 360° about a central axis such as a hub or shaft 316. The housing may include a transmitter / receiver aperture 312 made of a material that is transparent to light. Figure 2A single hole is shown in , but the present solution is not limited in this regard. In other cases, multiple holes for transmitting and / or receiving light can be provided. Either way, when the housing 306 rotates about the internal assembly, the LiDAR system 300 can transmit light through one or more of the (one or more) holes 312 and receive reflected light back toward one or more of the (one or more) holes 211. In an alternative scenario, the housing of the housing 306 can be a fixed dome that is at least partially made of a material that is transparent to light, with the rotatable assembly inside the housing 306.

[0050] Inside the rotating shell or fixed dome is a light emitter system 304, which is configured and positioned to generate and emit light pulses through the hole 312 or through the transparent dome of the housing 306 via one or more laser emitter chips or other light emitting devices. The emitter system 304 may include any number of individual emitters (e.g., 8 emitters, 64 emitters, or 128 emitters). The emitters may emit light of substantially the same intensity or light of different intensities. The individual beams emitted by the light emitter system 304 will have a well-defined polarization state that is not the same across the array. As an example, some beams may have vertical polarization, while other beams may have horizontal polarization. The LiDAR system will also include a light detector 308, which includes a photodetector or a photodetector array, which is positioned and configured to receive light reflected back into the system. The emitter system 304 and the light detector 308 will rotate with the rotating shell, or they will rotate within the fixed dome of the housing 306. One or more optical element structures 310 may be positioned in front of the light emitting unit 304 and / or the light detector 308 to act as one or more lenses or wave plates to focus and direct light passing through the optical element structures 310 .

[0051] One or more optical element structures 310 can be positioned in front of the reflector 312 to focus and direct light passing through the optical element structure 310. As shown below, the system includes an optical element structure 310 located in front of the reflector 312 and connected to a rotating element of the system so that the optical element structure 310 rotates with the reflector 312. Alternatively or additionally, the optical element structure 310 can include a plurality of such structures (e.g., lenses and / or wave plates). Optionally, a plurality of optical element structures 310 can be arranged in an array on the shell portion of the housing 306 or integrated with the shell portion of the housing 306.

[0052] Optionally, each optical element structure 310 may include a beam splitter that separates the light received by the system from the light generated by the system. The beam splitter may include, for example, a quarter wave or half wave plate to perform the separation and ensure that the received light is directed to the receiver unit rather than the transmitter system (which may occur without such a wave plate, as the emitted light and the received light should exhibit the same or similar polarization).

[0053] The LiDAR system will include a power supply unit 318, a motor 316, and electronic components for powering the light emitting unit 304. The LiDAR system will also include an analyzer 314 having elements such as a processor 322 and a non-transitory computer readable memory 320 containing programming instructions configured to enable the system to receive data collected by the light detector unit, analyze it to measure characteristics of the received light, and generate information that a connected system can use to make decisions about operating in the environment in which the data was collected. Optionally, the analyzer 314 can be integrated with the LiDAR system 300 as shown, or some or all of the analyzer 314 can be external to the LiDAR system and communicatively connected to the LiDAR system via a wired or wireless communication network or link.

[0054] Reference now Figure 4 , a diagram of an illustrative architecture for a computing device 400 is provided. Figure 1 The computing device 110 and / or Figure 2 The onboard computing device 220 of the vehicle is the same as or similar to the computing device 300. Thus, the discussion of the computing device 300 is sufficient to understand Figure 1 The computing device 110 and Figure 2 The vehicle onboard computing device 220.

[0055] The computing device 400 may include Figure 4 There may be more or fewer components than those shown in FIG. However, the components shown are sufficient to disclose an illustrative solution for implementing the present solution. Figure 4 The hardware architecture of represents one implementation of a representative computing device configured to operate a vehicle, as described herein. Thus, Figure 4 The computing device 400 implements at least a portion of the methods described herein.

[0056] Some or all components of computing device 400 may be implemented as hardware, software, and / or a combination of hardware and software. Hardware includes, but is not limited to, one or more electronic circuits. Electronic circuits may include, but are not limited to, passive components (e.g., resistors and capacitors) and / or active components (e.g., amplifiers and / or microprocessors). Passive and / or active components may be adapted, arranged, and / or programmed to perform one or more of the methods, processes, or functions described herein.

[0057] like Figure 4 As shown, computing device 400 includes user interface 402, central processing unit ("CPU") 406, system bus 410, memory 412 connected to and accessible by other parts of computing device 400 via system bus 410, system interface 460, and hardware entities 414 connected to system bus 410. The user interface may include input devices and output devices that facilitate user-software interaction for controlling the operation of computing device 400. Input devices include, but are not limited to, physical and / or touch keyboard 450. Input devices may be connected via a wired or wireless connection (e.g., The system interface 460 is connected to the computing device 400. Output devices include, but are not limited to, speakers 452, display 454, and / or light emitting diodes 456. The system interface 460 is configured to facilitate wired or wireless communications to and from external devices (e.g., network nodes such as access points, etc.).

[0058] At least some of the hardware entities 414 perform actions involving accessing and using memory 412, which may be random access memory ("RAM"), a disk drive, flash memory, a compact disk read-only memory ("CD-ROM"), and / or another hardware device capable of storing instructions and data. The hardware entities 414 may include a disk drive unit 416, which includes a computer-readable storage medium 418 having stored thereon one or more sets of instructions 420 (e.g., software code), the one or more sets of instructions 420 being configured to implement one or more of the methods, processes, or functions described herein. The instructions 420 may also reside completely or at least partially within the memory 412 and / or within the CPU 406 during execution thereof by the computing device 400. The memory 412 and the CPU 406 may also constitute machine-readable media. The term "machine-readable medium" as used herein refers to a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more sets of instructions 420. The term "machine-readable medium" as used herein also refers to any medium that is capable of storing, encoding, or carrying a set of instructions 420 for execution by the computing device 400 and causing the computing device 400 to perform any one or more of the methods of the present disclosure.

[0059] Reference now Figure 5 , provides a block diagram for understanding how to implement the movement of AV according to the present solution. All operations performed in blocks 502-512 can be performed by a vehicle (e.g., Figure 1 The AV 1021) is executed by an onboard computing device.

[0060] In block 502, an AV (e.g., Figure 1 The position of the AV 1021 can be based on the position sensor of the AV (e.g., Figure 2 The detection is performed using sensor data output by a position sensor 260 of the AV. The sensor data may include, but is not limited to, GPS data. The detected position of the AV is then passed to block 506.

[0061] In block 504, an object (e.g., Figure 1 The detection is based on a camera from the AV or another vehicle (e.g., Figure 2 camera 262) and / or a LiDAR system of an AV or another vehicle (e.g., Figure 2 The sensor data processing is performed on sensor data output by a LiDAR system 264). For example, image processing is performed to detect instances of a particular class of objects (e.g., vehicles or pedestrians) in one or more images. The LiDAR dataset is also processed to detect instances of certain classes of objects represented by point cloud data. This sensor data processing can be implemented using a machine learning algorithm trained on a non-modal cuboid generated / produced according to the present solution. The machine learning algorithm is trained to detect patterns in images and / or LiDAR datasets that identify objects of a given class (e.g., vehicles or pedestrians). Any machine learning algorithm can be used here. For example, one or more of the following machine learning algorithms are used here: supervised learning; unsupervised learning; semi-supervised learning; and reinforcement learning.

[0062] Additionally, a predicted trajectory of the object is determined in block 504. In block 504, a predicted trajectory of the object is determined based on the results of the machine learning algorithm (e.g., object category), the cuboid geometry, and / or the trajectory (defined by the cuboid over time, such as Figure 8 The trajectory of the object is predicted based on the content of the map 518 (as shown) and / or the content of the map 518 (e.g., a road / topographic map including information specifying the location of sidewalks, lane locations, lane travel directions, driving rules, etc.).

[0063] The cuboid geometry is determined using LiDAR datasets, images, and / or maps 518. Techniques for predicting object trajectories based on cuboid geometry are well known in the art. Any known or to-be-known technique for predicting object trajectories based on cuboid geometry may be used herein without limitation. For example, one technique involves predicting that an object moves on a linear path in the same direction as the cuboid's forward direction. The predicted object trajectory may include, but is not limited to, the following trajectory:

[0064] A trajectory defined by the object’s actual speed (e.g., 1 mile per hour) and actual direction of travel (e.g., west);

[0065] A trajectory defined by the object's actual speed (e.g., 1 mile per hour) and another possible direction of travel for the object (e.g., south, southwest, or X (e.g., 40°) degrees from the object's actual direction of travel in a direction toward the AV);

[0066] A trajectory defined by another possible speed of the object (e.g., 2-10 miles per hour) and the object's actual direction of travel (e.g., west); and / or

[0067] • A trajectory defined by another possible speed of the object (e.g., 2-10 miles per hour) and another possible direction of travel of the object (e.g., south, southwest, or X (e.g., 40°) degrees in a direction toward the AV from the object's actual direction of travel).

[0068] Possible speed(s) and / or possible direction(s) of travel may be predefined for objects in the same category and / or subcategory as the object. It should be noted again that the cuboid defines the complete extent of the object and the heading of the object. The heading defines the direction in which the front of the object is pointing and thus provides an indication of the actual and / or possible direction of travel of the object.

[0069] Information 520 specifying the predicted trajectory of the object and the cuboid geometry is provided to box 506. In some scenarios, the classification of the object is also passed to box 506. In box 506, the vehicle trajectory is generated using the information from boxes 502 and 504. The technology of determining the vehicle trajectory using a cuboid is well known in the art. Any known or to-be-known technology for determining the vehicle trajectory using a cuboid can be used herein without limitation. For example, in some scenarios, such a technology involves determining the trajectory of the AV that will pass through the object when the object is in front of the AV, the cuboid has a heading direction aligned with the direction of movement of the AV, and the cuboid has a length greater than a threshold. The present solution is not limited to the details of the scenario. The vehicle trajectory 520 can be determined based on the position information from box 502, the object detection information from box 504, and / or the map information 514 (which is pre-stored in the data storage of the vehicle). The vehicle trajectory 520 can represent a smooth path without sudden changes, otherwise the sudden changes will provide passenger discomfort. For example, the vehicle trajectory is defined by a travel path along a given lane of a road where the object is not predicted to travel within a given amount of time. The vehicle trajectory 520 is then provided to block 508 .

[0070] In block 508, steering angle and speed commands are generated based on the vehicle trajectory 520. The steering angle and speed commands are provided to block 510 for vehicle dynamics control, i.e., the steering angle and speed commands cause the AV to follow the vehicle trajectory 508.

[0071] Illustrative methods

[0072] Referring now to FIG. 6 , a flow chart of an illustrative method 600 for operating a vehicle is provided. The method 600 begins at 602 and continues at 604 , wherein a mobile platform (e.g., Figure 1 The AV 1021 of the embodiment performs operations to capture at least one image and / or generate at least one LiDAR dataset. The image(s) and / or LiDAR dataset(s) are transmitted to one or more computing devices (e.g., Figure 1 The computing device 110 and / or Figure 2 onboard computing device 220).

[0073] At the computing device(s), the LiDAR dataset(s) are plotted on 3D graph(s), as shown at 608. Each 3D graph has an x-axis, a y-axis, and a z-axis, with the origin defined at the center of the LiDAR sensor, the x-axis pointing forward, and the z-axis pointing upward. Figure 77 provides an illustration of a LiDAR dataset 702 plotted on a graph 700. It is worth noting that for ease of illustration, the graph 700 only shows a 2D viewpoint from the x-axis and the z-axis. Techniques for plotting LiDAR datasets on 3D graphics are well known in the art and will not be described here. Any known or to-be-known technique for plotting LiDAR datasets on 3D graphics may be used here.

[0074] In 610, the image(s) and / or 3D graphics(s) are used to detect objects located near the AV. The detection may be performed manually by an individual, or automatically / automated by a computing device. In a manual scenario, the individual analyzes the 3D graphics displayed on the screen of the computing device to identify data points that appear to define the object. In an automatic / automated scenario, the computing device may employ any known or to-be-known algorithm to identify data points that appear to define the object. Machine learning algorithms may be used here to facilitate object detection(s) and / or classification(s).

[0075] A loosely fitting cuboid is defined on each 3D shape at 612. The loosely fitting cuboid comprises a 3D oriented bounding box that represents (i) an object (e.g., Figure 1 1022), (ii) the orientation of the object (e.g., Figure 1 The entire extent of the object 1022), and / or the center / center of mass of the object. In this regard, the cuboid contains the LiDAR data points associated with the detected object in the 3D figure. The loosely fitting cuboid can be defined manually or automatically / automatically. In a manual scenario, an individual can perform a user-software interaction to draw a cuboid shape around the LiDAR data points identified in 610, where the facing direction is defined by the orientation of the object. In an automatic / automated scenario, the computing device can obtain the predefined cuboid dimensions (length, width, height) and set the center of the cuboid equal to the center of the LiDAR data point identified in 610. The loosely fitting cuboid can include a 3D shape that (i) contains a given percentage (e.g., >90%) of the LiDAR data points of the object, and (ii) has no or a minimal number of LiDAR data points for other objects (but allows the inclusion of LiDAR data points for the ground).

[0076] exist Figure 7 A diagram showing an illustrative loosely fitting cuboid 704 defined on graph 700 is provided in FIG. Figure 7As shown, data points of the LiDAR dataset 702 reside within the cuboid 704. Notably, none of the edges 706, 708, 710, 712 of the cuboid touch or are otherwise connected to data points of the LiDAR dataset 702. The present solution is not limited to the details of this illustration. One or more of the edges of the cuboid may touch or be otherwise connected to data points of the LiDAR dataset. The loosely fitted cuboid is constructed according to known or to-be-known techniques. For example, the cuboid is constructed by: fusing the LiDAR dataset, a vector map, and a visual orientation; and defining a cuboid with the highest probability along the visual orientation. The vector map contains a lane direction, which provides a strong indication of the orientation of the loosely fitted cuboid. The visual orientation is estimated for the object from the camera image. The present solution is not limited to the details of this example.

[0077] At 614, an amodal cuboid is defined for the object based on the loosely fitting cuboid. The amodal cuboid is overlaid on the 3D graphic so as to align with the LiDAR data points of the object. The manner in which the amodal cuboid is defined will become apparent as the discussion proceeds. Fig. 9 A flowchart is provided showing how to define a non-modal cuboid according to the present solution. Fig. 9 . An amodal cuboid is overlaid on each 3D figure by, for example, (i) aligning the corners of the amodal cuboid with the nearest visible corners of the tightly fitting cuboid; and (ii) aligning the corners of the amodal cuboid so that the boundary accuracy between the amodal cuboid and the loosely fitting cuboid is maximized. The term "boundary accuracy" refers to the calculated intersection area between the area of ​​the amodal cuboid and the area of ​​the loosely fitting cuboid divided by the total area of ​​the amodal cuboid. In some scenarios, an additional range margin on the amodal cuboid is provided. This is because after aligning to the nearest visible corner, a portion of the object may fall outside the cuboid (a portion of the object may actually be closer to the LiDAR sensor than the nearest corner of the tightly fitting cuboid). Therefore, a given margin (e.g., 10-20 cm) can be added to the final amodal cuboid to accommodate this.

[0078] The superimposed non-modal cuboids and / or 3D graphics are then used in 616-634 to facilitate driving-related operations of the AV (e.g., object detection, object tracking, object trajectory prediction, and / or AV trajectory determination). 616-624 are performed to select cuboids for subsequent use in training a machine learning algorithm. Figure 8 616-624 are repeated at each time interval of track 800).

[0079] At 616, it is determined whether the object is within a certain distance of the LiDAR sensor (e.g., Figure 2If the distance is less than the first distance (e.g., 30 meters) of the LiDAR sensor 264, then the amodal cuboid is selected for subsequent use, as shown in 618. The computing device may also perform temporal smoothing of the center of the amodal cuboid over time so that the trajectory of the object in the track is relatively smooth. Figure 8 An illustrative track 800 is shown in FIG. Track 800 includes a plurality of cuboids of an object over time. The size of the cuboids may be the same or different at each time interval. Techniques for temporal smoothing of the cuboid center positions are well known in the art.

[0080] If not [616: No], the method 600 proceeds to 622, where it is determined whether the object is located between a first distance (e.g., 30 meters) and a second distance (e.g., 50 meters) from the LiDAR sensor. It should be noted that the present solution is not limited to using two distance thresholds here. If there are other priors (such as the number of points in the loosely fitting cuboid, the motion model of the object), these other priors can be used to adjust the cuboid center.

[0081] When it is determined that the object is located at a distance greater than the second distance from the LiDAR system [622: No], method 600 proceeds to 628. In 624, the loosely fitted cuboid is selected for subsequent use. Conversely, when it is determined that the object is located at a distance greater than the second distance from the LiDAR sensor [622: Yes], method 600 proceeds to 626, where a weighted cuboid is generated using the loosely fitted cuboid and the amodal cuboid. The weighted cuboid is generated by the following mathematical equation (1).

[0082] W center (x,y,z)=(w1(L center (x,y,z))+w2(A center (x,y,x))) / (w1+w2) (1)

[0084] Among them, W center (x, y, z) represents the coordinates of the center of the weighted cuboid, L center (x, y, z) represents the coordinates of the center of the loosely fitted cuboid, A center (x,y,x) represents the coordinates of the center of the amodal cuboid, w1 represents the weight associated with the loosely fitting cuboid, and w2 represents the weight associated with the amodal cuboid. The overall size / area of ​​the weighted cuboid is set to the amodal size / area.

[0085] Once a cuboid is selected, method 600 may proceed to 628, where the selected cuboid is used to train one or more machine learning algorithms for object detection / classification purposes, motion prediction purposes, and / or motion planning purposes (e.g., to make predictions about object trajectories). Methods for training machine learning algorithms using cuboids are well known.

[0086] Next, method 600 continues Figure 6B Optional operations 630-634. Figure 6B As shown, 630-634 involve: performing object detection / classification operations using (one or more) machine learning algorithms; determining a predicted trajectory of the object and / or a trajectory of the AV based on the results of the object detection / classification operations; and causing the AV to follow the trajectory. Subsequently, execution proceeds to 636, where method 600 ends or performs other operations.

[0087] Reference now Fig. 9 , provides a flow chart for defining a non-modal cuboid according to the present solution. Fig. 9 As shown, operation 614 of FIG. 6 begins at 902 and continues at 904, where the LiDAR data points within each loosely fitting cuboid are identified. The identified LiDAR data points are processed at 906 to remove or otherwise filter ground points therefrom. Ground points may be identified using the content of a road / terrain map, known sensor heights, and / or a plane fitting algorithm. Road / terrain maps are well known. The remaining LiDAR data points associated with each loosely fitting cuboid are referred to as a set of LiDAR data points.

[0088] At 908, a tight fitting cuboid is defined that fits each set of LiDAR data points tightly. Each tight fitting cuboid is defined by transforming the coordinates of the loose fitting cuboid from a first coordinate system to a different second coordinate system; and adjusting the xyz coordinates of the cuboid corners to have a minimum value for a given number of LiDAR data points containing the object (with tolerance for outlier LiDAR data points). The first coordinate system may include a LiDAR system / sensor coordinate system, i.e., an xyz coordinate system with an origin of three axes at the center of the LiDAR system / sensor center. The second coordinate system may include an xyz coordinate system with an origin of three axes at the center of the object, with the x-axis pointing forward (i.e., toward the orientation of the object), and the z-axis pointing upward.

[0089] In some scenarios, a tightly fitting cuboid is defined by adjusting the coordinate values ​​of each face based on the coordinates of the LiDAR data point with the smallest x-axis value among the object's LiDAR data points, the LiDAR data point with the smallest y-axis value among the object's LiDAR data points, the LiDAR data point with the smallest z-axis value among the object's LiDAR data points, the LiDAR data point with the largest x-axis value among the object's LiDAR data points, the LiDAR data point with the largest y-axis value among the object's LiDAR data points, and the LiDAR data point with the largest z-axis value among the object's LiDAR data points. For example, the geometric coordinates of the tightly fitting cuboid are selected so that the listed LiDAR data points are contained therein. A tolerance threshold may need to be met. For example, 95% of all LiDAR data points of the object need to be included in the tightly fitting cuboid. The present solution is not limited to the details of these examples.

[0090] Fig.10 An illustration of a tightly fitting cuboid 1004 plotted on the graph 1000 is provided in FIG. Fig.10 As shown, data points of LiDAR dataset 1002 reside within cuboid 1004. LiDAR dataset 1002 may include Figure 7 1004 , where data points for the ground are removed from the LiDAR dataset 702 . Notably, edges 1006 , 1008 , 1010 , 1012 of the cuboid touch or are otherwise connected to data points of the LiDAR dataset 1002 . The present solution is not limited to the details of this illustration. For example, one or more data points of the LiDAR dataset may reside outside of the tight fitting cuboid 1004 based on a tolerance threshold. The present solution is not limited Fig.10 and / or example details.

[0091] Next, at 910, the LiDAR data points in the loosely fitted cuboid of the track are transformed into a second coordinate system and then accumulated into a single 3D graph. For example, Figure 8 The LiDAR data points of the loosely fitting cuboids 802, 804, 806, 808 of the track 800 are shown accumulated into a single 3D figure. A cuboid that tightly fits the accumulated LiDAR data points is defined at 912. This cuboid may be defined in the same or similar manner as the tightly fitting cuboid of 908. The cuboid of 912 is referred to herein as the accumulated fitting cuboid.

[0092] At 914 , an amodal range is calculated or otherwise determined using the cumulative fitted cuboid of the object and / or the previous amodal range(s). Fig.11A flowchart is provided in which can be used to understand how to calculate / determine the non-modal range. Fig.11 At 916 , a modeless cuboid is defined for the object. The modeless cuboid has dimensions specified by the modeless extent calculated at 914 .

[0093] At 918, xyz coordinates of a center of the non-modal cuboid are determined based on the tightly fitting cuboid and / or the loosely fitting cuboid. For example, the coordinates of the center of the non-modal cuboid are determined by: (i) aligning the corners of the non-modal cuboid with the closest visible corners of the tightly fitting cuboid; and (ii) aligning the corners of the non-modal cuboid so that the accuracy of the boundary between the non-modal cuboid and the loosely fitting cuboid is maximized. Once aligned, the coordinates of the center of the non-modal cuboid are specified or otherwise obtained. Subsequently, 920 is performed, where the method 900 ends or performs other operations.

[0094] Reference now Fig.11 , a flowchart of an illustrative method 1100 for determining a non-modal range is provided. The present solution is not limited to Fig.11 For example, operations 1104 and 1106 may be performed simultaneously or in parallel. Fig.11 Execute in the reverse order shown in .

[0095] Method 1100 starts at 1102 and continues to 1104, where a previous non-modal extent having a maximum size of the object is obtained. Another previous non-modal extent having a minimum size of the object is obtained at 1106. If the size of the cumulative fit cuboid is greater than the maximum previous size [1108: Yes], the non-modal extent is set equal to the maximum previous size, as shown in 1110. If the size of the cumulative fit cuboid is less than the minimum previous size [1112: Yes], the non-modal extent is set equal to the minimum previous size, as shown in 1114. Otherwise, the non-modal extent is set equal to the extent of the cumulative fit cuboid, as shown in 1116. Subsequently, 1118 is executed, where method 1100 ends or continues with other operations.

[0096] An illustrative algorithm for defining the height of a non-modal extent using the largest previous size is provided below.

[0097] ·std::min(std::max(frame_metadata.per_frame_fit_box.height,gt_acc_metadata.max_dims.get(“h”_s)),min_object_dimensions_.at(label_class.at(“height”)),max_object_dims.at(“height”));

[0098] Summary of steps (repeated for each frame, and step 3 is the final output from the last frame that exists):

[0099] 1.compute max of{per-frame-lidar-fit-height; current max height}->o_s1

[0100] 2.compute max of{o_s1;min-prior-size-fit-for-height}->o_s2

[0101] 3.compyte min of{o_s2,max-prior-size-for-height}

[0102] An illustrative algorithm for defining a non-modal range using a maximum previous size is provided below.

[0103] ●Compare tight fit with labels (GT / User labels)

[0104] ○Std:max(std::min(v_fit,v_label),v_current)←select size(v_current,v_fit,v_label)

[0105] ■v_fit->accumulated lidar fit (accumulated lidar fit)

[0106] v_current->current max length (current maximum length)

[0107] ■v_label->GT

[0108] max_I = std::min(select_size(gt_acc_metadata.max_bbox.length, I_fit_acc, frame_metadata.label_box.length), max_object_dims.at("length")); Overall summary (at each time step):

[0109] ●Find the minimum of{accumulated lidar fit;GT laebl}:o_s1[Note that this is the cumulative fit of the time steps]

[0110] ●Find the maximum of{o_s1;current_max_length};o_s2

[0111] ●Find the maximum of{o_s2;prior_max_dimensions}

[0112] Although the present solution has been shown and described with respect to one or more embodiments, equivalent changes and modifications will occur to those skilled in the art after reading and understanding this specification and the accompanying drawings. In addition, although specific features of the present solution may have been disclosed only with respect to one of several embodiments, such features may be combined with one or more other features of other embodiments, as may be desirable and advantageous for any given or specific application. Therefore, the breadth and scope of the present solution should not be limited by any of the above-described embodiments. Instead, the scope of the present solution should be defined in accordance with the appended claims and their equivalents.

Claims

1. A method for operating an autonomous vehicle, comprising: Obtaining, by a computing device, loosely fitted cuboids overlaid on the 3D shape such that each contains a LiDAR data point associated with a given object; defining, by the computing device, a non-modal cuboid based on the loosely fitting cuboid; using the amodal cuboid, by the computing device, to train a machine learning algorithm to detect a given class of objects using sensor data generated by a sensor of the autonomous vehicle or another vehicle; as well as Determining, by the computing device, whether the given object is less than a first distance from the LiDAR sensor, wherein when it is determined that the given object is less than the first distance from the LiDAR sensor, the non-modal cuboid is used to train the machine learning algorithm, and when it is determined that the given object is more than the first distance from the LiDAR sensor, at least one other cuboid is used, wherein the at least one other cuboid includes the loosely fitted cuboid or a weighted cuboid generated based on the loosely fitted cuboid and the non-modal cuboid; The computing device causes the use of the machine learning algorithm to control the operation of the autonomous vehicle.

2. The method according to claim 1, wherein: The defining includes determining coordinates of a center of the amodal cuboid based on at least one of a tight fitting cuboid or one of the loose fitting cuboids.

3. The method according to claim 2, wherein: The center of the non-modal cuboid is determined by: (i) aligning the corners of the non-modal cuboid with the closest visible corners of the tightly fitting cuboid; and (ii) aligning the corners of the non-modal cuboid so as to maximize the boundary accuracy between the non-modal cuboid and the loosely fitting cuboid.

4. The method according to claim 1, wherein: The defining includes identifying LiDAR data points within one of the loosely fitting cuboids.

5. The method according to claim 4, wherein: The defining includes removing data points associated with the ground from the LiDAR data points.

6. The method according to claim 4, wherein: The defining includes defining a close fitting cuboid that closely fits the LiDAR data points.

7. The method according to claim 6, wherein: The defining includes accumulating LiDAR data points in the loosely fitting cuboid.

8. The method according to claim 7, wherein: The defining includes defining a cumulative fitting cuboid of cumulative LiDAR data points that closely fits the loose fitting cuboid.

9. The method according to claim 8, wherein: The defining includes determining an amodal range using the cumulative fitted cuboid and a previous amodal range of the object.

10. The method according to claim 9, wherein: The modeless cuboid has dimensions specified by the modeless extent.

11. The method according to claim 9, wherein: When the size of the cumulative fitting cuboid is greater than a maximum previous size of a previous amodal range, the amodal range is set equal to the maximum previous size.

12. The method according to claim 9, wherein: When the size of the cumulative fitting cuboid is smaller than a minimum previous size of a previous amodal range, the amodal range is set equal to the minimum previous size.

13. The method according to claim 9, wherein: When the size of the amodal range falls between a maximum previous size and a minimum previous size of the previous amodal range, the amodal range is set equal to the range of the cumulative fitting cuboid.

14. A system for operating an autonomous vehicle, comprising: processor; A non-transitory computer-readable storage medium comprising programming instructions configured to cause the processor to implement a method for operating a vehicle, wherein the programming instructions include instructions to: Obtain loosely fitted cuboids overlaid on the 3D shape such that each contains a LiDAR data point associated with a given object; defining a non-modal cuboid based on the loosely fitted cuboid; using the amodal cuboid to train a machine learning algorithm to detect a given class of objects using sensor data generated by a sensor of the autonomous vehicle or another vehicle; as well as Determine whether the given object is less than a first distance from the LiDAR sensor, wherein when it is determined that the given object is less than the first distance from the LiDAR sensor, use the non-modal cuboid to train the machine learning algorithm, and when it is determined that the given object is greater than the first distance from the LiDAR sensor, use at least one other cuboid, wherein, The at least one other cuboid includes the loosely fitted cuboid or a weighted cuboid generated based on the loosely fitted cuboid and the non-modal cuboid; The machine learning algorithm is used to control the operation of the autonomous vehicle.

15. The system of claim 14, wherein: The modeless cuboid is defined by determining coordinates of a center of the modeless cuboid based on at least one of a tight fitting cuboid or one of the loose fitting cuboids.

16. The system of claim 15, wherein: The center of the non-modal cuboid is determined by: (i) aligning the corners of the non-modal cuboid with the closest visible corners of the tightly fitting cuboid; and (ii) aligning the corners of the non-modal cuboid so as to maximize the boundary accuracy between the non-modal cuboid and the loosely fitting cuboid.

17. The system of claim 14, wherein: The amodal cuboid is defined by identifying LiDAR data points within one of the loosely fitting cuboids.

18. The system of claim 17, wherein: The amodal cuboid is defined by removing data points associated with the ground from the LiDAR data points.

19. The system of claim 17, wherein: The amodal cuboid is defined by defining a close fitting cuboid that closely fits the LiDAR data points.

20. The system of claim 19, wherein: The amodal cuboid is defined by accumulating LiDAR data points in the loosely fitting cuboid.

21. The system of claim 20, wherein: The amodal cuboid is defined by defining a cumulative fitting cuboid of cumulative LiDAR data points that closely fits the loose fitting cuboid.

22. The system of claim 21, wherein: The amodal cuboid is defined by determining an amodal range using the cumulative fitted cuboid and a previous amodal range of the object.

23. The system of claim 22, wherein: The modeless cuboid has dimensions specified by the modeless extent.

24. The system of claim 21, wherein: When the size of the cumulative fitting cuboid is greater than a maximum previous size of a previous amodal range, the amodal range is set equal to the maximum previous size.

25. The system of claim 21, wherein: When the size of the cumulative fitting cuboid is smaller than a minimum previous size of a previous amodal range, the amodal range is set equal to the minimum previous size.

26. The system of claim 22, wherein: When the size of the amodal range falls between a maximum previous size and a minimum previous size of the previous amodal range, the amodal range is set equal to the range of the cumulative fitting cuboid.

Images