SPEED CORRECTION IN OBJECT POSE DETERMINATION

Abstract

A computer includes a processor and a memory, the memory storing instructions executable by the processor to generate first and second point sets from first and second scans acquired from a lidar sensor to determine a first velocity-compensated position of an object represented by a third point set at a first valid time intermediate between the respective times of the first and second scans. The instructions may additionally comprise receiving a parameter from the memory of the computer, the parameters determined based on a training process, to modify an amodal representation of the object, the modified amodal representation being determined based on a difference between a second velocity-compensated position of the object and an unmodified amodal representation of the object.The instructions may additionally comprise determining a pose of the object represented by the third point set based on the parameter.

Description

FIELD OF TECHNOLOGY

[0001] This disclosure relates to the control or operation of a vehicle in a traffic environment. GENERAL STATE OF THE ART

[0002] Modern vehicles may include a variety of sensors. Some sensors may detect static or moving objects external to the vehicle, such as other vehicles, lane markings on a roadway, traffic lights and / or signs, animals, natural objects, etc. Types of vehicle sensors may include radar sensors (e.g., wide-beam or narrow-beam radar sensors), ultrasonic sensors, satellite positioning system sensors (e.g., GPS), and light detection and ranging (lidar) devices. Output signals from sensors may be utilized by control units or the like that provide an output related to vehicle operation, including for controlling one or more vehicle components. SUMMARY

[0003] This disclosure describes techniques that may be provided for controlling or operating a machine, such as a vehicle operating in a traffic environment. Such techniques may include those utilized to determine the pose (i.e., the position and orientation) of a static or moving object visible to a lidar sensor or other type of sensor that generates a point set using energy reflected from a surface of the static or moving object. In one example, a lidar sensor may generate pulses of a coherent collimated light beam that are reflected from a surface of a static or moving vehicle during a lidar scan.The lidar sensor can output data that includes a measurement of a distance from the lidar sensor to a point on the surface of the object, which may include a static or moving vehicle. By collecting, for example, a large number of points reflected from the static or moving object, the lidar sensor can output data representing a set of points, such as a point cloud, in which each point represents a specific location on the surface of the static or moving object.

[0004] In another example, a narrow-beam radar sensor, such as a radar sensor that uses a phased array antenna to direct pulses of an RF, microwave, or millimeter-wave signal toward a static or moving object, may be used instead of, or in addition to, a lidar sensor. In such an example, the narrow-beam radar sensor may be operable to output data including a measurement of the distance from the narrow-beam radar sensor to a point on the surface of the static or moving object. By collecting a plurality of points reflected from the static or moving object, the narrow-beam radar sensor may output data representing a point set, where each point represents a particular location on the surface of the static or moving object.

[0005] In one example, a lidar sensor, such as a vehicle-mounted lidar sensor, may operate by scanning, e.g., in elevation and azimuth, with respect to a lens or other type of sensing surface of the lidar sensor. In one example, a vehicle-mounted lidar sensor may perform a measurement scan of a 360° field of view, e.g., in azimuth, with respect to the lidar sensor over a duration of 100 milliseconds, 500 milliseconds, one second, etc. Accordingly, as a lidar-equipped vehicle moves, for example, along a travel path 50 in a traffic environment, the lidar sensor may generate dozens or perhaps hundreds of point sets, each point set representing the pose of individual objects in a traffic environment. In response to obtaining output data representing the various point sets from the lidar sensor, a vehicle computer may classify the point sets, for example, according to a variety of classifications.In one example, output signals representing a set of measurement points on an object detected by a lidar sensor may be transmitted to a vehicle computer, which executes instructions to classify the static or moving object based on the geometric properties and / or pose of the object represented by the set of measurement points. Such classifications or labels may indicate a stationary vehicle, a moving vehicle, a lamppost, a traffic sign, a support pillar, a natural object (e.g., a tree, a bush, a rock, etc.), a bicycle, and so on. In one example, a point set may be classified based on points of the point set conforming to a predetermined shape and / or being within a relatively small distance from other points of the point set, e.g., within 0.25 meters, 0.5 meters, etc., with respect to the receiving surface of a vehicle-mounted lidar sensor, are classified as a rear surface of a vehicle.

[0006] In another example, a narrow-beam radar sensor, such as a vehicle-mounted radar sensor, may function similarly to a vehicle-mounted lidar sensor. In such an example, the radar sensor may generate dozens or perhaps hundreds of point sets as a radar-equipped vehicle moves along travel path 50 in a traffic environment, with each point set representing the pose of individual objects in a traffic environment. In response to obtaining output data representing the various point sets from the radar sensor, a vehicle computer may classify the point sets according to a variety of classifications.In one example, output signals representing a set of measurement points of an object detected by a radar sensor may be transmitted to a vehicle computer, which executes instructions to classify the static or moving object based on the geometric properties and / or pose of the object represented by the radar-measured points. Classifications or labels may indicate a stationary vehicle, a moving vehicle, a lamppost, a traffic sign, a support pillar, a natural object (e.g., a tree, a bush, a rock, etc.), a bicycle, and so on.

[0007] In one example, instructions executed by a vehicle computer may, for example, assign a three-dimensional geometric bin to a set of measurement points acquired during a lidar sensor scan and / or during a narrow-beam radar sensor scan. In this context, a “geometric bin” means a system or set of curved or unbent lines that encloses the volume of a set of points acquired by a sensor and assembled via vehicle computer programming. For example, instructions executed by a vehicle computer may assign a geometric bin, such as a hexahedron (i.e.,a geometric container having rectangular left, right, top, and bottom sides and square front and back sides), a cuboid (i.e., a geometric container having square left, right, top, bottom, front, and back sides), or a cuboid having a thickened appearance (i.e., a geometric container having one or more square sides and one or more boundaries having curved or arched edges). In this context, a “cuboid” means a three-dimensional geometric container having a top side, a bottom horizontally oriented side, two opposite side-oriented (i.e., left and right) sides, and two additional side-oriented (i.e.,rear and front) sides that enclose a stationary or moving object derived or determined from a measurement point set resulting from a lidar or radar sensor scan. In one example, a cuboid may enclose the volume of an object, such as a car, bus, bicycle, truck, camper trailer, or any other static or moving object that may be present in a traffic environment. In one example, instructions executed by a vehicle computer may track or monitor the movement of the cuboid as the object represented by the cuboid moves with respect to the vehicle-mounted lidar or radar sensor.

[0008] In a traffic environment, an object may experience a displacement during the sensor scan in response to the object being in motion with respect to a vehicle-mounted lidar or radar sensor. In one example, one or more points of a set of points reflected from an object during an initial portion of a sensor scan may appear displaced with respect to one or more points measured during a final portion of the sensor scan. For example, an object moving at 10 meters per second (approximately 22.4 miles per hour) may experience a displacement of 1.0 meter during a lidar or radar scan that occurs over a duration of 100 milliseconds. Accordingly, aggregating, e.g.,via instructions executed by a vehicle computer, the displacement of points in the point set acquired during the 100-millisecond lidar or radar sensor scan may introduce discrepancies in determining the actual location of points in the point set representing the object relative to the location of the lidar or radar sensor. Such displacement of points in the point set may cause the programming of a vehicle computer to construct a cuboid that has thickened or curved lines, which may, for example, impair the vehicle computer's ability to assign a class label to a moving object.

[0009] Additionally, a vehicle-mounted lidar or radar sensor may utilize two or more simultaneously scanning laser beams, such as a first scanning beam capable of detecting objects at relatively large distances from the vehicle, such as distances greater than 50 meters, greater than 100 meters, greater than 150 meters, etc., and a second scanning beam capable of detecting objects at closer distances from the vehicle, such as distances less than 50 meters, less than 25 meters, etc. In one example, at a first time (e.g., time T0), the first scanning lidar or radar beam may be directed toward areas in front of the vehicle, while a second scanning laser may be directed toward areas behind the vehicle. At a second time (e.g., time T1), the first scanning laser may be directed toward areas behind the vehicle, while the second scanning laser may be directed toward areas in front of the vehicle.Accordingly, instructions executed by the vehicle computer may attempt to assemble a geometric bin (e.g., a cuboid) representing an object by integrating output signals based, for example, on first and second lidar or radar scans taken over the first and second sampling intervals (e.g., T0 and T1). In response to the detected object being in motion, instructions executed by the vehicle computer may form a relatively large geometric bin having significantly curved, bulged, or thickened boundaries. Such large geometric bins may further impair the vehicle computer's ability to assign a class label to a moving object and / or determine its pose.

[0010] According to examples described herein, a point set obtained via a sensor scan and representing a moving object may be velocity-corrected to form a geometric container having a shape more closely resembling a cuboid shape, having substantially non-curved, e.g., horizontal and vertical, lines defining boundaries of the cuboid, rather than another type of geometric container having bulged, thickened, curved, or other-shaped lines defining the boundary of the geometric container. An ability to generate a cuboid having horizontal and vertical lines (or at least lines that are at least predominantly horizontal and vertical) may enhance an ability of instructions executed by a vehicle computer to label and / or classify a point set representing a moving object.Such a capability may additionally enhance an ability of a vehicle computer to separate point sets representing a moving object from other point sets representing other objects in the field of view, thereby enhancing an ability of a lidar or radar sensor interacting with a vehicle computer to label and / or classify other point sets representing additional static or moving objects detected in a traffic environment.

[0011] In one example, a vehicle computer may utilize an amodal representation of a point set, such as points representing the pose of a moving object detected in a traffic environment. In this context, an "amodal" or "amodalized" representation of an object refers to a representation of an object's pose obtained using an aggregation of one or more historical point sets describing the distance to at least a portion of the object acquired during a previous sensor scan. Thus, in one example, an amodal or amodalized representation of a moving vehicle detected in a traffic environment may be obtained by aggregating historical point sets acquired via previous lidar or radar sensor scans.Such lidar or radar sensor scans may include scans of a rear portion of the moving vehicle and / or a side portion of the moving vehicle, which may be aggregated via instructions executed by the vehicle computer. Through an aggregation of past sensor scans, each of which may provide a set of points representing a vehicle detected from different aspects, e.g., a left side portion of the vehicle, a right side portion of the vehicle, a rear portion of the vehicle, instructions executed by the vehicle computer may form a cuboid of the vehicle, allowing the vehicle computer to track the pose of the three-dimensional volume of the vehicle as the vehicle moves within the traffic environment.

[0012] During an amodalization process, a vehicle computer may use a calculated velocity of a moving object obtained via successive scans, e.g., a first and a second scan, of the moving object as input to the amodalization process. The velocity of the moving object may be calculated or inferred using the first and second sets of points representing the moving object obtained during the sequential lidar or radar sensor scans. In one example, instructions executed by a vehicle computer may calculate a first amodalized pose of a moving object to form a geometric container, such as a cuboid, which may include curved, bulged, or thickened lines defining the boundaries of the geometric container.The first amodalized representation of the moving object may represent the pose of the moving object at a first interpolated, e.g., a third, time point between successive lidar or radar sensor scans, e.g., a first and a second lidar or radar sensor scan. After forming the first geometric bin, the amodalization process may be performed again using the calculated or inferred velocity of the moving object as input signals to modify the amodalized representation. A geometric bin formed in response to the second execution of the amodalization process may include a geometric bin (e.g., a cuboid) having straightened or less curved, less bulged, or less thickened lines defining the boundaries of the geometric bin.Based on the cuboid being less curved, less curved, or less thickened, instructions executed by the vehicle computer may calculate a second velocity of the moving object at a second selectable interpolation time to align the modified amodalized pose of the object with an interpolated or third set of points derived from the successive lidar or radar sensor scans, e.g., the first and second lidar or radar sensor scans. Thus, in one example, a process of iterating is provided to determine a time at which amodalization using a calculated or derived velocity of a moving object results in a geometric container (e.g., a cuboid) having substantially non-curved, non-curved lines defining the geometric container.

[0013] In one example, a vehicle computer may acquire a parameter or set of parameters in response to training an offline computer, which may involve an offline computer implementing a machine learning application in a supervised, unsupervised, or reinforcement learning environment. Based on the training of a machine learning application, a parameter or set of parameters may be transmitted to an on-board vehicle computer, which may allow the vehicle computer to execute instructions to align an amodalized pose of a moving object encountered in a traffic environment to a set of points determined or inferred from successive lidar or radar sensor scans.Accordingly, an on-board vehicle computer may consume reduced processing resources when forming an amodalized pose representing a moving object, such as a geometric container (e.g., a cuboid) of the moving object that has straightened, less curved, less curved, or less thickened lines defining the boundaries of the container. In one example, a geometric container (e.g., a cuboid) that has straightened, i.e., less curved, less curved, or less thickened lines may improve the ability of the vehicle computer to assign a class label to the moving object and determine the actual pose of the moving object in a traffic environment.By determining the pose of the moving object, the vehicle computer may provide an output to control or regulate one or more vehicle operations or components, for example, providing assistance to a vehicle driver in a traffic environment.

[0014] In one example, a system may include a computer having a processor and a memory, wherein the memory may include instructions executable by the processor to generate first and second point sets from first and second scans acquired from a lidar sensor or from a radar sensor.The instructions may additionally include instructions to determine a first velocity-compensated position of an object represented by a third set of points at a first validity time that is between respective times of the first and second scans, and to receive a parameter from the memory of the computer, the parameter determined using a training process, to modify an amodal representation of the object, the modified amodal representation being determined based on a difference between a second velocity-compensated position of the object and an unmodified amodal representation of the object. The instructions may additionally include instructions to determine the pose of the object represented by the third set of points based on the parameter.

[0015] In one example, the parameter may be generated based on an iterative adjustment of the amodal representation of the object, wherein the iterative adjustment of the amodal representation is based on a difference between the second velocity-compensated position of the object and the modified amodal representation of the object being greater than a threshold.

[0016] In one example, the parameter may be determined based on the iterative adjustment of the amodal representation of the object, which is terminated in response to the difference between the second velocity-compensated position of the object and the modified amodal representation of the object being less than the threshold.

[0017] In one example, the iterative adjustment of the amodal representation of the object can be done via supervised machine learning.

[0018] In one example, the amodal representation of the object can be determined based on an aggregated history of samples of the object.

[0019] In one example, the instructions may further include instructions to create a geometric bin containing the third point set and to assign a class label to the geometric bin.

[0020] In one example, the class label assigned to the geometric container may be a cuboid enclosing a vehicle.

[0021] In one example, the instructions may further include instructions to actuate a vehicle component based on the determined pose of the object.

[0022] In one example, the vehicle component may be a steering component or a drive component of the vehicle.

[0023] In one example, the parameter may represent the first validity time at which the modified amodal representation of the object is computed.

[0024] In one example, the first validity time may be determined based on an interpolation between the unmodified amodal representation of the object and the modified amodal representation of the object.

[0025] In one example, a method may include generating first and second point sets from first and second scans acquired from a lidar sensor or a radar sensor. The method may additionally include determining a first velocity-compensated position of an object, represented by a third point set, at a first validity time that is between respective times of the first and second scans. The method may additionally include receiving a parameter from a computer memory, the parameter determined based on a training process, to modify an amodal representation of the object, wherein the modified amodal representation is determined based on a difference between a second velocity-compensated position of the object and an unmodified amodal representation of the object.The method may further include determining a pose of the object represented by the third point set based on the parameter.

[0026] In one example, the parameter may be determined based on an iterative adjustment of the amodal representation of the object, wherein the iterative adjustment of the amodal representation is based on a difference between the second velocity-compensated position of the object and the modified amodal representation of the object being greater than a threshold.

[0027] In one example, the parameter may be determined based on the iterative adjustment of the amodal representation of the object, which is terminated in response to the difference between the second velocity-compensated position of the object and the modified amodal representation of the object being less than the threshold.

[0028] In one example, an iterative adjustment of the amodal representation of the object can be performed via a supervised machine learning environment.

[0029] In one example, the amodal representation of the object can be determined based on an aggregated history of samples of the object.

[0030] In one example, the method may further include creating a geometric bin containing the third point set and assigning a class label to the geometric bin.

[0031] In one example, the method may further include actuating a vehicle component based on the determined pose of the object.

[0032] In one example, the vehicle component may be a steering component or a drive component.

[0033] In one example, the parameter may represent the first validity time at which the modified amodal representation of the object is computed. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram of an example vehicle. Fig. Figure 2A shows a system for processing an example scene including a static vehicle visible to a lidar or radar sensor. Fig. Figure 2B shows a system for processing an example scene that includes a moving vehicle visible to a lidar or radar sensor. Fig. Figure 3A is an example diagram showing the location of point sets determined from scans from a lidar or radar sensor. Fig. Figure 3B is an example diagram showing the location of velocity-corrected point sets determined from scans from a lidar or radar sensor. Fig. Figure 4 is a schematic representation of an exemplary training environment that generates a parameter for use in a vehicle computer. Fig. 5 is an example timeline illustrating interpolation between a first and a second set of points determined from scans of a vehicle's lidar or radar sensor. Fig. Figure 6 is a flowchart of a speed correction process in object pose determination. Fig. Figure 7 is a flowchart for a process for using a velocity correction parameter in a vehicle to determine the pose of an object. DETAILED DESCRIPTION

[0034] Fig. 1 is a block diagram of an example vehicle 100. The vehicle 100 may be any passenger vehicle or commercial vehicle, such as a car, a truck, a camper, a sport utility vehicle, a crossover, a van, a minivan, a taxi, a bus, etc. The vehicle 100 may include a computer 104, a communications network 106, a sensor set 108, vehicle actuators 110, a human-machine interface (HMI) 112, and a communications interface 114 (e.g., to provide Wi-Fi communications, communications with a satellite or terrestrial network, communications with other vehicles, etc.). In one example, the communications interface 114 may communicate with the off-board computer 115 via the wide area network 107.In one example, off-vehicle computer 115 may perform velocity correction of amodalized point sets, which may assist (or replace) amodalization functions performed by vehicle computer 104. In one example, off-vehicle computer 115 may represent a cloud computing service provider, such as Amazon.com, Inc., located at 410 Terry Ave. N, Seattle, WA 98109.

[0035] The sensor set 108 may include a camera sensor, a long-range radar sensor, an ultrasonic sensor, and a lidar / radar sensor 108A. The sensor set 108 may additionally include navigation sensors, such as satellite positioning system (e.g., GPS) sensors, inertial measurement unit sensors, etc. In one example, the lidar / radar sensor 108A may be mounted on an upward and / or outward-facing structure 103 of the vehicle body 102. In one example, the sensor 108A may include a lidar sensor having a laser emitter for emitting a coherent, collimated energy beam that can be directed at any azimuth angle with respect to the longitudinal axis of the vehicle body 102.In another example, sensor 108A may include a radar sensor having an RF, microwave, or millimeter-wave antenna for emitting a signal that can be directed at any azimuth angle relative to the longitudinal axis of vehicle body 102. Lidar / radar sensor 108A may operate by emitting the energy beam for a short period of time, e.g., a pulse width of 0.1 microseconds, one microsecond, two microseconds, etc., and measuring the time it takes for a reflected pulse to return to a detector of lidar / radar sensor 108A. Lidar / radar sensor 108A may transmit and receive thousands, hundreds of thousands, or any other number of pulses in a single second to output a set of reflected measurement points that can be used by the lidar / radar sensor to calculate distances to static or moving objects in the traffic environment of vehicle 100.In the example from . Fig. 1, the lidar / radar sensor 108A operates by scanning in azimuth (e.g., 360°) and elevation angles of ±-10°, ±15°, ±-20°, or another range of elevation angles with respect to the vehicle body 102.

[0036] In the example from Fig. 1, the lidar / radar sensor 108A provides a capability for simultaneous sensor scanning in opposite directions. As shown in Fig. 1, the lidar / radar sensor 108A may scan in a forward direction with respect to the vehicle body 102 at a first time while scanning in a rearward direction with respect to the vehicle body 102. In one example, the lidar / radar sensor 108A may generate a beam in a forward direction with respect to the vehicle body 102 at a first time, which beam is oriented in a first direction (e.g., in azimuth and elevation) at a time T0 and oriented in a second direction at a time T1. Accordingly, the lidar / radar sensor 108A may simultaneously transmit laser signals and receive reflected backbeams from objects located in a forward direction and from objects located rearward of the vehicle body 102 during a first scanning period (e.g., T0 to T1). The lidar / radar sensor 108A may perform the scanning, e.g.,in azimuth, into sectors during which a set of laser pulses is emitted and subsequently received. In one example, the lidar / radar sensor 108A may divide the 360° azimuth scan into 18 sectors, with each sector including a field of view, such as fields of view 150, extending over 20 degrees of arc in the azimuth plane.

[0037] In one example, the lidar / radar sensor 108A may continuously or intermittently sample regions around the vehicle body 102. Thus, in one example, scanning the field of view 150 at a first time may output a set of measurement points located in a forward direction relative to the vehicle body 102. At a second time, the field of view 150 may output a set of measurement points located rearward of the vehicle body 102. Accordingly, the fields of view 150 and 160 may alternate between being oriented in forward directions relative to the vehicle body 102 and in directions rearward of the vehicle body 102, thereby providing continuous sampling of all or substantially all areas exterior to the vehicle body 102.In one example, a first pulsed beam emitted by lidar / radar sensor 108A may include signals having a greater output power than a second pulsed beam emitted by the lidar / radar sensor. Thus, as lidar / radar sensor 108A rotates or pans, fields of view 150 and 160 rotate in azimuth with respect to vehicle body 102. Thus, objects detected within field of view 150 may include objects located at a greater distance from vehicle body 102 than objects detected within field of view 160. By utilizing different beam output power levels, the lidar / radar sensor 108A may have the capability to detect static or moving objects that may be close (e.g., up to 50 meters, up to 25 meters, up to 10 meters, etc.) from the vehicle body 102, as well as static or moving objects that may be farther (e.g.,up to 75 meters, up to 100 meters, up to 200 meters, etc.) from the vehicle body 102.

[0038] The vehicle actuators 110 may include actuators for controlling a propulsion system to convert stored energy (e.g., gasoline, diesel fuel, electric charge, etc.) into motion to propel the vehicle 100. The vehicle actuators 110 may include actuators for controlling a conventional vehicle propulsion subsystem, for example, a conventional powertrain including an internal combustion engine coupled to a transmission that transmits torque generated by the engine to the wheels of the vehicle 100. The vehicle actuators 110 may also include actuators for controlling a hybrid powertrain utilizing elements of a conventional powertrain and an electric powertrain; or may include any other type of powertrain.The vehicle actuators 110 may include an electronic control unit (ECU) or the like that communicates with and receives inputs from the vehicle computer 104 and / or a human operator. The human operator may control the drive system and / or a gear shift lever.

[0039] The vehicle actuators 110 may include actuators for controlling a conventional vehicle steering subsystem to turn the wheels of the vehicle 100. The steering subsystem may include rack and pinion controls with electric power steering, a steer-by-wire system, or another suitable system. The steering subsystem may include an electronic control unit (ECU) or the like that communicates with and receives inputs from the vehicle computer 104 and / or a human vehicle operator. The human vehicle operator may control the steering subsystem, for example, via a steering wheel.

[0040] An MMS 112 presents information to and receives information from a driver of the vehicle 100. The MMS 112 may include controls and displays positioned, for example, on a dashboard in a passenger compartment of the vehicle 100, or may be positioned elsewhere accessible to the driver of the vehicle 100. The MMS 112 may include dials, digital displays, screens, speakers, etc., for providing information to the driver of the vehicle 100. The MMS 112 may include buttons, knobs, keypads, microphones, and so on for receiving information from a driver of the vehicle 100.

[0041] The vehicle 100 may additionally include vehicle actuators 110 operable to exert a mechanical or electromotive force to control an aspect of the vehicle 100. For example, the vehicle actuators 110 may include a steering actuator operable to modify the orientation of the front wheels of the vehicle 100 in response to input from a human operator and / or a vehicle computer. In another example, the actuators 110 may include a vehicle propulsion component operable to reposition a throttle control of the vehicle 100 to increase or decrease the speed of the vehicle 100.

[0042] The computer 104 of the vehicle 100 and / or the off-board computer 115 may be a microprocessor-based computing device, e.g., a generic computing device including a processor and memory, an electronic controller, or the like, a field-programmable gate array (FPGA), a system-on-a-chip, an application-specific integrated circuit (ASIC), a combination of the above, etc. In one example, a hardware description language, such as VHDL (Very High Speed Integrated Circuit (VHSIC) Hardware Description Language), may be used in electronic design automation to describe digital and mixed-signal systems, such as FPGAs and ASICs.For example, an ASIC is manufactured based on VHDL programming provided prior to manufacture, whereas logical components within an FPGA may be configured based on the VHDL programming, e.g., stored on memory coupled to the FPGA circuitry. The vehicle computer 104 and / or the off-board computer 115 may thus include a processor, memory, etc. A memory of the computer 104 and / or the off-board computer 115 may include a physical medium for storing processor-executable instructions, as well as for electronically storing data and / or databases. Alternatively or additionally, the computer 104 and / or the off-board computer 115 may include structures, such as those above, by which executable instructions are provided.In one example, the computer 104 and / or the off-board computer 115 may consist of multiple computers coupled together to function as a single computing resource.

[0043] The vehicle computer 104 can transmit and receive data through the communication network 106. The communication network 106 can include, for example, a controller area network (CAN) bus, Ethernet, Wi-Fi, a local interconnect network (LIN), an on-board diagnostic port (OBD-II), and / or any other wired or wireless communication network. The vehicle computer 104 can be communicatively coupled to the lidar / radar sensor 108A, the vehicle actuators 110, the HMI 112, the communication interface 114, and other vehicle systems and / or subsystems via the communication network 106.

[0044] The vehicle computer 104 may execute instructions to perform signal processing of point sets representing objects detected via the lidar / radar sensor 108A. As described herein with reference to the Fig. 2-5, instructions executed by the computer 104 may be to assemble point sets from outputs of the lidar / radar sensor 108A to identify and / or classify such points as representing static or moving objects in the traffic environment of the vehicle 100. Based on repeated detections of point sets, the computer 104 may execute instructions to demodalize sets of detected points representing an object using previous point sets representing historical point sets representing the object. Additionally, an object is detected at a first and second sample time (e.g., T0 and T1). Additionally, instructions executed by the vehicle computer 104 may be to calculate and interpolate a speed of a moving object, e.g., using an optimization filter (i.e.,a Kalman filter, an extended Kalman filter, a particle filter, etc.) to update and refine the velocity of the moving object at a third point between successive sensor samples, e.g., a first and second sensor sample. A velocity of a moving object, such as at an interpolated point, may be used as input to an amodalized model of the moving object to perform a velocity correction of the amodalized model. Based on a modified and velocity-corrected amodalization of the moving object, the calculated pose of the object may be aligned to a velocity derived or interpolated, for example, from two or more measurements of the position of the moving object relative to the vehicle body 102.

[0045] Alternatively or additionally, the off-board computer 115, which communicates with the communication interface 114 of the vehicle 100, may execute instructions to perform signal processing of point sets representing objects detected via the lidar / radar sensor 108A. In one example, instructions executed by the off-board computer 115 may be used to assemble point sets from signals communicated, for example, via the wide area network 107, to assemble point sets from outputs of the lidar / radar sensor 108A. The off-board computer 115 may be used to assemble point sets from outputs of the lidar / radar sensor 108A to identify and / or classify such points as representing static or moving objects in the traffic environment of the vehicle 100.Based on repeated acquisitions of point sets, computer 104 may execute instructions to demodalize sets of acquired points representing an object using previous point sets representing historical point sets representing the object. Additionally, an object is detected at a first and second sample time (e.g., T0 and T1). Additionally, instructions executed by vehicle computer 104 may be to calculate and interpolate a speed of a moving object, e.g., using an optimization filter (i.e., a Kalman filter, an extended Kalman filter, a particle filter, etc.), to update and refine the speed of the moving object at a third point between successive sensor samples, e.g., a first and second sensor sample.A velocity of a moving object, such as at an interpolated point, can be used as input to an amodalized model of the moving object to perform a velocity correction of the amodalized model. Based on a modified and velocity-corrected amodalization of the moving object, the calculated pose of the object can be aligned to a velocity derived or interpolated, for example, from two or more measurements of the position of the moving object relative to the vehicle body 102.

[0046] In one example, instructions executed by the vehicle computer 104 and / or the off-board computer 115 may utilize a parameter uploaded by a machine learning application of a training network, which may include a neural network, to define settings or weights used by the computer 104 and / or the off-board computer 115. Settings or weights used by the vehicle computer 104 and / or the off-board computer 115 may serve to reduce the processing resources consumed by the computer 104 involved in aligning an amodalized representation of the pose of a moving object with respect to a point set derived from measurement samples of the lidar / radar sensor 108A. Example system processes

[0047] Fig. 2A shows a system 200 for processing an example scene 205 that includes a static vehicle visible to a lidar or radar sensor. In the example of Fig. 2A, a vehicle 210 may represent a stationary vehicle (V = 0) among other objects, such as buildings, street signs, etc., that may be detectable in the scene 205 using the vehicle-mounted lidar / radar sensor 108A. In one example, the vehicle 210 may represent a stationary vehicle observable within the fields of view 150 and 160 of the lidar / radar sensor 108A. Accordingly, successive scans of the lidar / radar sensor 108A may detect an area of the vehicle 210, which may result in a point set 220 representing the vehicle 210. As shown in Fig. 2A, the sensed area of the vehicle 210 results in a point set 220 being bounded within a two-dimensional or three-dimensional array of points, where each point within the point set 220 represents a point on the body of the vehicle 210. Additionally, subsequent scans of the vehicle 210 may result in inputs to an amodalization process performed by the vehicle computer 104 to form a cuboid having non-curved lines enclosing the volume of the vehicle 210, as shown by cuboid 235. In one example, in response to the lidar / radar sensor 108A being in motion with respect to the vehicle 210, instructions executed by the vehicle computer 104 and / or the off-board computer 115 may detect pose changes of the vehicle 210 resulting from such motion (e.g.,B Movement of the sensor 108A relative to the vehicle 210, which may be referred to as self-motion) by transferring coordinates of the point set 220 into a reference system related to the vehicle 100.

[0048] Fig. Figure 2B shows a system 250 for processing an example scene 255 that includes a moving vehicle visible to a lidar / radar sensor. In the example of Fig. 2B, a vehicle 260 may represent a moving vehicle among stationary objects, such as buildings, road signs, etc., that may be detected using the vehicle-mounted lidar / radar sensor 108A. In one example, the vehicle 260 may represent a moving vehicle observable within the fields of view 150 / 160 of the lidar / radar sensor 108A. Accordingly, successive scans of the lidar / radar sensor 108A may capture an area of the vehicle 210, which may result in a point set 220 representing the vehicle 260 as the vehicle moves relative to the vehicle 100.Based on the vehicle 260 being in motion relative to the vehicle 100, successive scans performed by the lidar / radar sensor 108A may result in certain points of a point set 275 being outside the actual or ground truth position of a particular area of the vehicle 260. Accordingly, in one example, during an initial portion of a scan of the lidar / radar sensor 108A, the sensor may detect certain points of the point set 275 that are displaced from other points of the point set 275. In one example, the displacement of points of the point set 275 may be expressed according to expression (1) below: Displacement=∫TpTSTOVVp(t)dt

[0049] In expression (1), T p represents the time at which a point of the point set 275 is recorded, T STOVrepresents a sampling validity time, which denotes the time at which the point set 275 is aggregated into a single validity time during an amodalization process executed by the computer 104 and / or the vehicle-external computer 115. V p(t) of expression (1) represents the speed of the moving object represented by the point set 275. In this context, a "scan valid time" or a "valid time" refers to a time at which a point set (e.g., 275) collected during a scan of the lidar or radar sensor is aggregated to form the point set. Thus, in one example, instructions executed by the vehicle computer 104 and / or the off-board computer 115 may attempt to aggregate a collected point set into a single valid time. Instructions executed by the vehicle computer 104 and / or the off-board computer 115 may thus attempt to include all collected points in the geometric bin 285, which may include a cuboid that may have thickened, curved, or bulged lines bounding the geometric bin.In one example, in response to the geometric container 285 being shaped as a cuboid having thickened, curved, or bulged lines, an ability of the computer 104 and / or the off-vehicle computer 115 to classify the vehicle 260 as representing a vehicle in a traffic environment may be impaired. Additionally, based on the geometric container 285 being shaped as a cuboid with respect to the cuboid 235 of FIG. Fig. 2A is relatively large, an ability of instructions executed by the computer 104 and / or the off-vehicle computer 115 to calculate the pose of the vehicle 260 and / or detect other objects present in the scene 255 may be impaired.

[0050] Fig. 3A is an example diagram 300 showing the location of point sets determined via scans from a lidar or radar sensor. In the example of Fig. 3A, point set 315 may represent a point set generated in response to a first scan by lidar / radar sensor 108A of an object in motion relative to vehicle 100. Point set 325 may represent a point set generated in response to a second scan by lidar / radar sensor 108A of the object in motion. Accordingly, based on the relative motion of the object with respect to vehicle 100, programming of computer 104 may execute instructions to demodalize point sets 315 and 325 to form a geometric container (e.g., cuboid 330) at a scan validity time, as described with respect to expression (1). In one example, an amodalization process results in the vehicle computer 104 generating a cuboid 330 at a sample validity time that includes bent, thickened, and / or curved lines that define the boundaries of the cuboid.In one example, cuboid 330 may represent a relatively large geometric container relative to a detected object in motion. In one example, for a moving object having a length, width, and height of one cubic meter, an amodalization process to create cuboid 330 may result in a cuboid enclosing a volume of, for example, two cubic meters, three cubic meters, four cubic meters, etc. Accordingly, based on cuboid 330 enclosing a relatively large volume, other static or moving objects that are also visible in scene 255 and may be adjacent to the moving object may remain undetected.Furthermore, based on the cuboid 330 enclosing a relatively large volume, the vehicle computer 104 and / or the off-board computer 115 may consume increased processing resources in determining the pose and / or a class label of a vehicle represented by the point sets 315 and 325.

[0051] Fig. 3B is an example diagram 350 showing the location of velocity-corrected point sets determined from scans of a lidar or radar sensor. In the example of Fig. 3B, point set 365 may represent a point set generated in response to a first scan by lidar / radar sensor 108A of an object in motion relative to vehicle 100. Point set 375 may represent a point set generated in response to a second scan by lidar / radar sensor 108A of the object in motion. Accordingly, programming of computer 104 or / or off-vehicle computer 115 may be executed based on the relative motion of the object relative to vehicle 100, including instructions to demodalize point sets 365 and 375 to form a geometric bin (e.g., cuboid 380) at a scan validity time, as described with respect to expression (1).

[0052] In the example from Fig. 3B, however, an amodalization process may be modified to include velocity correction inputs that may be calculated using a velocity derived or inferred from two or more (e.g., consecutive) lidar or radar sensor scans. Accordingly, as in Fig. 3B, the point set 365 may be modified or adjusted to overlap the point set 375. In one example, the point sets 365 and 375 may both undergo a speed correction, as indicated by arrows 370 and 380, to include a greater overlap with each other. Accordingly, in one example, an amodalization process performed by the vehicle computer 104 and / or the off-board computer 115 may result in the formation of a relatively small geometric container with respect to a detected object in motion. In the example of Fig. 3B, for a moving object having a length, width, and height of one cubic meter, an amodalization process for generating the cuboid 390 may result in a cuboid having a volume smaller than that of the cuboid 330 (from Fig. 3A), such as a cuboid enclosing a volume of, for example, 1.5 cubic meters, 1.3 cubic meters, 1.2 cubic meters, etc. Based on the cuboid 390 enclosing a relatively small volume, the vehicle computer 104 and / or the off-vehicle computer 115 may consume reduced processing resources in determining the pose of a vehicle represented by the point sets 315 and 325.

[0053] In one example, the velocity correction of amodalized point sets may proceed iteratively, with a velocity being calculated based on successive scans via the lidar / radar sensor 108A. Based on a calculated velocity, amodalized point sets may be modified, which may result in the generation of a cuboid enclosing a point set (e.g., 365, 375) within a smaller cuboid at a selectable and predetermined scan validity time. The iterative adjustment of the amodalized point sets may continue in response to a generated cuboid having progressively smaller dimensions. In response to successively generated cuboids converging at certain boundaries having straightened lines defining the boundaries of the cuboid, the iterative process may be halted.

[0054] The Fig. 3A and Fig. 3B may represent a training process performed by the offline machine learning application 415 (see Fig. 4). In response to the iterative process of refining a cuboid to enclose detected point sets (e.g., 365, 375), the machine learning application may adjust weights or settings utilizing characteristics of various moving objects encountered in a traffic environment. In one example, as further described with reference to the machine learning application 415, iterative adjustment of amodalized point sets may be used to refine a cuboid to enclose detected point sets (e.g., 365, 375). Such iterative adjustment may involve processing hundreds or thousands of point sets to obtain a set of one or more parameters that may be uploaded to the computer 104 of the vehicle 100 and / or uploaded to the off-board computer 115.In one example, the machine learning application 415 may utilize supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc., to determine whether the machine learning application can converge on a cuboid having straightened or non-bent lines defining the boundaries of the cuboid.

[0055] Fig. 4 is a schematic representation 400 of an exemplary training environment that generates a parameter for use in a vehicle computer 104 and / or for use by an off-board computer 115. In the example of Fig. 4, labeled lanes 405 may include actual (e.g., ground truth) point sets representing objects in motion, which are obtained using output data resulting from scans performed by the lidar / radar sensor 108A. In one example, labeled lanes 405 may include point sets representing moving bicycles, moving vehicles (e.g., automobiles, SUVs, trucks, buses, cargo vehicles, etc.) that may be moving in front of the vehicle 100, to the side of the vehicle 100, behind the vehicle 100, etc. Moving objects may include objects moving sideways with respect to the vehicle 100, objects moving toward the vehicle 100, objects moving away from the vehicle 100. In the example of Fig. 4 Objects may be marked or classified according to class markings such as ‘bicycle’, ‘compact vehicle’, ‘cargo vehicle’, ‘bus’, or using another marking or classification.

[0056] The amodalization component 410 represents computer-executable instructions for amodalizing a moving object. Accordingly, the amodalization component 410 represents a process for aggregating one or more historical point sets from at least a portion of the object acquired during one or more previous sensor scans. Historical point sets may include points resulting from lidar or radar sensor scans of a rear portion of a moving vehicle, a side portion of the moving vehicle, or a front portion of the moving vehicle, which may be aggregated via instructions executed by the vehicle computer 104 and / or the off-board computer 115.

[0057] Point sets and velocities derived or inferred using successive measurements from the lidar / radar sensor 108A may be formatted by the input layer to include parameters, ie, data values such as weights or settings, and input to the deep sensor training component 430 of the machine learning application 415. In the example of Fig. 4, the machine learning application 415 includes a neural network, such as a convolutional neural network. In this context, a convolutional neural network is a feed-forward artificial neural network having at least three layers (i.e., an input layer, an output layer, and at least one hidden layer). In one example, the input layer is to receive a set of points, such as the set of points related to Fig. 3B, which represent measurement points detected by the lidar / radar sensor 108A. Point sets may be representative of relatively large objects in motion (e.g., buses, freight vehicles, etc.) as well as relatively small objects in motion (e.g., bicycles, compact vehicles, etc.). Point sets may additionally represent objects moving at relatively low speeds, e.g., five kilometers per hour, 10 kilometers per hour, 15 kilometers per hour, as well as objects moving at higher speeds, such as speeds of 25 kilometers per hour, 30 kilometers per hour, 40 kilometers per hour, etc., relative to the vehicle 100.

[0058] Outputs from an output layer of the machine learning application 415 may be used to compute a loss function representing an ability of the machine learning application to accurately predict an expected output. The loss function may be backpropagated through hidden layers of the machine learning application 415, wherein the weights or settings stored in hidden layers of the machine learning application 415 are gradually changed to minimize the loss function. In this context, a "weight" or a "setting" refers to a parameter within the deep sensor training component 430 that at least partially controls or regulates the transformation or output of data from a machine learning application by performing an operation, e.g., an addition, multiplication, convolution, or other function, to provide data, e.g.,in an output layer that can be observed by a human, and / or by a detection and tracking metrics component 435.

[0059] In response to the loss function being sufficiently minimized, the machine learning application 415 may be considered trained, and the current parameters (e.g., formulated or derived from weights and / or settings within a hidden layer of the machine learning application 415) may be uploaded for use by the vehicle computer 104 and / or by the off-vehicle computer 115.

[0060] The lane dynamics inference component 420 is operable to infer or deduce a velocity of a point set representing a moving object. Thus, in one example, the lane dynamics inference component 420 calculates an estimate of the velocity of a moving object. Thus, in one example, the velocity of an object moving laterally with respect to the vehicle 100 may be estimated using a first point set collected at a validity time T=0 and a second point set collected at a validity time T=0.5 seconds. Based on the point set being separated by 1.0 meter, the velocity of the object may be inferred or deduced as moving laterally at a velocity of 2.0 meters per second (7.2 kilometers per hour).The signals output by the lane dynamics inference component 420 may then be sent to the amodalization component 410, which may be used to modify the amodalized point set with the calculated velocity.

[0061] As in Fig. 4, the amodalization component 410, the machine learning application 415, and the lane dynamics inference component 420 operate in an iterative loop. Accordingly, in one example, the lane dynamics inference component 420 outputs velocity corrections to amodalized point sets calculated by the amodalization component 410. Velocity-corrected amodalized point sets may then be input to the machine learning application 415, which modifies parameters (e.g., weights, settings, or parameters derived from weights and / or settings) utilized by the deep sensor training component 430. The machine learning application 415 may then interact with the lane dynamics inference component 420 to output an update to a calculated or derived velocity.An updated calculated or derived velocity may then be input to the amodalization component 410, which uses the updated calculated or derived velocity to output a modified amodalized point set. The modified amodalized point set may then be sent to the machine learning application 415 for further modifications to parameters (e.g., settings or weights) of the deep sensor training component 430.

[0062] In the example from Fig. 4, the lane dynamics inference component 420 may output a standardized text-based (e.g., JavaScript object notation) output file that may be input to the acquisition and tracking metrics component 435. The acquisition and tracking metrics component 435 may output a performance measurement of the alignment between velocity-corrected amodalized point sets and point sets representing output signals from a lidar sensor or from a radar sensor. In one example, the acquisition and tracking metrics component 435 may be used to identify divergence between velocity-corrected amodalized point sets and point sets representing output signals from the lidar / radar sensor 108A using output files from the lane dynamics inference component 420. In another example, the acquisition and tracking metrics component 435 may be used to identify slow convergence (i.e.,Hundreds of iterations, thousands of iterations, or any other number of iterations without resulting in alignment between velocity-corrected amodalized point sets and point sets representing output signals from lidar / radar sensors 108A). In one example, divergence or slow convergence between velocity-corrected amodalized point sets and point sets representing output signals from lidar / radar sensor 108A may indicate that a threshold level of training of machine learning application 415 has not been performed and that additional training may be performed.In one example, the amodalization component 410 may detect slow convergence between velocity-corrected amodalized point sets and point sets representing output signals from the lidar / radar sensor 108A in response to a difference between a velocity-compensated position of point sets and the amodalized representation of the point set in an iteration being less than a threshold (e.g., 2%, 1%, 0.5%, etc.).

[0063] In response to a training process of the machine learning application 415, one or more parameters may be uploaded to the computer 104 of the vehicle 100 and / or to the off-board computer 115. Such parameters may enhance the ability of the vehicle computer 104 and / or the off-board computer 115 to align amodalized point sets representing moving objects in a traffic environment with speed measurements calculated or derived from successive measurements from the lidar / radar sensor 108A. In one example, one or more parameters uploaded by the machine learning application 415 may be utilized by a pose calculation component of a vehicle application that controls the steering and / or propulsion of the vehicle, etc., in response to detecting, classifying, and / or labeling moving objects in the traffic environment.

[0064] Fig. 5 is an example timeline 500 showing an interpolation between a first and a second set of points determined via scans of a vehicle sensor, such as a lidar / radar sensor 108A. In Fig. In Figure 5, the horizontal axis represents the time at which a first lidar or radar sample can be modeled as occurring at a first validity time (sample 1) and a second validity time (sample 2). The interpolation time T Q represents a point at which an amodalized third point set can be rendered using sample 1 and sample 2. Thus, in one example, a point set acquired at the first validity time (sample 1) can be interpolated with a point set acquired at the second validity time (sample 2) to produce an amodalized third point set at T QIn response to a misalignment between the interpolated point set and the amodalized third point set at T Q the amodalized point set can be velocity corrected as described with reference to Fig. 4. In one example, a further iterative adjustment of an amodalized point set results in the rendering of the amodalized third point set at the interpolation time T Q . In one example, the coincidence of an amodalized point set with an interpolated point set acquired using a first and a second validity time (sample 1, sample 2) results in instructions executed by the vehicle computer 104 and / or the off-vehicle computer 115 generating a geometric container (e.g., a cuboid) having straightened, non-curved, non-bent lines defining the boundaries of the generated cuboid.

[0065] Fig. 6 is a flowchart for a process 600 for generating a velocity correction parameter in object pose determination. In one example, the process 600 occurs during a training process, where the machine learning application 415 uses a calculated velocity of a moving object, obtained over successive scans of the moving object, as input to the amodalization process. The velocity of the moving object may be calculated or derived using two or more sets of points representing the moving object obtained during different or successive lidar or radar sensor scans.In one example, instructions executed by the amodalization component 410, the machine learning application 415, and the lane dynamics inference component 420 may be used to compute a first amodalized representation of a moving object to form a geometric container, such as a hexahedron, a cuboid, etc., that may include curved, bulged, or thickened lines defining the boundaries of the geometric container. The first amodalized representation of the moving object may represent the position of the moving object at a first selectable interpolated or third time point between a first and second lidar or radar sensor sample.After forming the first geometric container, the amodalization process may be executed again, using the computer-derived velocity of the moving object as input signals to modify the amodalized representation. A geometric container formed in response to the second execution of the amodalization process may include a geometric container (e.g., a cuboid) having straightened, non-bent, or non-curved lines defining the boundaries of the container.Based on the cuboid being less curved, less curved, or less thickened, instructions executed by the vehicle computer may determine a second velocity of the moving object at a second selectable interpolation time to align the modified amodalized representation of the object with an interpolated or third set of points derived from the scans using the lidar / radar sensor 108A.

[0066] The process 600 begins at block 605, which involves the amodalization component 410 acquiring point sets in response to scanning the lidar / radar sensor 108A, which may be mounted on an upper outboard portion of the body 102 of the vehicle 100. Point sets may represent detected objects, such as bicycles, vehicles (e.g., cars, buses, trucks, RVs, etc.).

[0067] The process 600 continues at block 610, which includes an amodalization component 410 calculating a position of a moving object represented by point sets. The amodalization component 410 may calculate an amodalized representation of the moving object using an aggregation of one or more historical point sets of at least a portion of the detected object acquired during one or more previous sensor scans. Portions of detected objects may include side portions of moving vehicles, rear portions of moving vehicles, etc.

[0068] The process 600 continues at block 615, which includes the machine learning application 415 executing instructions to calculate a speed of a moving object using point sets determined by the lane dynamics inference component 420 in response to successive lidar or radar scans of the moving object. In one example, a speed of a moving object may be calculated by obtaining first and second point sets, calculating the displacement between the first and second point sets, and dividing by an interval between validity times of the first and second scans.

[0069] The process 600 continues at block 620, which includes the machine learning application 415 generating a cuboid enclosing velocity-corrected amodalized points representing the detected object. In one example, the velocity correction of the amodalized point set may represent the amodalized point set at an interpolated or third time point (T Q ) between validity times of consecutive lidar or radar sensor samples, such as sample 1 and sample 2, as described with reference to Fig. 5 described.

[0070] The process 600 continues at block 625 where the machine learning application 415 determines whether the cuboid enclosing the velocity-corrected amodalized point set includes thickened, curved, or bulged lines that define the boundary of the cuboid that is present at the interpolated or third time point (T Q). In response to the cuboid boundaries including thickened, curved, or arched lines, the process 600 continues to block 630, where the machine learning application 415 may modify the amodalized point set representing the moving object using the derived or calculated object velocity from block 615.

[0071] The process 600 returns to block 610, which includes calculating a (second) amodalized object position and further refining the speed of the moving object at block 615.

[0072] Process 600 continues at block 620, which includes generating a second cuboid to determine whether the velocity-corrected amodalized point set includes thickened, bent, or cambered lines defining the boundary of the cuboid. Blocks 625, 630, 610, 615, and 620 may be executed iteratively until a cuboid generated at block 620 is bounded by straightened, non-bent, or non-cambered lines.

[0073] In the example of process 600, in response to the machine learning application 415 determining that the cuboid enclosing the speed-corrected amodalized point set is acceptable, e.g., including straightened, non-bent, and / or non-cambered lines defining the cuboid, block 635 may be executed. At block 635, the machine learning application 415 may generate one or more parameters for in-vehicle use, such as with reference to Fig. 7 described.

[0074] After executing block 635, process 600 ends.

[0075] Fig. 7 is a flowchart for a process 700 for utilizing a velocity correction parameter in a vehicle to determine the pose of an object. One or more parameters derived during the training process (blocks 605-635) may be uploaded and stored in memory accessible by the vehicle computer 104 and / or the off-board computer 115 to calculate a velocity-corrected amodalized representation of a moving object detected in a traffic environment of the vehicle 100. In one example, a process of iterating to determine a time at which amodalization using a calculated or inferred velocity of a moving object results in a geometric bin (e.g., a cuboid) having substantially non-bent, non-cambered lines defining the geometric bin may be obtained.By modifying an amodalized representation of a moving object to include a velocity-corrected amodalized representation of the object, the vehicle computer 104 and / or the off-board computer 115 may compute a cuboid having straightened boundaries defining the cuboid. Thus, the vehicle computer 104 and / or the off-board computer 115 may determine a pose of the moving object for input to a vehicle-based assisted driving application.

[0076] The process 700 begins at block 705, which includes the machine learning application 415 sending one or more amodalization parameters for storage in a memory accessible to the vehicle computer 104 and / or the off-board computer 115. Amodalization parameters may include parameters such as an interpolation or a third time point (T Q) between validity times of lidar or radar sensor samples (e.g. sample 1 and sample 2, which are calculated with reference to Fig. 5) and / or other parameters (e.g., derived from weights or settings) of the machine learning application 415 when speed correcting an amodalized point set.

[0077] The process 700 continues at block 710, which includes generating point sets from lidar or radar sensor scans using the lidar / radar sensor 108A mounted on the vehicle body 102.

[0078] The process 700 continues at block 715, which includes the vehicle computer 104 and / or the off-board computer 115 switching between a first and a second set of points at an interpolation or third time point (T Q) interpolated. Block 715 may include the vehicle computer 104 and / or the off-board computer 115 forming a third point set from the interpolated first and second point sets.

[0079] The process 700 continues at block 720, which includes the vehicle computer 104 and / or the off-board computer 115 arranging an interpolated or third set of points into a cuboid.

[0080] The process 700 continues at block 725, which includes determining the pose of the cuboid positioned at block 720. In one example, determining the pose of a cuboid may be utilized by an assisted driving application executing on the vehicle computer 104 and / or the off-board computer 115.

[0081] The process 700 continues at block 730, which includes actuating a control component of the vehicle 100 to control one or more of a propulsion system, a steering system, etc., and / or the HMI 112. In one example, the vehicle computer 104 may control the actuators 110 to execute a driver assistance system (ADAS). ADAS are electronic technologies that assist drivers with driving and parking functions. Examples of ADAS include lane departure detection, blind spot detection, adaptive cruise control, and lane keeping assistance. The vehicle computer 104 may actuate a system of the vehicle 100 to stop the vehicle before it reaches a moving object represented by a set of points detected by the lidar / radar sensor 108A, according to an algorithm that operates without human input.

[0082] After executing block 730, process 700 ends.

[0083] In general, the described computing systems and / or devices may employ any of a variety of computer operating systems, including, but not limited to, versions and / or variants of the Ford Sync® application, the AppLink / Smart Device Link middleware, the Microsoft Automotive® operating system, the Microsoft Windows® operating system, the Unix operating system (e.g., the Solaris® operating system distributed by Oracle Corporation in Redwood Shores, California), the AIX UNIX operating system distributed by International Business Machines in Armonk, New York, the Linux operating system, the Mac OSX and iOS operating systems distributed by Apple Inc. in Cupertino, California, the BlackBerry OS distributed by Blackberry, Ltd. in Waterloo, Canada, and the Android operating system developed by Google, Inc. and the Open Handset Alliance, or the QNX® CAR Platform for Infotainment offered by QNX Software Systems.Examples of computing devices include, without limitation, an on-board vehicle computer, a computer workstation, a server, a desktop, notebook, laptop, or handheld computer, or other computing system and / or device.

[0084] Computing devices generally include computer-executable instructions, where the instructions may be executable by one or more computing devices, such as those listed above. Computer-executable instructions may be compiled or interpreted by computer programs created using a variety of programming languages and / or technologies, including, but not limited to, Java™, C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Perl, HTML, etc., either alone or in combination. Some of these applications may be compiled and executed on a virtual machine, such as the Java Virtual Machine, the Dalvik Virtual Machine, or the like. Generally, a processor (e.g., a microprocessor) receives instructions from, e.g., memory, a computer-readable medium, etc.and executes those instructions, thereby performing one or more processes that include one or more of the processes described herein. Such instructions and other data may be stored and transmitted using a variety of computer-readable media. A file in a computing device is generally a collection of data stored on a computer-readable medium, such as a storage medium, random access memory, etc.

[0085] A computer-readable medium (also referred to as a processor-readable medium) includes any non-transitory (e.g., physical) medium involved in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such a medium can take many forms, including, without limitation, non-transitory media and volatile media. Instructions can be transmitted through one or more transmission media, including fiber optics, wires, wireless communications, and internal structural elements comprising a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, a PROM, an EPROM, a FLASH EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0086] Databases, data repositories, or other data stores described in this document may include various types of mechanisms for storing, accessing, and retrieving various types of data, including a hierarchical database, a set of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), a non-relational database (NoSQL), a graph database (GDB), etc. Each such data store is generally contained within a computing device employing a computer operating system such as one of those listed above and may be accessed in one or more of a variety of ways over a network. A file system may be accessed by a computer operating system and may include files stored in various formats.An RDBMS generally uses the Structured Query Language (SQL) in addition to a language for creating, storing, editing, and executing stored procedures, such as the PL / SQL language mentioned above.

[0087] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on computer-readable media (e.g., disks, memory, etc.) associated with the computing devices. A computer program product may include such instructions stored on computer-readable media for performing the functions described herein.

[0088] In the drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements could be changed. With respect to the media, processes, systems, methods, heuristics, etc., described in this specification, it is understood that although the steps of such processes, etc., have been described as occurring according to a certain ordered sequence, such processes could be practiced with the described steps performed in an order that differs from the order described in this specification. It is further understood that certain steps could be performed concurrently, other steps could be added, or certain steps described in this specification could be omitted.Operations, systems and procedures described in this document should always be implemented and / or performed in accordance with any applicable owner / user manual and / or safety guidelines.

[0089] The disclosure has been described in an illustrative manner, and it is understood that the terminology used is intended to be descriptive rather than limiting. The adjectives "first" and "second" are used throughout the specification as identifiers and are not intended to indicate meaning, order, or quantity. The use of "in response to" and "at determining" indicates a causal relationship, not merely a temporal relationship. Many modifications and variations of the present disclosure are possible in light of the above teachings, and the disclosure may be practiced otherwise than as specifically described.

[0090] According to the present invention, a system is provided comprising: a computer having a processor and a memory, the memory including instructions executable by the processor to: generate a first and second set of points from a first and second sample obtained from a lidar sensor or a radar sensor; determine a first velocity-compensated position of an object represented by a third set of points at a first validity time that lies between respective times of the first and second sample;Receiving a parameter from the computer's memory, the parameter determined based on a training process to modify an amodal representation of the object, the modified amodal representation being determined based on a difference between a second velocity-compensated position of the object and an unmodified amodal representation of the object; and determining a pose of the object represented by the third point set based on the parameter.

[0091] According to one embodiment, the parameter is generated based on an iterative adjustment of the amodal representation of the object, wherein the iterative adjustment of the amodal representation is based on a difference between the second velocity-compensated position of the object and the modified amodal representation of the object being greater than a threshold value.

[0092] According to one embodiment, the parameter is determined based on the iterative adjustment of the amodal representation of the object, which is terminated in response to the difference between the second velocity-compensated position of the object and the modified amodal representation of the object being less than the threshold.

[0093] According to one embodiment, the iterative adjustment of the amodal representation of the object is performed via supervised machine learning.

[0094] According to one embodiment, the amodal representation of the object is determined based on an aggregated history of scans of the object.

[0095] According to one embodiment, the instructions further comprise instructions for: creating a geometric bin containing the third point set; and assigning a class label to the geometric bin.

[0096] According to one embodiment, the class label assigned to the geometric container is a cuboid enclosing a vehicle.

[0097] According to one embodiment, the instructions further comprise instructions for: actuating a vehicle component based on the determined pose of the object.

[0098] According to one embodiment, the vehicle component is a steering component or a drive component.

[0099] According to one embodiment, the parameter represents the first validity time at which the modified amodal representation of the object is calculated.

[0100] According to one embodiment, the first validity time is determined based on an interpolation between the unmodified amodal representation of the object and the modified amodal representation of the object.

[0101] According to the present invention, a method includes: generating first and second point sets from first and second scans obtained from a lidar sensor or a radar sensor; determining a first velocity-compensated position of an object represented by a third point set at a first validity time that is between respective times of the first and second scans; receiving a parameter from a computer memory, the parameter determined based on a training process, to modify an amodal representation of the object, the modified amodal representation being determined based on a difference between a second velocity-compensated position of the object and an unmodified amodal representation of the object; and determining a pose of the object represented by the third point set based on the parameter.

[0102] In one aspect of the invention, the parameter is determined based on an iterative adjustment of the amodal representation of the object, wherein the iterative adjustment of the amodal representation is based on a difference between the second velocity-compensated position of the object and the modified amodal representation of the object being greater than a threshold value.

[0103] In one aspect of the invention, the parameter is determined based on the iterative adjustment of the amodal representation of the object, which is terminated in response to the difference between the second velocity-compensated position of the object and the modified amodal representation of the object being less than the threshold.

[0104] In one aspect of the invention, the iterative adjustment of the amodal representation of the object is performed via a supervised machine learning environment.

[0105] In one aspect of the invention, the amodal representation of the object is determined based on an aggregated history of scans of the object.

[0106] In one aspect of the invention, the method includes: creating a geometric container containing the third point set and assigning a class label to the geometric container.

[0107] In one aspect of the invention, the method includes: actuating a vehicle component based on the determined pose of the object.

[0108] In one aspect of the invention, the vehicle component is a steering component or a drive component.

[0109] In one aspect of the invention, the parameter represents the first validity time at which the modified amodal representation of the object is calculated.

Claims

[1] Method comprising: Generating a first and a second point set from a first and a second sample obtained from a lidar sensor or from a radar sensor; Determining a first velocity-compensated position of an object represented by a third set of points at a first validity time that lies between respective times of the first and second scans; Receiving a parameter from a memory of a computer, the parameter being determined based on a training process to modify an amodal representation of the object, the modified amodal representation being determined based on a difference between a second velocity-compensated position of the object and an unmodified amodal representation of the object; and Determine a pose of the object represented by the third point set based on the parameter. [2] The method of claim 1, further comprising: Generating the parameter via an iterative adjustment of the amodal representation of the object, wherein the iterative adjustment of the amodal representation is based on a difference between the second velocity-compensated position of the object and the modified amodal representation of the object being greater than a threshold. [3] The method of claim 2, further comprising: Terminating the iterative adjustment of the amodal representation in response to the difference between the second velocity-compensated position of the object and the modified amodal representation of the object being less than the threshold. [4] The method of claim 2, wherein the iterative adjustment of the amodal representation of the object is performed via supervised machine learning. [5] The method of claim 1, further comprising: Determining the amodal representation of the object based on an aggregated history of samples of the object. [6] The method of claim 1, further comprising: Creating a geometric container containing the third point set; and Assign a class label to the geometric container. [7] The method of claim 6, wherein the class label assigned to the geometric container is a cuboid enclosing a vehicle. [8] The method of claim 1, further comprising: Actuating a vehicle component based on determining the pose of the object. [9] The method of claim 8, wherein the vehicle component is a steering component. [10] The method of claim 8, wherein the vehicle component is a drive component. [11] The method of claim 1, wherein the parameter represents the first validity time at which the modified amodal representation of the object is calculated. [12] The method of claim 10, further comprising: Determining the first validity time based on an interpolation between the unmodified amodal representation of the object and the modified amodal representation of the object. [13] The method of claim 1, wherein the sensor is a lidar sensor. [14] A computer programmed to carry out the method according to any one of claims 1-13. [15] A vehicle comprising the computer of claim 14.

Images