Velocity correction in object pose determination

Abstract

The invention provides velocity correction in object pose determination. A computer includes a processor and a memory storing instructions executable by the processor to generate a first set of points and a second set of points from a first scan and a second scan obtained from a lidar sensor, a first velocity compensated position of the object represented by a third set of points is determined at a first valid time between respective times of the first and second scans. The instructions may additionally be to receive a parameter from a memory of a computer, where the parameter is determined according to a training process to modify a non-modal representation of the object, the modified non-modal representation being determined according to a difference between a second velocity compensation position of the object and an unmodified non-modal representation of the object. The instructions may additionally be used to determine a pose of the object represented by the third set of points based on the parameters.

Description

Technical Field

[0001] The present disclosure relates to the control or operation of vehicles in a traffic environment. Background Art

[0002] Modern vehicles can include a variety of sensors. Some sensors can detect static or moving objects outside the vehicle, such as other vehicles, lane markings on the road, traffic lights and / or signs, animals, natural objects, etc. The types of vehicle sensors can include radar sensors (e.g., wide-beam or narrow-beam radar sensors), ultrasonic sensors, sensors of a satellite positioning system (e.g., GPS), and light detection and ranging (lidar) devices. Output signals from the sensors can be utilized by a control unit or the like, which provides outputs related to vehicle operation, including controlling one or more vehicle components. Summary of the Invention

[0003] The present disclosure describes techniques that can be provided for controlling or operating a machine, such as a vehicle operating in a traffic environment. Such techniques can include those for determining the pose (i.e., position and orientation) of static or moving objects observable by a lidar or other type of sensor that generates a point set using energy reflected from the surface of the static or moving object. In one example, a lidar sensor can generate pulses of a coherent collimated beam of light reflected from the surface of a static or moving vehicle during a lidar scan. The lidar sensor can output data including measurements of the distance from the lidar sensor to points on the surface of an object, which can include a static or moving vehicle. Via a set of multiple points reflected, for example, from a static or moving object, the lidar sensor can output data representing a point set (e.g., a point cloud), where each point represents a specific position on the surface of the static or moving object.

[0004] In another example, a narrow-beam radar sensor (such as a radar sensor that uses a phased array antenna to direct pulses of RF, microwave, or millimeter-wave signals towards a static or moving object) can be utilized instead of or in addition to the lidar sensor. In such an example, the narrow-beam radar sensor can operate to output data including measurements of the distance from the narrow-beam radar sensor to points on the surface of a static or moving object. Via a set of multiple points reflected from a static or moving object, the narrow-beam radar sensor can output data representing a point set, where each point represents a specific position on the surface of the static or moving object.

[0005] In one example, a lidar sensor (such as an automotive lidar sensor) can operate by scanning relative to the lens of the lidar sensor or other types of sensing surfaces, for example, in elevation and azimuth angles. In one example, an automotive lidar sensor can perform a measurement scan of a 360° field of view (e.g., azimuth angle) relative to the lidar sensor over a duration of 100 milliseconds, 500 milliseconds, one second, etc. Thus, when a vehicle equipped with lidar moves along a travel path 50 in a traffic environment, for example, the lidar sensor can generate dozens or possibly hundreds of point sets, where each point set represents the pose of an individual object in the traffic environment. In response to obtaining output data representing various point sets from the lidar sensor, a vehicle computer can classify the point sets according to multiple categories. In one example, an output signal representing a measurement point set of an object detected by the lidar sensor can be transmitted to the vehicle computer, which executes instructions to classify and / or label static or moving objects based on the geometric characteristics and / or pose of the object represented by the measurement point set. Such classification or labeling can represent a stationary vehicle, a moving vehicle, a lamp post, a traffic sign, a bridge abutment, natural objects (e.g., trees, shrubs, rocks, etc.), a bicycle, etc. In one example, based on the points in the point set conforming to a predetermined shape and / or being within a relatively small distance of other points in the point set (e.g., within 0.25 meters, 0.5 meters, etc. relative to the receiving surface of the automotive lidar sensor), the point set can be classified as the rear surface of a vehicle.

[0006] In another example, a narrow-beam radar sensor (such as an automotive radar sensor) can operate similarly to an automotive lidar sensor. In such examples, when a vehicle equipped with radar moves along a travel path 50 in a traffic environment, the radar sensor can generate dozens or possibly hundreds of point sets, where each point set represents the pose of an individual object in the traffic environment. In response to obtaining output data representing various point sets from the radar sensor, a vehicle computer can classify the point sets according to multiple categories. In one example, an output signal representing a measurement point set of an object detected by the radar sensor can be transmitted to the vehicle computer, which executes instructions to classify and / or label static or moving objects based on the geometric characteristics and / or pose of the object represented by the radar measurement point set. The classification or labeling can represent a stationary vehicle, a moving vehicle, a lamp post, a traffic sign, a bridge abutment, natural objects (e.g., trees, shrubs, rocks, etc.), a bicycle, etc.

[0007] In one example, instructions executed by a vehicle computer can, for example, assign a three-dimensional geometric container to a set of measurement points obtained during a scan of a lidar sensor and / or during a scan of a narrow-beam radar sensor. In this context, a "geometric container" refers to a system or a set of curved or non-curved lines that enclose a volume of a set of points detected by a sensor and aggregated via programming of the vehicle computer. For example, based on successive lidar or radar sensor scans of an object, such as lidar or radar sensor scans of the rear and side portions of a moving object, instructions executed by the vehicle computer can assign a geometric container, such as a hexahedron (i.e., a geometric container having a rectangular left side, right side, top side, and bottom side, and square front and rear sides), a cuboid (i.e., a geometric container having square left, right, top, bottom, front, and rear sides, or a cuboid with a thickened appearance (i.e., a geometric container having one or more square sides and one or more boundaries with curved or arcuate edges)). In this context, a "cuboid" means a three-dimensional geometric container that has a top side, a bottom horizontally oriented side, two relatively transverse oriented (i.e., left and right hand) sides, and two additional transverse oriented (i.e., rear and front) sides that enclose a stationary or moving object derived or determined from a set of measurement points generated by a lidar or radar sensor scan. In one example, a cuboid can enclose the volume of an object, such as a car, bus, bicycle, truck, camping trailer, or any other static or moving object that may be present in a traffic environment. In one example, when an object represented by a cuboid moves relative to an on-vehicle lidar or radar sensor, instructions executed by the vehicle computer can track or monitor the movement of the cuboid.

[0008] In a traffic environment, in response to an object being in motion relative to an on-vehicle lidar or radar sensor, the object can experience displacement during a sensor scan. In one example, one or more points in a set of points reflected from an object during a start portion of a sensor scan may appear displaced relative to one or more points measured during an end portion of the sensor scan. For example, an object traveling at 10 meters per second (about 22.4 miles per hour) can experience a 1.0-meter displacement during a lidar or radar scan that occurs over a 100-millisecond duration. Thus, aggregating points in a set of points obtained during a 100-millisecond lidar or radar sensor scan, for example, via instructions executed by the vehicle computer, can result in a discrepancy in determining the actual position of points in the set of points representing the object relative to the position of the lidar or radar sensor. Such displacement of points in the set of points can cause the programming of the vehicle computer to construct a cuboid with thickened or curved lines, which can reduce the vehicle computer's ability to, for example, assign class labels to moving objects.

[0009] Additionally, an in-vehicle lidar or radar sensor can utilize two or more simultaneously scanned laser beams, such as a first scanning beam capable of detecting objects at relatively large distances from the vehicle (such as distances greater than 50 meters, greater than 100 meters, greater than 150 meters, etc.) and a second scanning beam capable of detecting objects at smaller distances from the vehicle (such as distances less than 50 meters, less than 25 meters, etc.). In one example, at a first moment (e.g., time T0), the first scanning lidar or radar beam can be directed towards an area in front of the vehicle, while the second scanning laser can be directed towards an area behind the vehicle. At a second moment (e.g., time T1), the first scanning laser can be directed towards an area behind the vehicle, while the second scanning laser is directed towards an area in front of the vehicle. Thus, instructions executed by the vehicle computer can attempt to aggregate geometric containers (e.g., rectangular prisms) representing objects via integration of output signals based on, for example, the first and second lidar or radar scans occurring within the first and second scanning intervals (e.g., T0 and T1). In response to the detected object being in motion, the instructions executed by the vehicle computer can form relatively large geometric containers having boundaries that are significantly curved, bowed, or thickened. Such large geometric containers may further reduce the ability of the vehicle computer to assign class labels to moving objects and / or determine the pose of moving objects.

[0010] According to the examples described herein, velocity correction can be performed on a point set representing a moving object obtained via sensor scanning in order to form a geometric container having a shape that more closely conforms to the shape of a rectangular prism, having horizontal and vertical lines that are substantially non-curved, such as those defining the boundaries of the rectangular prism, rather than another type of geometric container having lines of another shape that are bowed, thickened, curved, or define the boundaries of the geometric container. The ability to generate a rectangular prism having horizontal and vertical lines (or at least lines that are at least primarily horizontal and vertical) can enhance the ability of the instructions executed by the vehicle computer to label and / or classify the point set representing the moving object. This ability can additionally improve the vehicle computer's ability to separate the point set representing the moving object from other point sets representing other objects in the field of view, thereby enhancing the ability of the lidar or radar sensor interacting with the vehicle computer to label and / or classify other point sets representing additional static or moving objects detected in the traffic environment.

[0011] In one example, a vehicle computer may utilize an amodal representation of a point set, such as points representing the pose of a moving object detected in a traffic environment. In this context, an "amodal" or "amodalized" representation of an object means a representation of the pose of the object obtained by aggregating one or more historical point sets that describe distances to at least a portion of the object obtained during previous sensor scans. Thus, in one example, an amodal or amodalized representation of a moving vehicle detected in a traffic environment may be obtained by aggregating historical point sets obtained via previous lidar or radar sensor scans. Such lidar or radar sensor scans may include scans of the rear portion and / or the side portions of the moving vehicle, and the scans may be aggregated via instructions executed by the vehicle computer. By aggregating past sensor scans, each of which may provide a point set representing the vehicle detected from a different aspect (e.g., the left side portion of the vehicle, the right side portion of the vehicle, the rear portion of the vehicle), program instructions executed by the vehicle computer may form the cuboid of the vehicle, thus allowing the vehicle computer to track the pose of the three-dimensional volume of the vehicle as the vehicle moves within the traffic environment.

[0012] During the demodulation process, the vehicle computer can utilize the calculated speed of a moving object obtained via consecutive (e.g., first and second) scans of the moving object as an input to the demodulation process. The first and second point sets representing the moving object obtained during consecutive lidar or radar sensor scans can be used to calculate or infer the speed of the moving object. In one example, instructions executed by the vehicle computer can calculate a first demodulated pose of the moving object to form a geometric container, such as a cuboid, that can include curved, arcuate, or thickened lines defining the boundaries of the geometric container. The first demodulated representation of the moving object can represent the pose of the moving object at a first interpolation (e.g., third) time between consecutive (e.g., first and second) lidar or radar sensor scans. After forming the first geometric container, the calculated or inferred speed of the moving object can be utilized as an input signal to re-execute the demodulation process, and the input signal operates to modify the demodulated representation. The geometric container formed in response to the second execution of the demodulation process can include a geometric container (e.g., cuboid) having straight or less curved, less arcuate, or less thickened lines defining the boundaries of the geometric container. Based on the cuboid being less curved, less arcuate, or less thickened, the instructions executed by the vehicle computer can calculate a second speed of the moving object at a second optional interpolation time to align the modified demodulated pose of the object with an interpolated or third point set derived from consecutive (e.g., first and second) lidar or radar sensor scans. Thus, in one example, an iterative process is provided to determine the time at which demodulation using the calculated or inferred speed of the moving object results in a geometric container (e.g., cuboid) having lines that are substantially non-curved and non-arcuate defining the geometric container.

[0013] In one example, a vehicle computer may obtain a parameter or set of parameters in response to training of an offline computer, which may include an offline computer implementing a machine learning application in a supervised, unsupervised, or reinforcement learning environment. Based on the training of the machine learning application, the parameter or set of parameters may be transmitted to the in-vehicle computer, which may allow the vehicle computer to execute instructions to align the unmodalized pose of a moving object encountered in a traffic environment with a point set determined or derived from successive lidar or radar sensor scans. Thus, the in-vehicle computer may consume reduced processing resources when forming an unmodalized pose of a moving object, such as a geometric container (e.g., a cuboid) of the moving object, the geometric container having lines that are straight, less curved, less bowed, or less thickened that define the boundaries of the container. In one example, a geometric container (e.g., a cuboid) having straight, i.e., less curved, less bowed, or less thickened lines, may enhance the ability of the vehicle computer to assign a class label to the moving object and determine the actual pose of the moving object in the traffic environment. By determining the pose of the moving object, the vehicle computer may provide an output to control or manage one or more vehicle operations or components, and thus assist, for example, a vehicle operator in the traffic environment.

[0014] In one example, a system may include a computer having a processor and a memory, where the memory may include instructions executable by the processor to perform the following operations: generate a first point set and a second point set based on a first scan and a second scan obtained from a lidar sensor or from a radar sensor. The instructions may additionally include instructions for performing the following operations: determine a first speed-compensated position of an object represented by a third point set at a first valid time between respective times of the first scan and the second scan, and receive a parameter from the computer's memory, the parameter being determined according to a training process for modifying an unmodal representation of the object, the modified unmodal representation being determined based on a difference between a second speed-compensated position of the object and an unmodified unmodal representation of the object. The instructions may additionally include instructions for determining a pose of the object represented by the third point set based on the parameter.

[0015] In one example, the parameter may be generated based on an iterative adjustment of the unmodal representation of the object, the iterative adjustment of the unmodal representation being based on a difference between the second speed-compensated position of the object and the modified unmodal representation of the object being greater than a threshold.

[0016] In one example, the parameter may be determined based on an iterative adjustment of the unmodal representation of the object that terminates in response to the difference between the second speed-compensated position of the object and the modified unmodal representation of the object being less than the threshold.

[0017] In one example, the iterative adjustment of the non-modal representation of the object can occur via supervised machine learning.

[0018] In one example, the non-modal representation of an object can be determined based on the aggregated history of scans of the object.

[0019] In one example, the instructions can further include instructions for generating a geometric container that includes the third point set and assigning a class label to the geometric container.

[0020] In one example, the class label assigned to the geometric container can be a cuboid that encloses the vehicle.

[0021] In one example, the instructions can further include instructions for actuating a vehicle component based on the determined pose of the object.

[0022] In one example, the vehicle component can be a steering component or a propulsion component of the vehicle.

[0023] In one example, the parameter can represent the first valid time when calculating the modified non-modal representation of the object.

[0024] In one example, the first valid time can be determined based on an interpolation between the unmodified non-modal representation of the object and the modified non-modal representation of the object.

[0025] In one example, a method can include generating a first point set and a second point set based on a first scan and a second scan obtained from a lidar sensor or from a radar sensor. The method can additionally include determining a first speed-compensated position of an object represented by a third point set at a first valid time between the respective times of the first scan and the second scan. The method can additionally include receiving a parameter from a computer memory, the parameter being determined according to a training process for modifying the non-modal representation of the object, the modified non-modal representation being determined based on a difference between a second speed-compensated position of the object and the unmodified non-modal representation of the object. The method can further include determining a pose of the object represented by the third point set based on the parameter.

[0026] In one example, the parameter can be determined based on an iterative adjustment of the non-modal representation of the object, the iterative adjustment of the non-modal representation being based on a difference between the second speed-compensated position of the object and the modified non-modal representation of the object being greater than a threshold.

[0027] In one example, the parameter can be determined based on the iterative adjustment of the amodal representation of the object that terminates when the difference between the second speed-compensated position of the object in response to the object and the modified amodal representation of the object is less than the threshold.

[0028] In one example, the iterative adjustment of the amodal representation of the object can occur via a supervised machine learning environment.

[0029] In one example, the amodal representation of the object can be determined based on the aggregated history of scans of the object.

[0030] In one example, the method can further include generating a geometric container including the third point set and assigning a class label to the geometric container.

[0031] In one example, the method can further include actuating a vehicle component based on the determined pose of the object.

[0032] In one example, the vehicle component can be a steering component or a propulsion component.

[0033] In one example, the parameter can represent the first effective time when calculating the modified amodal representation of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a block diagram of an exemplary vehicle.

[0035] Figure 2A illustrates a system for processing an exemplary scenario that includes a static vehicle observable by a lidar or radar sensor.

[0036] Figure 2B illustrates a system for processing an exemplary scenario that includes a moving vehicle observable by a lidar or radar sensor.

[0037] Figure 3A is an exemplary graph showing the positions of point sets determined by scans via a lidar or radar sensor.

[0038] Figure 3B is an exemplary graph showing the positions of speed-corrected point sets determined by scans via a lidar or radar sensor.

[0039] Figure 4 is a schematic diagram of an exemplary training environment for generating parameters for use in a vehicle computer.

[0040] Figure 5 is an exemplary timeline showing the interpolation between a first point set and a second point set determined by scans via a vehicle lidar or radar sensor.

[0041] Figure 6 is a flowchart of a process for velocity correction in object pose determination.

[0042] Figure 7 is a flowchart of a process for determining an object's pose using velocity correction parameters in a vehicle. Detailed implementation

[0043] Figure 1 is a block diagram of an exemplary vehicle 100. The vehicle 100 can be any passenger or commercial vehicle, such as a car, truck, recreational vehicle, sport utility vehicle, crossover vehicle, van, minivan, taxi, bus, etc. The vehicle 100 can include a computer 104, a communication network 106, a sensor set 108, vehicle actuators 110, a human-machine interface (HMI) 112, a communication interface 114 (e.g., to provide Wi-Fi communication, communication with a satellite or terrestrial network, communication with other vehicles, etc.). In one example, the communication interface 114 can communicate with an off-vehicle computer 115 via a wide area network 107. In one example, the off-vehicle computer 115 can perform velocity correction of an unmodalized point set, which can assist (or replace) the unmodalized function performed by the vehicle computer 104. In one example, the off-vehicle computer 115 can represent a cloud computing service provider, such as Amazon.com, Inc., located at 410 Terry Avenue North, Seattle, WA 98109.

[0044] The sensor set 108 may include a camera sensor, a remote radar sensor, an ultrasonic sensor, and a lidar / radar sensor 108A. The sensor set 108 may additionally include navigation sensors, such as sensors of a satellite positioning system (e.g., GPS), sensors of an inertial measurement unit, etc. In one example, the lidar / radar sensor 108A may be mounted on an upward and / or outward-facing structure 103 of the vehicle body 102. In one example, the sensor 108A may include a lidar sensor having a laser emitter for emitting a coherent collimated energy beam that may be directed at any azimuth angle relative to the longitudinal axis of the vehicle body 102. In another example, the sensor 108A may include a radar sensor having an RF, microwave, or millimeter-wave antenna for emitting a signal that may be directed at any azimuth angle relative to the longitudinal axis of the vehicle body 102. The lidar / radar sensor 108A may operate by emitting an energy beam within a short time period (e.g., a pulse width of 0.1 microsecond, 1 microsecond, 2 microseconds, etc.) and measuring the time it takes for the reflected pulse to return to the detector of the lidar / radar sensor 108A. The lidar / radar sensor 108A may emit and receive thousands, hundreds of thousands, or another number of pulses within a single second in order to output a set of reflected measurement points that may be utilized by the lidar / radar sensor to calculate distances to static or moving objects in the traffic environment of the vehicle 100. In Figure 1 an example, the lidar / radar sensor 108A operates by scanning at an azimuth angle (e.g., 360°) and + -10°, + 15°, + an elevation angle of -20° or another range of elevation angles relative to the vehicle body 102.

[0045] In Figure 1 an example, the lidar / radar sensor 108A provides the ability to perform sensor scans simultaneously in opposite directions. As Figure 1As shown, at a first time, the lidar / radar sensor 108A can scan in the direction in front of the vehicle body 102 while also scanning in the direction behind the vehicle body. In one example, at a first moment, the lidar / radar sensor 108A can generate a beam in the direction in front of the vehicle body 102, the beam being oriented in a first direction (e.g., in terms of azimuth and elevation) at time T0 and in a second direction at time T1. Thus, during a first scan duration (e.g., T0 to T1), the lidar / radar sensor 108A can simultaneously emit laser signals and receive reflected echoes from objects located in the forward direction and from objects located behind the vehicle body 102. The lidar / radar sensor 108A can, for example, subdivide the scan in azimuth into sectors during which a set of laser pulses are emitted and then received. In one example, the lidar / radar sensor 108A can subdivide a 360° azimuth scan into 18 sectors, where each sector includes a field of view with an angular separation of 20° in the azimuth plane, such as the field of view 150.

[0046] In one example, the lidar / radar sensor 108A can scan the area around the vehicle body 102 continuously or intermittently. Thus, in one example, at a first moment, the scan of the field of view 150 can output a set of measurement points located in the forward direction relative to the vehicle body 102. At a second moment, the field of view 150 can output a set of measurement points located behind the vehicle body 102. Thus, the fields of view 150 and 160 can alternate between being oriented in the forward direction relative to the vehicle body 102 and in the direction behind the vehicle body 102, thereby providing a continuous scan of all or substantially all areas outside the vehicle body 102. In one example, the first pulse beam emitted from the lidar / radar sensor 108A can include a signal having a greater output power than the second pulse beam emitted from the lidar / radar sensor. Thus, as the lidar / radar sensor 108A rotates or sweeps, the fields of view 150 and 160 rotate in azimuth relative to the vehicle body 102. Thus, compared to the objects detected within the field of view 160, the objects detected within the field of view 150 can include objects located at a greater distance from the vehicle body 102. By utilizing different beam output power levels, the lidar / radar sensor 108A can include the ability to detect static or moving objects that may be close to the vehicle body 102 (e.g., up to 50 meters, up to 25 meters, up to 10 meters, etc.) as well as static or moving objects that may be located farther from the vehicle body 102 (e.g., up to 75 meters, up to 100 meters, up to 200 meters, etc.).

[0047] The vehicle actuator 110 may include an actuator for controlling a propulsion system to convert stored energy (e.g., gasoline, diesel fuel, charge, etc.) into motion to propel the vehicle 100. The vehicle actuator 110 may include an actuator for controlling a conventional vehicle propulsion subsystem, such as a conventional powertrain that includes an internal combustion engine coupled to a transmission, and the transmission transmits the torque generated by the engine to the wheels of the vehicle 100. The vehicle actuator 110 may also include an actuator for controlling a hybrid powertrain that utilizes elements of a conventional powertrain and an electric powertrain; or may include another type of powertrain. The vehicle actuator 110 may include an electronic control unit (ECU) or the like that communicates with and receives input from the vehicle computer 104 and / or a human operator. The human operator may control the propulsion system and / or the shift lever.

[0048] The vehicle actuator 110 may include an actuator for controlling a conventional vehicle steering subsystem to turn the wheels of the vehicle 100. The steering subsystem may include a rack and pinion steering member with electric power steering, a steer-by-wire system, or another suitable system. The steering subsystem may include an electronic control unit (ECU) or the like that communicates with and receives input from the vehicle computer 104 and / or a human operator. The human operator may control the steering subsystem via, for example, a steering wheel.

[0049] The HMI 112 presents information to and receives information from the operator of the vehicle 100. The HMI 112 may include controls and displays located on, for example, a dashboard in the passenger compartment of the vehicle 100, or may be located at another position accessible to the operator of the vehicle 100. The HMI 112 may include dials, digital displays, screens, speakers, etc. for providing information to the operator of the vehicle 100. The HMI 112 may include buttons, knobs, keypads, microphones, etc. for receiving information from the operator of the vehicle 100.

[0050] The vehicle 100 may additionally include the vehicle actuator 110 that operates to apply a mechanical force or electromotive force to control an aspect of the vehicle 100. For example, the vehicle actuator 110 may include a steering actuator that operates in response to input from a human operator and / or the vehicle computer to modify the orientation of the front wheels of the vehicle 100. In another example, the actuator 110 may include a vehicle propulsion component that operates to reposition a throttle control of the vehicle 100 to increase or decrease the speed of the vehicle 100.

[0051] The computer 104 of the vehicle 100 and / or the off-vehicle computer 115 may include a microprocessor-based computing device, such as a general-purpose computing device (which includes a processor and a memory, an electronic controller, etc.), a field-programmable gate array (FPGA), a system-on-chip, an application-specific integrated circuit (ASIC), a combination of the foregoing, etc. In one example, a hardware description language such as VHDL (VHSIC (Very High Speed Integrated Circuit) Hardware Description Language) may be utilized in electronic design automation to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided prior to manufacturing, and the logic components within an FPGA may be configured based on, for example, VHDL programming stored in a memory coupled to the FPGA circuit. Thus, the vehicle computer 104 and / or the off-vehicle computer 115 may include a processor, a memory, etc. The memory of the computer 104 and / or the off-vehicle computer 115 may include a tangible medium for storing instructions executable by the processor and for electronically storing data and / or databases. Alternatively or additionally, the computer 104 and / or the off-vehicle computer 115 may include a structure that provides executable instructions, such as the foregoing structures. In one example, the computer 104 and / or the off-vehicle computer 115 may be multiple computers coupled together to operate as a single computing resource.

[0052] The vehicle computer 104 may transmit and receive data via the communication network 106. The communication network 106 may include, for example, a controller area network (CAN) bus, Ethernet, WiFi, a local interconnect network (LIN), an on-board diagnostic connector (OBD-II), and / or another wired or wireless communication network. The vehicle computer 104 may be communicatively coupled via the communication network 106 to the lidar / radar sensor 108A, the vehicle actuator 110, the HMI 112, the communication interface 114, and other vehicle systems and / or subsystems.

[0053] The vehicle computer 104 may execute instructions to perform signal processing on a point set representing an object detected via the lidar / radar sensor 108A. As described herein with reference to FIGS. 2 to Figure 5More specifically, the instructions executed by computer 104 can operate to aggregate a set of points from the output of lidar / radar sensor 108A to identify and / or classify such points as representing static or moving objects in the traffic environment of vehicle 100. Based on repeated detections of the set of points, computer 104 can execute instructions to denoise the detected set of points representing an object using past sets of points representing the historical set of points of the object. Additionally, an object can be detected at a first scan time and a second scan time (e.g., T0 and T1). Additionally, the instructions executed by vehicle computer 104 can operate to calculate the speed of a moving object and interpolate, for example, using an optimized filter (i.e., Kalman filter, extended Kalman filter, particle filter, etc.) to update and refine the speed of the moving object at a third point between the consecutive (e.g., first and second) sensor scans. The speed of the moving object (such as at the interpolated point) can be used as an input to the denoising model of the moving object in order to perform speed correction on the denoising model. Based on the modified and speed-corrected denoising of the moving object, the calculated pose of the object can be aligned with the speed inferred or interpolated based on two or more measurements of the position of the moving object relative to the vehicle body 102.

[0054] Alternatively or additionally, an off-vehicle computer 115 communicating with communication interface 114 of vehicle 100 can execute instructions to perform signal processing on a set of points representing an object detected via lidar / radar sensor 108A. In one example, the instructions executed by off-vehicle computer 115 can operate to aggregate the set of points according to a signal transmitted via wide area network 107, for example, to aggregate the set of points according to the output of lidar / radar sensor 108A. Off-vehicle computer 115 can operate to aggregate a set of points from the output of lidar / radar sensor 108A to identify and / or classify such points as representing static or moving objects in the traffic environment of vehicle 100. Based on repeated detections of the set of points, computer 104 can execute instructions to denoise the detected set of points representing an object using past sets of points representing the historical set of points of the object. Additionally, an object can be detected at a first scan time and a second scan time (e.g., T0 and T1). Additionally, the instructions executed by vehicle computer 104 can operate to calculate the speed of a moving object and interpolate, for example, using an optimized filter (i.e., Kalman filter, extended Kalman filter, particle filter, etc.) to update and refine the speed of the moving object at a third point between the consecutive (e.g., first and second) sensor scans. The speed of the moving object (such as at the interpolated point) can be used as an input to the denoising model of the moving object in order to perform speed correction on the denoising model. Based on the modified and speed-corrected denoising of the moving object, the calculated pose of the object can be aligned with the speed inferred or interpolated based on two or more measurements of the position of the moving object relative to the vehicle body 102.

[0055] In one example, instructions executed by vehicle computer 104 and / or off-vehicle computer 115 may utilize parameters uploaded from a machine learning application of a training network, which may include a neural network, and the parameters operate to define settings or weights utilized by computer 104 and / or off-vehicle computer 115. The settings or weights utilized by vehicle computer 104 and / or off-vehicle computer 115 may operate to reduce the processing resources consumed by computer 104 in aligning an amodal representation of the pose of a moving object relative to a point set derived from a measurement scan of lidar / radar sensor 108A.

[0056] Exemplary system operations

[0057] Figure 2A Shown is a system 200 for processing an exemplary scenario 205, the scenario including a static vehicle observable by a lidar or radar sensor. In Figure 2A the example, vehicle 210 may represent a stationary vehicle (V = 0) and other objects (such as buildings, road signs, etc.), which may be detected in scenario 205 using on-vehicle lidar / radar sensor 108A. In one example, vehicle 210 may represent a stationary vehicle observable within the fields of view 150 and 160 of lidar / radar sensor 108A. Thus, successive scans of lidar / radar sensor 108A may detect the surface of vehicle 210, which may result in a point set 220 representing vehicle 210. As Figure 2A shown, the detected surface of vehicle 210 causes point set 220 to be constrained within a two- or three-dimensional point array, where each point within point set 220 represents a point on the body of vehicle 210. Additionally, subsequent scans of vehicle 210 may produce an input to an amodalization process executed by vehicle computer 104 for forming a cuboid with non-bending lines enclosing the volume of vehicle 210, as shown by cuboid 235. In one example, in response to lidar / radar sensor 108A being in motion relative to vehicle 210, instructions executed by vehicle computer 104 and / or off-vehicle computer 115 may compensate for a change in the pose of vehicle 210 caused by such motion (e.g., the motion of sensor 108A relative to vehicle 210, which may be referred to as ego-motion) by transforming the coordinates of point set 220 to the coordinate system of reference vehicle 100.

[0058] Figure 2B Shown is a system 250 for processing an exemplary scenario 255, the scenario including a moving vehicle observable by a lidar / radar sensor. In Figure 2BIn the example, vehicle 260 may represent a moving vehicle as well as stationary objects (such as buildings, road signs, etc.), which may be detected using on-vehicle lidar / radar sensor 108A. In one example, vehicle 260 may represent a moving vehicle observable within the field of view 150 / 160 of lidar / radar sensor 108A. Thus, successive scans of lidar / radar sensor 108A may detect the surface of vehicle 210, which may generate a point set 220 representing vehicle 260 as vehicle 260 moves relative to vehicle 100. Based on vehicle 260 being in motion relative to vehicle 100, successive scans performed by lidar / radar sensor 108A may result in some points in point set 275 being outside the actual or ground truth position of a particular surface of vehicle 260. Thus, in one example, during an initial portion of the scan of lidar / radar sensor 108A, the sensor may detect some points in point set 275 that are displaced from other points in point set 275. In one example, the displacement of the points in point set 275 may be expressed according to the following expression (1):

[0059]

[0060] In expression (1), T p represents the time at which a point in point set 275 is detected, and T STOV represents the scan valid time, which represents the time at which point set 275 is aggregated into a single valid time during the non-modalization process performed by computer 104 and / or off-vehicle computer 115. V p (t) in expression (1) represents the speed of the moving object represented by point set 275. In this context, "scan valid time" or "valid time" refers to the moment at which a point set (e.g., 275) detected during a lidar or radar sensor scan is aggregated to form the point set. Thus, in one example, instructions executed by vehicle computer 104 and / or off-vehicle computer 115 may attempt to aggregate the detected point sets into a single valid time. Thus, instructions executed by vehicle computer 104 and / or off-vehicle computer 115 may attempt to fit all detected points into a geometric container 285, which may include a cuboid having thickened, curved, or arcuate lines defining the geometric container. In one example, in response to geometric container 285 being shaped as a cuboid with thickened, curved, or arcuate lines, the ability of computer 104 and / or off-vehicle computer 115 to classify vehicle 260 as representing a vehicle in a traffic environment may be reduced. Additionally, based on geometric container 285 being relatively large with respect to Figure 2A the cuboid 235, the ability of instructions executed by computer 104 and / or off-vehicle computer 115 to calculate the pose of vehicle 260 and / or detect other objects present in scene 255 may be reduced.

[0061] Figure 3A Exemplary graph 300 shows the positions of a set of points determined via scans of a lidar or radar sensor. In Figure 3A the example, the set of points 315 may represent a set of points generated in response to a first scan of an object moving relative to vehicle 100 by lidar / radar sensor 108A. The set of points 325 may represent a set of points generated in response to a second scan of the moving object by lidar / radar sensor 108A. Thus, based on the relative motion of the object with respect to vehicle 100, the programming of computer 104 may execute instructions to unmodalize the sets of points 315 and 325 to form a geometric container (e.g., cuboid 330) at the scan valid time as described with respect to expression (1). In one example, the unmodalization process causes vehicle computer 104 to generate cuboid 330 at the scan valid time, the cuboid including curved, thickened, and / or bowed lines that define the boundaries of the cuboid. In one example, cuboid 330 may represent a relatively large geometric container with respect to the detected moving object. In one example, for a moving object having a length, width, and height of one cubic meter, the unmodalization process for generating cuboid 330 may cause the cuboid to enclose a volume of, for example, two cubic meters, three cubic meters, four cubic meters, etc. Thus, based on cuboid 330 enclosing a relatively large volume, other static or moving objects that may be visible in scene 255 and that may be near the moving object may remain undetected. Additionally, based on cuboid 330 enclosing a relatively large volume, vehicle computer 104 and / or off-vehicle computer 115 may consume increased processing resources when determining the pose and / or class label of the vehicle represented by sets of points 315 and 325.

[0062] Figure 3B Exemplary graph 350 shows the positions of a velocity-corrected set of points determined via scans of a lidar or radar sensor. In Figure 3B the example, the set of points 365 may represent a set of points generated in response to a first scan of an object moving relative to vehicle 100 by lidar / radar sensor 108A. The set of points 375 may represent a set of points generated in response to a second scan of the moving object by lidar / radar sensor 108A. Thus, based on the relative motion of the object with respect to vehicle 100, the programming of computer 104 and / or off-vehicle computer 115 may be executed, including instructions for unmodalizing the sets of points 365 and 375 to form a geometric container (e.g., cuboid 380) at the scan valid time as described with respect to expression (1).

[0063] However, in Figure 3BIn an example, the demodulation process can be modified to include a velocity correction input that can be calculated using velocities inferred or derived from two or more (e.g., successive) lidar or radar sensor scans. Thus, as Figure 3B shown, the point set 365 can be modified or adjusted to overlap with the point set 375. In one example, both the point sets 365 and 375 can undergo velocity correction, as indicated by arrows 370 and 380, to include a greater overlap with each other. Thus, in one example, the demodulation process performed by the vehicle computer 104 and / or the off-vehicle computer 115 can result in a relatively smaller geometric container being formed with respect to the detected moving object. In Figure 3B an example, for a moving object having a length, width, and height of one cubic meter, the demodulation process for generating the cuboid 390 may result in a cuboid having a volume smaller than ( Figure 3A that of) the cuboid 330, such as a cuboid enclosing a volume of, for example, 1.5 cubic meters, 1.3 cubic meters, 1.2 cubic meters, etc. Based on the cuboid 390 enclosing a relatively smaller volume, the vehicle computer 104 and / or the off-vehicle computer 115 can consume reduced processing resources to determine the pose of the vehicle represented by the point sets 315 and 325.

[0064] In one example, the velocity correction of the demodulated point set can be performed iteratively, where the velocity is calculated based on successive scans via the lidar / radar sensor 108A. Based on the calculated velocity, the demodulated point set can be modified, which can result in the generation of a cuboid that encloses the point sets (e.g., 365, 375) in a smaller cuboid at an optional and predetermined scan valid time. In response to the generated cuboids having progressively smaller dimensions, the iterative adjustment of the demodulated point set can continue. In response to the continuously generated cuboids converging on a specific boundary, a straight line defining the boundary of the cuboid, the iterative process can be stopped.

[0065] Figure 3A and Figure 3B can represent a training process performed by the offline machine learning application 415 (see Figure 4)。In response to an iterative process of refining the cuboid to enclose the detected point sets (e.g., 365, 375), the machine learning application can utilize the characteristics of various objects in motion encountered in the traffic environment to adjust the weights or settings. In one example, as further described with reference to the machine learning application 415, the iterative adjustment of the unmodalized point sets can operate to refine the cuboid to enclose the detected point sets (e.g., 365, 375). Such iterative adjustment can include processing hundreds or thousands of point sets to obtain a set of one or more parameters, which can be uploaded to the computer 104 of the vehicle 100 and / or uploaded to an off-vehicle computer 115. In one example, the machine learning application 415 can utilize supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, etc. to determine whether the machine learning application can converge on a cuboid with a straight or non-curved line defining the boundary of the cuboid.

[0066] Figure 4 FIG. 400 is a schematic diagram of an exemplary training environment for generating parameters for use in the vehicle computer 104 and / or for use by the off-vehicle computer 115. In Figure 4 the example, the labeled trajectory 405 can include an actual (e.g., ground truth) point set representing an object in motion obtained using the output data generated by the scans performed by the lidar / radar sensor 108A. In one example, the labeled trajectory 405 can include point sets representing a moving bicycle, a moving vehicle (e.g., car, sport utility vehicle, truck, bus, freight vehicle, etc.) that can move in front of the vehicle 100, to the side of the vehicle 100, to the rear of the vehicle 100, etc. The moving objects can include objects moving laterally with respect to the vehicle 100, objects moving towards the vehicle 100, and objects moving away from the vehicle 100. In Figure 4 the example, the objects can be labeled or classified according to class labels (such as "bicycle", "compact vehicle", "freight vehicle", "bus") or using another label or category.

[0067] The unmodalized component 410 represents computer-executable instructions for unmodalizing moving objects. Thus, the unmodalized component 410 represents a process for aggregating one or more historical point sets of at least a portion of the objects detected during one or more previous sensor scans. The historical point sets can include points obtained from lidar or radar sensor scans of the rear portion of the moving vehicle, the side portion of the moving vehicle, the front portion of the moving vehicle, and the points can be aggregated via instructions executed by the vehicle computer 104 and / or the off-vehicle computer 115.

[0068] A set of points and velocities derived or inferred using successive measurements from the lidar / radar sensor 108A can be formatted by the input layer to include parameters (i.e., data values such as weights or settings) and input into the depth sensor training component 430 of the machine learning application 415. In Figure 4 an example, the machine learning application 415 includes a neural network, such as a convolutional neural network. In this context, a convolutional neural network means a feedforward artificial neural network having at least three layers (i.e., an input layer, an output layer, and at least one hidden layer). In one example, the input layer operates to receive a set of points representing measurement points detected by the lidar / radar sensor 108A, such as the set of points 365 and 375 Figure 3B described. The set of points can represent relatively large moving objects (e.g., buses, trucks, etc.) as well as relatively small moving objects (e.g., bicycles, compact vehicles, etc.). The set of points can additionally represent objects moving at relatively low speeds (e.g., 5 km / h, 10 km / h, 15 km / h) relative to the vehicle 100, as well as objects moving at greater speeds (such as speeds of 25 km / h, 30 km / h, 40 km / h, etc.).

[0069] The output from the output layer of the machine learning application 415 can be used to compute a loss function that represents the ability of the machine learning application to accurately predict an expected output. The loss function can be backpropagated through the hidden layers of the machine learning application 415, thereby incrementally changing the weights or settings stored at the hidden layers of the machine learning application 415 to minimize the loss function. In this context, "weights" or "settings" mean parameters within the depth sensor training component 430 that at least partially control or manage the transformation or output of data from the machine learning application by performing operations such as addition, multiplication, convolution, or another function to provide data at the output layer that can be observed by the human and / or detection and tracking metric component 435.

[0070] In response to the loss function being sufficiently minimized, the machine learning application 415 can be considered trained, and the current parameters (e.g., formulated or derived from the weights and / or settings within the hidden layers of the machine learning application 415) can be uploaded for use by the vehicle computer 104 and / or the off-vehicle computer 115.

[0071] The trajectory dynamics inference component 420 operates to derive or infer the velocity of a point set representing a moving object. Thus, in one example, the trajectory dynamics inference component 420 computes an estimate of the velocity of the moving object. Thus, in one example, the first point set detected at the effective time T = 0 and the second point set detected at the effective time T = 0.5 seconds can be utilized to estimate the velocity of an object moving laterally with respect to the vehicle 100. Based on the point sets separated by 1.0 meter, the velocity of the object can be derived or inferred to be traveling laterally at a speed of 2.0 meters per second (7.2 kilometers per hour). Then, the output signal from the trajectory dynamics inference component 420 can be sent to the demodalization component 410, which can operate to modify the demodalized point set with the computed velocity.

[0072] As Figure 4 shown, the demodalization component 410, the machine learning application 415, and the trajectory dynamics inference component 420 operate in an iterative loop. Thus, in one example, the trajectory dynamics inference component 420 outputs a velocity correction for the demodalized point set computed by the demodalization component 410. Then, the velocity-corrected demodalized point set can be input to the machine learning application 415, which modifies the parameters (e.g., weights, settings, or parameters derived from the weights and / or settings) utilized by the depth sensor training component 430. Then, the machine learning application 415 can interact with the trajectory dynamics inference component 420 to output an update to the computed or derived velocity. Then, the updated computed or derived velocity can be input to the demodalization component 410, which utilizes the updated computed or derived velocity to output a modified demodalized point set. Then, the modified demodalized point set can be sent to the machine learning application 415 to further modify the parameters (e.g., settings or weights) of the depth sensor training component 430.

[0073] In Figure 4In the example, the trajectory dynamics inference component 420 may output a standardized text-based (e.g., JavaScript Object Notation) output file, and the standardized text-based output file may be input to the detection and tracking metric component 435. The detection and tracking metric component 435 may output a performance measurement of the alignment between the velocity-corrected unmodalized point set and the point set representing the output signal from the lidar sensor or from the radar sensor. In one example, the detection and tracking metric component 435 may operate to identify the divergence between the velocity-corrected unmodalized point set and the point set representing the output signal from the lidar / radar sensor 108A by using the output file from the trajectory dynamics inference component 420. In another example, the detection and tracking metric component 435 may operate to identify slow convergence (i.e., hundreds of iterations, thousands of iterations, or another number of iterations without resulting in the alignment between the velocity-corrected unmodalized point set and the point set representing the output signal from the lidar / radar sensor 108A). In one example, the divergence or slow convergence between the velocity-corrected unmodalized point set and the point set representing the output signal from the lidar / radar sensor 108A may indicate that the threshold level of training for the machine learning application 415 has not been performed and additional training may be performed. In one example, in response to the difference between the velocity-compensated position of the point set and the unmodalized representation of the point set during the iteration being less than a threshold (e.g., 2%, 1%, 0.5%, etc.), the unmodalized component 410 may detect the slow convergence between the velocity-corrected unmodalized point set and the point set representing the output signal from the lidar / radar sensor 108A.

[0074] In response to the training process of the machine learning application 415, one or more parameters may be uploaded to the computer 104 of the vehicle 100 and / or uploaded to the off-vehicle computer 115. Such parameters may enhance the ability of the vehicle computer 104 and / or the off-vehicle computer 115 to align the unmodalized point set representing the moving objects in the traffic environment with the velocity measurement calculated or derived from the continuous measurements from the lidar / radar sensor 108A. In one example, one or more parameters uploaded from the machine learning application 415 may be utilized by the attitude calculation component of the vehicle application, and the attitude calculation component controls the steering and / or propulsion of the vehicle in response to detecting, classifying, and / or marking the moving objects in the traffic environment.

[0075] Figure 5 is an exemplary timeline 500 showing the interpolation between a first point set and a second point set determined via the scan of a vehicle sensor such as the lidar / radar sensor 108A. In Figure 5 it, the horizontal axis represents the time at which the first lidar or radar scan can be modeled as occurring at a first effective time (scan 1) and a second effective time (scan 2). The interpolation time TQ It is shown that the points of the non-modalized third point set can be rendered using Scan 1 and Scan 2. Thus, in one example, the point set detected at the first valid time (Scan 1) can be interpolated with the point set detected at the second valid time (Scan 2) so as to coincide with the non-modalized third point set at T Q . In response to a misalignment between the interpolated point set at T Q and the non-modalized third point set, a velocity correction can be performed on the non-modalized point set, such as described in reference Figure 4 . In one example, additional iterative adjustments to the non-modalized point set result in the rendering of the non-modalized third point set at the interpolation time T Q . In one example, the coincidence of the non-modalized point set with the interpolated point set detected using the first and second valid times (Scan 1, Scan 2) causes instructions executed by the vehicle computer 104 and / or the off-vehicle computer 115 to generate a geometric container (e.g., a cuboid) having straight, non-arcuate, non-curved lines defining the boundaries of the generated cuboid.

[0076] Figure 6 is a flowchart of process 600 for generating velocity correction parameters in object pose determination. In one example, process 600 occurs during a training process, where the machine learning application 415 uses the computed velocity of a moving object obtained via successive scans of the moving object as an input to the non-modalization process. The velocity of the moving object can be calculated or inferred using two or more point sets representing the moving object obtained during different or successive lidar or radar sensor scans. In one example, instructions executed by the non-modalization component 410, the machine learning application 415, and the trajectory dynamics inference component 420 can be used to calculate a first non-modalized representation of the moving object to form a geometric container, such as a hexahedron, a cuboid, etc., which can include curved, arcuate, or thickened lines defining the boundaries of the geometric container. The first non-modalized representation of the moving object can represent the position of the moving object at a first optional interpolation or third time between the first and second lidar or radar sensor scans. After forming the first geometric container, the non-modalization process can be performed again using the computer-derived velocity of the moving object as an input signal, which operates to modify the non-modalized representation. The geometric container formed in response to the second execution of the non-modalization process can include a geometric container (e.g., a cuboid) having straight, non-curved, or non-arcuate lines defining the boundaries of the container. Based on the cuboid being less curved, less arcuate, or less thickened, the instructions executed by the vehicle computer can determine a second velocity of the moving object at a second optional interpolation time so as to align the modified non-modalized representation of the object with an interpolated or third point set derived or inferred from a scan using the lidar or radar sensor 108A.

[0077] Process 600 begins at block 605, which includes the unmodalized component 410 obtaining a point set in response to a scan by the lidar / radar sensor 108A, which may be mounted on an outward-facing portion of the vehicle body 102 of the vehicle 100. The point set may represent detected objects such as bicycles, vehicles (e.g., cars, buses, trucks, recreational vehicles, etc.).

[0078] Process 600 continues at block 610, which includes the unmodalized component 410 calculating the position of the moving object represented by the point set. The unmodalized component 410 may utilize an aggregation of one or more historical point sets of at least a portion of the detected object obtained during one or more previous sensor scans to calculate an unmodalized representation of the moving object. The portion of the detected object may include a side portion of the moving vehicle, a rear portion of the moving vehicle, etc.

[0079] Process 600 continues at block 615, which includes the machine learning application 415 executing instructions to calculate the speed of the moving object in response to consecutive lidar or radar scans of the moving object using the point set determined by the trajectory dynamics inference component 420. In one example, the speed of the moving object may be calculated by obtaining a first point set and a second point set, calculating the displacement between the first point set and the second point set, and dividing by the interval between the effective times of the first scan and the second scan.

[0080] Process 600 continues at block 620, which includes the machine learning application 415 generating a cuboid that encloses the velocity-corrected unmodalized points representing the detected object. In one example, the velocity correction of the unmodalized point set may include interpolating or rendering the unmodalized point set at an interpolation or third time (T Figure 5 as described in reference Q ) between the effective times of consecutive lidar or radar sensor scans.

[0081] Process 600 continues at block 625, where the machine learning application 415 determines whether the cuboid enclosing the velocity-corrected unmodalized point set includes a thickened, curved, or bowed line that defines the boundary of the cuboid rendered at the interpolation or third time (T Q ). In response to a cuboid boundary that includes a thickened, curved, or bowed line, process 600 continues at block 630, where the machine learning application 415 may modify the unmodalized point set representing the moving object using the derived or calculated object speed from block 615.

[0082] Process 600 returns to block 610, which includes calculating the (second) demodalized object position and further refining the velocity of the moving object at block 615.

[0083] Process 600 continues at block 620, which includes generating a second cuboid to determine whether the velocity-corrected demodalized point set includes thickened, curved, or bowed lines that define the boundaries of the cuboid. Blocks 625, 630, 610, 615, and 620 can be iteratively executed until the cuboid generated at block 620 is bounded by straight, non-curved, or non-bowed lines.

[0084] In an example of process 600, in response to machine learning application 415 determining that the cuboid enclosing the velocity-corrected demodalized point set is acceptable, e.g., includes straight, non-curved, and / or non-bowed lines that define the cuboid, block 635 can be executed. At block 635, machine learning application 415 can generate one or more parameters for in-vehicle use, such as reference Figure 7 as described.

[0085] After executing block 635, process 600 ends.

[0086] Figure 7 is a flowchart of process 700 for determining the pose of an object using velocity correction parameters in a vehicle. One or more parameters derived during the training process (blocks 605 - 635) can be uploaded and stored in a memory accessible to vehicle computer 104 and / or off-vehicle computer 115 to compute a velocity-corrected demodalized representation of a moving object detected in the traffic environment of vehicle 100. In one example, a process can be obtained to iterate to determine the time at which demodalization using the computed or inferred velocity of the moving object produces a geometric container (e.g., a cuboid) having substantially non-curved, non-bowed lines that define the geometric container. By modifying the demodalized representation of the moving object to include the velocity-corrected demodalized representation of the object, vehicle computer 104 and / or off-vehicle computer 115 can compute a cuboid having straight boundaries that define the cuboid. Thus, vehicle computer 104 and / or off-vehicle computer 115 can determine the pose of the moving object for input to vehicle-based assisted driving applications.

[0087] Process 700 begins at block 705, which includes machine learning application 415 sending one or more demodalization parameters to be stored in a memory accessible to vehicle computer 104 and / or off-vehicle computer 115. The demodalization parameters can include, for example, an interpolation between the valid times of lidar or radar sensor scans (e.g., scans 1 and 2 as described in reference Figure 5 or a third time (T QThe parameters of ( ) and / or other parameters (e.g., derived from weights or settings) of the machine learning application 415 for velocity correction of the non-modalized point set.

[0088] The process 700 continues at block 710, which includes generating a point set based on lidar or radar sensor scans using the lidar / radar sensor 108A mounted on the vehicle body 102.

[0089] The process 700 continues at block 715, which includes the vehicle computer 104 and / or the off-vehicle computer 115 interpolating between the first point set and the second point set at an interpolation or third time (T Q ). Block 715 may include the vehicle computer 104 and / or the off-vehicle computer 115 forming a third point set from the interpolated first and second point sets.

[0090] The process 700 continues at block 720, which includes the vehicle computer 104 and / or the off-vehicle computer 115 arranging the interpolated or third point set into a cuboid.

[0091] The process 700 continues at block 725, which includes determining the pose of the cuboid arranged at block 720. In one example, determining the pose of the cuboid may be utilized by an assisted driving application executed on the vehicle computer 104 and / or the off-vehicle computer 115.

[0092] The process 700 continues at block 730, which includes actuating control components of the vehicle 100 to control one or more of the propulsion system, steering system, etc. and / or the HMI 112. In one example, the vehicle computer 104 may control the actuator 110 to perform an advanced driver assistance system (ADAS). ADAS is an electronic technology that can assist the driver in achieving driving functions and parking functions. Examples of ADAS include lane departure detection, blind spot detection, adaptive cruise control, and lane keeping assistance. The vehicle computer 104 may actuate the systems of the vehicle 100 according to an algorithm that operates without human input to stop the vehicle before reaching a moving object represented by the point set detected by the lidar / radar sensor 108A.

[0093] After executing block 730, the process 700 ends.

[0094] Generally, the described computing systems and / or devices may employ any of a number of computer operating systems, including but not limited to the following versions and / or variants: Ford Applications, AppLink / Smart Device Link middleware, Microsoft Operating System, Microsoft Operating systems, Unix operating systems (e.g., the operating system released by Oracle Corporation, Redwood Shores, California), AIX UNIX operating system released by International Business Machines Corporation, Armonk, New York, Linux operating system, Mac OS X and iOS operating systems released by Apple Inc., Cupertino, California, BlackBerry OS released by BlackBerry Limited, Waterloo, Canada, and Android operating system developed by Google Inc. and the Open Handset Alliance, or CAR infotainment platform supplied by QNX Software Systems. Examples of computing devices include, but are not limited to, in-vehicle computers, computer workstations, servers, desktops, notebooks, laptop computers or handheld computers, or some other computing system and / or device. Operating systems), the AIX UNIX operating system released by International Business Machines Corporation, Armonk, New York, Linux operating system, Mac OS X and iOS operating systems released by Apple Inc., Cupertino, California, BlackBerry OS released by BlackBerry Limited, Waterloo, Canada, and Android operating system developed by Google Inc. and the Open Handset Alliance, or the CAR infotainment platform supplied by QNX Software Systems. Examples of computing devices include, but are not limited to, in-vehicle computers, computer workstations, servers, desktops, notebooks, laptop computers or handheld computers, or some other computing system and / or device.

[0095] Computing devices typically include computer-executable instructions, where the instructions can be executed by one or more computing devices such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages and / or technologies, which alone or in combination include, but are not limited to, Java TM , C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as Java virtual machines, Dalvik virtual machines, etc. Generally, a processor (e.g., a microprocessor) receives instructions from, for example, a memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium (such as a storage medium, random access memory, etc.).

[0096] A computer-readable medium (also referred to as a processor-readable medium) includes any non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including but not limited to non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communication, including internal components that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0097] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing, and retrieving various data, including hierarchical databases, sets of files in a file system, application databases in a proprietary format, relational database management systems (RDBMS), non-relational databases (NoSQL), graph databases (GDB), etc. Each such data store is typically included within a computing device that employs a computer operating system such as one of those mentioned above, and can be accessed via a network in any one or more of a variety of ways. A file system can be accessed from a computer operating system and can include files stored in various formats. In addition to languages for creating, storing, editing, and executing stored programs (such as the PL / SQL language mentioned above), an RDBMS typically also employs the Structured Query Language (SQL).

[0098] In some examples, system components can be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) and stored on a computer-readable medium associated therewith (e.g., disks, memories, etc.). A computer program product can include such instructions stored on a computer-readable medium for performing the functions described herein.

[0099] In the drawings, the same reference numerals indicate the same elements. Additionally, some or all of these elements can be changed. With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes can be practiced by performing the steps in an order different from that described herein. It should also be understood that certain steps can be performed simultaneously, other steps can be added, or certain steps described herein can be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with the applicable owner / user manuals and / or safety guidelines.

[0100] The present disclosure has been described in an illustrative manner, and it is to be understood that the terms used are intended to be of the nature of descriptive words and not of a limiting nature. The adjectives "first" and "second" are used throughout this document as identifiers and are not intended to denote importance, order, or quantity. The use of "responsive to" and "after determining..." indicates a causal relationship and not merely a temporal relationship. Given the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure may be practiced in other ways than as specifically described.

[0101] According to the present invention, there is provided a system having: a computer having a processor and a memory, the memory including instructions executable by the processor to perform the following operations: generating a first point set and a second point set based on a first scan and a second scan obtained from a lidar sensor or from a radar sensor; determining a first speed-compensated position of an object represented by a third point set at a first valid time between respective times of the first scan and the second scan; receiving a parameter from the memory of the computer, the parameter being determined according to a training process for modifying an amodal representation of the object, the modified amodal representation being determined based on a difference between a second speed-compensated position of the object and an unmodified amodal representation of the object; and determining an attitude of the object represented by the third point set based on the parameter.

[0102] According to an embodiment, the parameter is generated based on an iterative adjustment of the amodal representation of the object, the iterative adjustment of the amodal representation being based on a difference between the second speed-compensated position of the object and the modified amodal representation of the object being greater than a threshold.

[0103] According to an embodiment, the parameter is determined based on the iterative adjustment of the amodal representation of the object that terminates in response to the difference between the second speed-compensated position of the object and the modified amodal representation of the object being less than the threshold.

[0104] According to an embodiment, the iterative adjustment of the amodal representation of the object occurs via supervised machine learning.

[0105] According to an embodiment, the amodal representation of the object is determined based on an aggregated history of scans of the object.

[0106] According to an embodiment, the instructions further include instructions for performing the following operations: generating a geometric container including the third point set; and assigning a class label to the geometric container.

[0107] According to an embodiment, the class label assigned to the geometric container is a cuboid surrounding a vehicle.

[0108] According to an embodiment, the instructions further include instructions for performing the following operation: actuating a vehicle component based on the determined pose of the object.

[0109] According to an embodiment, the vehicle component is a steering component or a propulsion component.

[0110] According to an embodiment, the parameter represents the first valid time when calculating the modified amodal representation of the object.

[0111] According to an embodiment, the first valid time is determined based on an interpolation between the unmodified amodal representation of the object and the modified amodal representation of the object.

[0112] According to the present invention, a method includes: generating a first point set and a second point set based on a first scan and a second scan obtained from a lidar sensor or a radar sensor; determining a first speed-compensated position of an object represented by a third point set at a first valid time between corresponding times of the first scan and the second scan; receiving a parameter from a computer memory, the parameter being determined according to a training process for modifying an amodal representation of the object, the modified amodal representation being determined based on a difference between a second speed-compensated position of the object and the unmodified amodal representation of the object; and determining a pose of the object represented by the third point set based on the parameter.

[0113] In one aspect of the present invention, the parameter is determined according to an iterative adjustment of the amodal representation of the object, and the iterative adjustment of the amodal representation is based on a difference between the second speed-compensated position of the object and the modified amodal representation of the object being greater than a threshold.

[0114] In one aspect of the present invention, the parameter is determined according to an iterative adjustment of the amodal representation of the object that terminates in response to the difference between the second speed-compensated position of the object and the modified amodal representation of the object being less than the threshold.

[0115] In one aspect of the present invention, the iterative adjustment of the amodal representation of the object occurs via a supervised machine learning environment.

[0116] In one aspect of the present invention, an amodal representation of an object is determined based on an aggregation history of scans of the object.

[0117] In one aspect of the present invention, the method includes: generating a geometric container including the third point set; and assigning a class label to the geometric container.

[0118] In one aspect of the present invention, the method includes: actuating a vehicle component based on the determined pose of the object.

[0119] In one aspect of the present invention, the vehicle component is a steering component or a propulsion component.

[0120] In one aspect of the present invention, the parameter represents the first valid time when calculating the modified non-modal representation of the object.

Claims

1. A method, comprising: Generating a first point set and a second point set based on a first scan and a second scan obtained from a lidar sensor or from a radar sensor; Determining a first velocity compensation position of an object represented by a third point set at a first valid time between corresponding times of the first scan and the second scan; Receiving a parameter from a memory of a computer, the parameter being determined according to a training process for modifying an amodal representation of the object, the modified amodal representation being determined according to a difference between a second velocity compensation position of the object and an unmodified amodal representation of the object; and Determining an attitude of the object represented by the third point set based on the parameter.

2. The method according to claim 1, further comprising: Generating the parameter via iterative adjustment of the amodal representation of the object, the iterative adjustment of the amodal representation being based on a difference between the second velocity compensation position of the object and the modified amodal representation of the object being greater than a threshold.

3. The method according to claim 2, further comprising: Terminating the iterative adjustment of the amodal representation in response to the difference between the second velocity compensation position of the object and the modified amodal representation of the object being less than the threshold.

4. The method according to claim 2, wherein the iterative adjustment of the amodal representation of the object occurs via supervised machine learning.

5. The method according to claim 1, further comprising: Determining the amodal representation of the object according to an aggregation history of scans of the object.

6. The method according to claim 1, further comprising: Generating a geometric container including the third point set; And Assigning a class label to the geometric container.

7. The method according to claim 6, wherein the class label assigned to the geometric container is a cuboid surrounding a vehicle.

8. The method according to claim 1, further comprising: Actuating a vehicle component based on determining the attitude of the object.

9. The method according to claim 8, wherein the vehicle component is a steering component.

10. The method according to claim 8, wherein the vehicle component is a propulsion component.

11. The method according to claim 1, wherein the parameter represents the first valid time when calculating the modified amodal representation of the object.

12. The method according to claim 10, further comprising: Determining the first valid time according to an interpolation between the unmodified amodal representation of the object and the modified amodal representation of the object.

13. The method according to claim 1, wherein the sensor is a lidar sensor.

14. A computer programmed to execute the method according to any one of claims 1 to 13.

15. A vehicle comprising the computer according to claim 14.

Images