Vehicle attitude

Abstract

The present disclosure provides a "vehicle pose". A computer includes a processor and a memory, and the memory stores instructions executable by the processor to determine an amount of relative motion between a host vehicle and a target vehicle; in response to the amount of relative motion being below a threshold, determining a pose of the host vehicle subject to constraints on motion of the host vehicle; and in response to the amount of relative motion being above the threshold, determining the attitude of the host vehicle without the constraint.

Description

Technical Field

[0001] The present disclosure provides techniques for determining the pose of a vehicle. Background Art

[0002] Vehicles often include cameras. Cameras can detect electromagnetic radiation within a certain wavelength range. Possible frequencies that a camera can detect include visible light, infrared radiation, ultraviolet light, or a certain wavelength range that includes visible light, infrared light, and / or ultraviolet light. Types of cameras include charge coupled devices (CCDs) and complementary metal oxide semiconductors (CMOSs). Summary of the invention

[0003] The vehicle will be referred to as the "host vehicle" to distinguish it from other vehicles in the environment, which will be referred to as "target vehicles". The host vehicle's computer may be programmed to determine the amount of relative motion between the host vehicle and the target vehicle; in response to the amount of relative motion being below a threshold, determine the attitude of the host vehicle subject to a constraint on the motion of the host vehicle; and in response to the amount of relative motion being above the threshold, determine the attitude of the host vehicle without the constraint. For example, if the target vehicle remains 100 feet in front of the host vehicle and 10 feet to the left for a period of time, the computer determines the attitude subject to the constraint, such as determining an attitude within or weighted toward a narrower range of possible attitudes. If the target vehicle increases its distance (e.g., lateral distance) from the host vehicle, the computer determines the attitude without the constraint. In some cases, the use of constraints can improve the accuracy of the determined attitude, and the amount of relative motion can be used to identify some of these situations. The computer can then actuate the host vehicle based on the attitude (e.g., by executing an advanced driver assistance system (ADAS) that takes the attitude of the host vehicle as input).

[0004] A computer includes a processor and a memory, and the memory stores instructions that can be executed by the processor to determine the relative motion between a main vehicle and a target vehicle; in response to the relative motion being lower than a threshold, determine the posture of the main vehicle subject to a constraint on the motion of the main vehicle; and in response to the relative motion being higher than the threshold, determine the posture of the main vehicle in the absence of the constraint.

[0005] In an example, the instructions may further include instructions for actuating a component of the host vehicle based on the posture of the host vehicle.

[0006] In an example, the constraint may constrain the pose of the host vehicle toward a predicted pose, and the predicted pose may be the current pose translated a certain distance in the current direction of travel of the host vehicle. In another example, the distance may be the product of the time step and the current speed of the host vehicle.

[0007] In an example, the instructions may further include instructions for determining the amount of relative motion based on image data depicting the target vehicle. In another example, the image data may include a first image frame depicting the target vehicle and a second image frame depicting the target vehicle, and the amount of relative motion may include a measure of change between the target vehicle as depicted in the first image frame and the target vehicle as depicted in the second image frame. In another example, the instructions for determining the amount of relative motion may include instructions for comparing a first set of pixels forming the target vehicle in the first image frame with a second set of pixels forming the target vehicle in the second image frame. In another example, the measure of change may be based on an average pixel shift of the first set of pixels and the second set of pixels, and the instructions for comparing the first set of pixels with the second set of pixels may include instructions for determining the average pixel shift.

[0008] In another example, the instructions may further include instructions for extracting a first feature set from the first set of pixels and extracting a second feature set from the second set of pixels, and the instructions for comparing the first set of pixels with the second set of pixels may include instructions for comparing the first feature set with the second feature set. In a continuing example, the instructions may further include instructions for determining a set of feature correspondences between the first set of pixels and the second set of pixels; and determining a pixel-by-pixel shift between the feature correspondences. In another continuing example, the measure of change may be an average pixel shift based on the first set of pixels and the second set of pixels, and the instructions for comparing the first set of pixels with the second set of pixels may include instructions for determining the average pixel shift from the pixel-by-pixel shift.

[0009] In another example, the instructions may also include instructions for performing semantic segmentation to classify pixels in the first image frame and the second image frame into a first category, the first category indicating a vehicle; and determining the first set of pixels and the second set of pixels based on the pixels classified into the first category.

[0010] In another example, the instructions may also include instructions for performing the following operations: performing object recognition on the first image frame and the second image frame to identify a vehicle; applying a bounding box around the corresponding identified vehicle; and determining the first set of pixels and the second set of pixels based on the bounding box. In a continuing example, the instructions may also include instructions for performing the following operations: performing semantic segmentation to classify pixels in the first image frame and the second image frame into a first category, the first category indicating a vehicle, and the first set of pixels and the second set of pixels may come from pixels that are both within the bounding box and classified as the first category.

[0011] In another example, the instructions may further include instructions for identifying the target vehicle in the second image frame as the same vehicle as the target vehicle in the first image frame.

[0012] In another example, the image data may originate from a camera mounted to the host vehicle.

[0013] In an example, the instructions for determining the pose of the host vehicle may include instructions for determining the pose of the host vehicle based on a measurement variance of a sensor of the host vehicle, and the constraint is that a value of the measurement variance increases.

[0014] In an example, the instructions for determining the posture of the host vehicle may include instructions for executing a Kalman filter, and the state variable of the Kalman filter may include the posture of the host vehicle. In another example, the instructions may also include instructions for calculating a gain of the Kalman filter based on a measurement variance of a sensor of the host vehicle, and the constraint may be an increase in the value of the measurement variance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example host vehicle.

[0016] Figure 2 is a top-down solution view of the host vehicle in the environment.

[0017] Figure 3 is a depiction of an image frame captured by a vehicle's camera.

[0018] Figure 4 is a flow chart of an example process for determining the posture of the host vehicle. DETAILED DESCRIPTION

[0019] Referring to the accompanying drawings, in which like reference numerals indicate like parts throughout the several views, a computer 105 includes a processor and a memory, and the memory stores instructions executable by the processor to determine an amount of relative motion between a subject vehicle 100 and a target vehicle 200; in response to the amount of relative motion being below a threshold, determine a posture of the subject vehicle 100 subject to a constraint on the motion of the subject vehicle 100; and in response to the amount of relative motion being above the threshold, determine the posture of the subject vehicle 100 in the absence of the constraint.

[0020] refer to Figure 1 , the host vehicle 100 may be any passenger vehicle or commercial vehicle, such as a car, truck, sport utility vehicle, crossover, van, minivan, taxi, bus, etc. The host vehicle 100 may include a computer 105, a communication network 110, an environmental sensor 115, a motion sensor 120, a propulsion system 125, a braking system 130, a steering system 135, and a user interface 140.

[0021] The computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, an electronic controller, etc.), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a combination of the foregoing, etc. Typically, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design automation to describe digital and mixed signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, and the logic components inside the FPGA can be configured based on, for example, VHDL programming stored in a memory electrically connected to the FPGA circuit. Therefore, the computer 105 may include a processor, a memory, etc. The memory of the computer 105 may include a medium for storing instructions executable by the processor and for electronically storing data and / or a database, and / or the computer 105 may include structures such as the aforementioned structures that provide programming. The computer 105 may be a plurality of computers connected together.

[0022] The computer 105 can transmit and receive data via the communication network 110. The communication network 110 may be, for example, a controller area network (CAN) bus, Ethernet, WiFi, a local interconnect network (LIN), an on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. The computer 105 can be communicatively coupled to environmental sensors 115, motion sensors 120, propulsion system 125, braking system 130, steering system 135, user interface 140, and other components via the communication network 110.

[0023] Environmental sensors 115 may detect the outside world, e.g., objects and / or features of the vehicle's surroundings, such as other vehicles, road lane markings, traffic lights and / or signs, pedestrians, etc. For example, environmental sensors 115 may include radar sensors, ultrasonic sensors, scanning laser rangefinders, light detection and ranging (lidar) devices, and image processing sensors such as cameras 145.

[0024] The camera 145 may detect electromagnetic radiation within a range of wavelengths. For example, the camera 145 may detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths that include visible light, infrared light, and / or ultraviolet light. For example, the camera 145 may be a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or any other suitable type. The camera 145 is mounted to the host vehicle 100 and may be fixed relative to the host vehicle 100, such as fixedly mounted to the body of the host vehicle 100. The camera 145 may be oriented generally horizontally.

[0025] The motion sensor 120 may detect the position and / or orientation of the host vehicle 100. For example, the motion sensor 120 may include a global navigation satellite system (GNSS) sensor (such as a global positioning system (GPS) sensor); an accelerometer, such as a piezoelectric system or a micro-electromechanical system (MEMS); a gyroscope, such as a rate gyroscope, a ring laser gyroscope, or a fiber optic gyroscope; an inertial measurement unit (IMU); and / or a magnetometer. The GPS sensor receives data from GPS satellites. The Global Positioning System (GPS) is a global navigation satellite system. Satellites broadcast time and geographic location data. The GPS sensor may determine the position of the host vehicle 100, i.e., latitude and longitude, based on receiving time and geographic location data from multiple satellites simultaneously.

[0026] The propulsion system 125 of the host vehicle 100 generates energy and converts the energy into motion of the host vehicle 100. The propulsion system 125 can be a conventional vehicle propulsion subsystem, such as a conventional powertrain, including an internal combustion engine coupled to a transmission that transfers rotational motion to the wheels; an electric powertrain, including a battery, an electric motor, and a transmission that transfers rotational motion to the wheels; a hybrid powertrain, including elements of a conventional powertrain and an electric powertrain; or any other type of propulsion device. The propulsion system 125 can include an electronic control unit (ECU) that communicates with and receives input from the computer 105 and / or a human operator, etc. The human operator can control the propulsion system 125 via, for example, an accelerator pedal and / or a gear shift lever.

[0027] The braking system 130 is typically a conventional vehicle braking subsystem and resists the motion of the host vehicle 100, thereby slowing and / or stopping the host vehicle 100. The braking system 130 may include friction brakes, such as disc brakes, drum brakes, band brakes, etc.; regenerative brakes; any other suitable type of brake; or a combination thereof. The braking system 130 may include an electronic control unit (ECU) or the like that communicates with and receives input from the computer 105 and / or a human operator. The human operator may control the braking system 130 via, for example, a brake pedal.

[0028] The steering system 135 is typically a conventional vehicle steering subsystem and controls the turning of the wheels. The steering system 135 can be a rack and pinion system with electric power steering, a steer-by-wire system (both of which are known), or any other suitable system. The steering system 135 may include an electronic control unit (ECU) or the like that communicates with and receives input from the computer 105 and / or a human operator. The human operator can control the steering system 135 via, for example, a steering wheel.

[0029] The user interface 140 presents information to and receives information from an operator of the host vehicle 100. The user interface 140 may be located, for example, on a dashboard in a passenger compartment of the host vehicle 100, or anywhere else that the operator can easily see it. The user interface 140 may include dials, digital readouts, screens, speakers, etc., such as known human-machine interface (HMI) elements, for providing information to the operator. The user interface 140 may include buttons, knobs, keypads, microphones, etc., for receiving information from the operator.

[0030] refer to Figure 2 , the host vehicle 100 may move through the environment 205 together with one or more target vehicles 200. For example, at least one of the host vehicle 100 and the target vehicle 200 may be traveling in the same direction on the road 210, such as Figure 2 as shown in .

[0031] The movement of the host vehicle 100 and each target vehicle 200 between two points in time defines the amount of relative motion between the host vehicle 100 and that target vehicle 200. The relative motion is the amount of motion of the target vehicle 200 relative to the host vehicle 100, i.e., the amount of motion of the target vehicle 200 in a reference frame fixed to the host vehicle 100, e.g., changing from 100 feet in front of and 10 feet to the left of the host vehicle 100 at a first time to 50 feet in front of and 10 feet to the left of the host vehicle 100 at a second time. Equivalently, the relative motion is the difference between the motion of the target vehicle 200 relative to the environment 205 and the motion (in vector form) of the host vehicle 100 relative to the environment 205 from the first time to the second time.

[0032] The computer 105 is programmed to determine the amount of relative motion between the host vehicle 100 and each target vehicle 200. For example, the computer 105 may determine the amount of relative motion based on image data depicting the target vehicles 200, as described below with respect to Figure 3 Detailed description. For another example, the computer 105 may determine the relative amount of motion based on the distance data (e.g., point cloud) from the distance sensor of the environmental sensor 115 at different times. The distance sensor may be a laser radar, a radar, etc. The distance data may include the relative position of the target vehicle 200, and the relative amount of motion may be the change in relative position between the first time and the second time. For another example, the computer 105 may determine the relative amount of motion based on: the change in position of the host vehicle 100 between the first time and the second time determined according to the data from the motion sensor 120, and the change in position of the target vehicle 200 between the first time and the second time received from the target vehicle 200, for example, via vehicle-to-vehicle (V2V) communication. Determining the relative amount of motion based on image data may be advantageous because the host vehicle 100 is more likely to include a camera 145 than a laser radar, and because the determination does not rely on receiving information from the target vehicle 200.

[0033] refer to Figure 3 , the computer 105 receives image data originating from the camera 145, for example, via the communication network 110. The image data is a series of image frames 300 of the field of view of the camera 145. Each image frame 300 is a two-dimensional matrix of pixels. The brightness or color of each pixel is represented as one or more numerical values, for example, a scalar unitless value of photometric light intensity between 0 (black) and 1 (white), or a value of each of red, green, and blue, for example, each in an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale. The pixels can be a mixture of representations, such as a repeating pattern of scalar values ​​of the intensity of three pixels and a fourth pixel having three numerical color values, or some other pattern. The position in the image frame 300 (i.e., the position in the field of view of the sensor when the image frame 300 was recorded) can be specified in pixel size or coordinates, for example, a pair of ordered pixel distances, such as a number of pixels from the top edge of the image frame 300 and a number of pixels from the left edge of the image frame.

[0034] The computer 105 may be programmed to determine an amount of relative motion between the host vehicle 100 and each target vehicle 200 based on the image data depicting the target vehicles 200. The amount of relative motion is between a first time and a second time, and the image data includes a first image frame 300 depicting the target vehicle 200 at the first time and a second image frame 300 depicting the target vehicle 200 at the second time. The amount of relative motion includes a measure of change between the target vehicle 200 as depicted in the first image frame 300 and the target vehicle 200 as depicted in the second image frame 300. For purposes of this disclosure, a "measure of change" is defined as a numerical quantity that describes a comparison of two depictions in the image data. As an overview, for each target vehicle 200, the computer 105 may determine a first set of pixels that form the target vehicle 200 in the first image frame 300 and a second set of pixels that form the target vehicle 200 in the second image frame 300, and compare the first set of pixels to the second set of pixels to determine the measure of change. As a more detailed overview of the steps to be described below, the computer 105 may: perform semantic segmentation to classify pixels in the image frame 300 as belonging to a vehicle; perform object recognition on the image frame 300 to identify the target vehicle 200; apply a bounding box 305 around the identified vehicle; determine a first set of pixels for each target vehicle 200 as pixels of the first image frame 300 that are both within the corresponding bounding box 305 and classified as belonging to a vehicle, determine the pixels of each target vehicle 200 as pixels of the second image frame 300 that are both within the corresponding bounding box 305 and classified as belonging to a vehicle; extract a first set of features from the first set of pixels; extract a second set of features from the second set of pixels; identify each target vehicle 200 in the second image frame 300 as the same vehicle as the corresponding target vehicle 200 in the first image frame 300; determine a set of feature correspondences between the corresponding first set of pixels and the corresponding second set of pixels for each target vehicle 200; determine a pixel-by-pixel shift between the feature correspondences; and determine an average pixel shift for each target vehicle 200 from the pixel-by-pixel shift. The result of these steps is a determination of how close each target vehicle 200 is to being stationary within the image plane of the camera 145 over time (specifically, from the first image frame 300 to the second image frame 300), which indicates the extent of the relative motion.

[0035] The computer 105 may be programmed to perform semantic segmentation, thereby classifying pixels in the first image frame 300 and pixels in the second image frame 300 into categories. For example, the computer 105 may execute a semantic segmentation algorithm. The semantic segmentation algorithm labels each pixel in the image with a category. The categories include at least a first category and a second category, and may include more than two categories. At least one category (e.g., a first category) indicates a vehicle, that is, the pixel is classified as belonging to a vehicle. The category indicating the vehicle may include a single category such as {vehicle} or multiple categories such as {consumer vehicle, commercial vehicle, semi-trailer truck}. The categories may be stored in the computer 105 as a list, such as {road, sidewalk, ground, tree, consumer vehicle, commercial vehicle, semi-trailer truck, bicycle, pedestrian, animal, building / wall, traffic sign, traffic light, sky}. The semantic segmentation algorithm may be a machine learning algorithm, such as a deep neural network, a convolutional neural network, etc. One such machine learning algorithm for performing semantic segmentation is Panoptic-Deeplab-v3. Another is Detectron2, and there are other examples. As another example, the computer 105 may execute a panoptic segmentation algorithm, which is a combination of semantic segmentation and instance segmentation.

[0036] The computer 105 may be programmed to perform object recognition on the first image and the second image to identify the target vehicle 200. This step may be independent of performing semantic segmentation. The computer 105 may use conventional image recognition techniques (e.g., a machine learning program such as a convolutional neural network or an encoder-decoder neural network, such as YOLOP) to identify the target vehicle 200. The machine learning program may accept the image frame 300 as input and output an identification of the object in the image frame 300.

[0037] The computer 105 may be programmed to apply a bounding box 305 around the corresponding identified target vehicle 200 in the first image frame 300 and the second image frame 300. For example, the computer 105 may execute a machine learning program. The machine learning program may be the same program used to identify the vehicle, and may output the location of the bounding box 305 paired with the corresponding identification (e.g., YOLOP) of the object such as the vehicle.

[0038] The computer 105 may be programmed to determine, for each target vehicle 200, a first set of pixels in the first image frame 300 that belong to the target vehicle 200 and a second set of pixels in the second image frame 300 that belong to the target vehicle 200. The computer 105 may determine the first set of pixels and the second set of pixels based on pixels classified as the first category (i.e., classified as vehicles) by semantic segmentation and / or based on the bounding box 305. For example, the first set of pixels and the second set of pixels may come from pixels that are both within the bounding box 305 and classified as the first category. For example, the computer 105 may perform a mask multiplication of a segmentation mask of the first image frame 300 and a bounding box mask of the first image frame 300 (and perform the same operation on the second image frame 300). The segmentation mask may code each pixel in the first category as "1" and code the remaining pixels as "0". The bounding box mask may code each pixel in the corresponding bounding box 305 as "1" and code each pixel outside the corresponding bounding box 305 as "0". Mask multiplication is a pixel-level multiplication, so if a pixel has a value of "1" in both the segmentation mask and the bounding box mask, the value of the corresponding pixel is "1", otherwise the value of the pixel is "0". The computer 105 may also set the value of the pixel to "0" if the pixel is part of a continuous area below a size threshold (e.g., 0.04% of the area of ​​the image frame 300). The first set of pixels are pixels having a value of "1" due to the mask multiplication of the first image frame 300, and the second set of pixels are pixels having a value of "1" due to the mask multiplication of the second image frame 300. The mask multiplication and the size threshold each help reduce noise in the first set of pixels and the second set of pixels.

[0039] The computer 105 may be programmed to extract a first set of features from the first set of pixels, and a second set of features from the second set of pixels. For the purposes of this disclosure, the term "feature" is used in its computer vision sense as a piece of information about the content of an image, specifically whether a certain area of ​​an image has certain properties. The types of features may include edges, corners, spots, etc. Extracting features includes executing a feature extractor. The types of features may be selected to be generally visible and trackable from a rear view of a typical target vehicle 200. The feature extractor may include one or more suitable techniques for feature extraction, such as low-level techniques such as edge detection, corner detection, spot detection, ridge detection, scale-invariant feature transform (SIFT), etc.; shape-based techniques such as thresholding, spot extraction, template matching, Hough transform, generalized Hough transform, etc.; flexible methods such as deformable parameterized shapes, active contours, etc. The feature extractor may include machine learning operations. For example, the first feature extractor may include a residual network (ResNet) layer followed by a convolutional neural network. Alternatively, the feature extractor may use feature extraction techniques that do not rely on machine learning, such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Oriented FAST, and Rotated BRIEF (ORB) features, etc., thereby providing simpler computations while still effectively tracking relevant features for the application.

[0040] Computer 105 may be programmed to identify each target vehicle 200 in second image frame 300 as the same vehicle as target vehicle 200 in first image frame 300. For example, computer 105 may determine the closest match between the second feature set of a given target vehicle 200 and one of the plurality of first feature sets extracted for the plurality of target vehicles 200, for example using mutual nearest neighbors. Computer 105 may ignore target vehicles 200 in second image frame 300 that do not have a match in first image frame 300 in the remaining steps for determining the amount of relative motion.

[0041] The computer 105 may be programmed to compare the first feature set of each target vehicle 200 with the second feature set, for example to determine a set of feature correspondences between the first pixel set and the second pixel set of each target vehicle 200. Each feature correspondence is a pairing of one of the features in the first feature set with one of the features in the second feature set, the pairings may have the same or different pixel coordinates. The feature correspondences may be outputs of mutual nearest neighbors.

[0042] The computer 105 may be programmed to determine a measure of change between the target vehicle 200 as depicted in the first image frame 300 and the target vehicle 200 as depicted in the second image frame 300, such as based on an average pixel shift of the first set of pixels and the second set of pixels. For example, the computer 105 may determine a pixel-by-pixel shift between feature correspondences. The pixel-by-pixel shift of each feature correspondence may be a change in pixel coordinates (e.g., pixel distances, such as Euclidean pixel distances) of paired features. The computer 105 may determine an average pixel shift based on the pixel-by-pixel shifts. The average pixel shift may be an average of the pixel distances, i.e., the sum of the pixel distances divided by the number of feature correspondences.

[0043] The computer 105 is programmed to determine whether the amount of relative motion (e.g., a measure of change (e.g., average pixel shift)) is above or below a threshold. The threshold is selected to indicate whether the target vehicle 200 is substantially stationary relative to the host vehicle 100. In response to the amount of relative motion being below the threshold, the computer 105 determines the attitude of the host vehicle 100 subject to constraints on the motion of the host vehicle 100, as will be described below. In response to the amount of relative motion being above the threshold, the computer 105 determines the attitude of the host vehicle 100 without the constraints.

[0044] The constraint constrains the determined posture of the host vehicle 100 toward the predicted posture; that is, the constraint operates to change the value of the determined posture toward the predicted posture. For example, the computer 105 may determine a posture that is within a narrow range of possible postures or weighted toward a narrow range of possible postures. The example described in detail below weights the determined posture toward a narrow range of possible postures. The predicted posture can be the current posture translated a certain distance in the current direction of travel of the host vehicle 100. The distance can be the product of the time step of the host vehicle 100 and the current speed, for example, as shown in the following equation:

[0045]

[0046] where k is the index of the time step, is the predicted pose at time step k+1, M k is the measured pose at time step k, δ is the length of the time step, and V k is the velocity vector at time step k.

[0047] The constraint can be implemented by measuring the variance. The measurement variance is an indicator of the posture M of the host vehicle 100. k The measurement results of the posture are derived from sensor data of sensors of the host vehicle 100, as will be described below.

[0048] Determining a posture subject to a constraint may include changing a value of a measurement variance. For example, a constraint may be an increase in the value of a measurement variance. An increase in the measurement variance may increase the weight assigned to the predicted posture compared to the weight assigned to the measured posture, thereby causing the determined posture to be more inclined toward the predicted posture rather than the measured posture. For example, the value of the measured variance may be changed according to a radial basis function kernel (e.g., by changing the value of a parameter of the radial basis function kernel). The radial basis function kernel is given by the exponent of the negative product of the parameter and the square norm of the difference between the measured posture and the predicted posture of the host vehicle 100, for example, for each translation axis, for example, as shown in the following equation:

[0049]

[0050]

[0051]

[0052] where the subscripts x, y, and z indicate the translation axes; K is the radial basis function kernel along one axis; exp() is the exponential function, i.e., the argument raised to the power of the Euler number e; and σ is a parameter, also called the bandwidth. The radial basis function kernel can be reformulated to equivalently use the parameter γ = 1 / 2σ 2 When the constraint is used, that is, when the relative motion is below a threshold, the two horizontal axes σ x and σ y The parameter of can be reduced by a preset factor α. The measurement variance can be based on a radial basis function kernel, for example, as follows:

[0053]

[0054] where v m is the measurement variance, and v m,def is the default value for measuring variance. The radial basis function kernel is more useful for large deviation penalties than for small deviation penalties.

[0055] The computer 105 is programmed to determine the attitude of the host vehicle 100, for example, based on data from the environmental sensors 115 and the motion sensors 120 and a motion model of the host vehicle 100, as will be described below. If the relative amount of motion is below a threshold, the computer 105 determines that the attitude is subject to a constraint, and if the relative amount of motion is above a constraint, the computer determines that the attitude is not constrained. For example, the computer 105 may determine the attitude of the host vehicle 100 based on a measurement variance, and the measurement variance may change value depending on whether the relative amount of motion is above or below a threshold, as described above.

[0056] As a specific example, the computer 105 may determine the attitude of the host vehicle 100 by executing a Kalman filter. The Kalman filter operates by forming a feedback loop between a prediction step (i.e., using a prediction equation to predict the vehicle attitude and error estimate for the next time step) and a measurement step (i.e., using a measurement equation to adjust the prediction made using measurements from the sensors). The Kalman filter may be an extended Kalman filter.

[0057] The Kalman filter tracks the movement of the host vehicle 100 in space using state variables. The state variables of the Kalman filter include the attitude of the host vehicle 100, the velocity of the host vehicle 100, and the rotation of the host vehicle 100. The attitude can be a six-degree-of-freedom attitude, for example, including three spatial dimensions and three angular dimensions. The velocity can be a three-dimensional vector. The rotation can be equivalently represented by a rotation vector (i.e., roll, pitch, and yaw), a rotation matrix, or a quaternion.

[0058] The prediction step predicts the value of the state variable based on the value of the state variable at the previous time step and the motion data from the motion sensor 120. The prediction step can be based on a physics-based model of the motion of the host vehicle 100. For example, the following equation is derived from a kinematic bicycle model of vehicle motion. The position of the host vehicle 100 can be predicted based on the position at the previous time step, the velocity at the previous time step, the rotation at the previous time step, and the acceleration data from the motion data, as shown in the following equation:

[0059]

[0060] where k is the index of the time step, p k is the 3D position at time step k, δ is the length of each time step, and v k is the three-dimensional velocity vector at time step k, R k is the rotation matrix at time step k, and i a is the three-dimensional acceleration from the motion data. The velocity of the host vehicle 100 may be predicted based on the velocity at the previous time step, the rotation at the previous time step, and the acceleration data from the motion data, as shown in the following equation:

[0061] v k =v k-1 +δR k-1 i a

[0062] The rotation of the host vehicle 100 may be predicted based on the rotation at the previous time step and the angular velocity from the motion data, as shown in the following equation:

[0063]

[0064] where qk is the quaternion representing the rotation at time step k, q{} is the transformation from the rotation vector to the quaternion, and I v is the three-dimensional angular velocity from the motion data, and Represents quaternion multiplication.

[0065] The measuring step corrects the predicted value of the state variable based on the gain of the Kalman filter and the covariance of the Kalman filter, which in turn are based on the process variance and the measurement variance. The process variance is also called process noise and represents the uncertainty inherent in the motion model. As an overview of the measuring step, the computer 105 calculates the covariance based on the process variance, calculates the gain based on the covariance and the measurement variance, and corrects the state variable based on the gain.

[0066] The computer 105 calculates the covariance based on the process variance. To this end, the computer 105 first calculates the process Jacobian as follows:

[0067]

[0068] where a x 、a y and a z is the component of acceleration from the motion data; [ ] × represents a vector as a skew-symmetric matrix; F x is the Jacobian of the process; is the e×e identity matrix; 0 e×f is an e×f matrix with entries of “0”; and R{} represents a rotation matrix representation of the angular axis representation of the independent variables. The computer 105 also calculates the process noise covariance based on the process variance as follows:

[0069]

[0070] where Q is the process noise covariance, and v p is the process variance. Computer 105 also calculates the Jacobian of the process noise as follows:

[0071]

[0072] Computer 105 calculates the covariance as follows:

[0073]

[0074] The computer 105 calculates the gain of the Kalman filter based on the measurement variance and based on the covariance and the measurement model. The gain can be calculated as follows:

[0075]

[0076] where G is the gain, H is the Jacobian of the measurement model, and v m is the measurement variance. The measurement model may include an N-point perspective (PnP) algorithm. As is known, the PnP algorithm determines the pose of the camera 145 based on three-dimensional points in the environment 205 and the corresponding pixel coordinates of these points in the image frame 300. The pose of the camera 145 can be converted to the pose of the host vehicle 100 by applying a pre-stored geometric transformation.

[0077] The computer 105 corrects the state variable based on the gain. The corrected state variable may be the sum of the predicted state variable and the difference between the predicted state variable and the measured state variable, the difference weighted by the gain. The difference weighted by the gain is sometimes referred to as the state error and may be given by the following equation:

[0078] δx=G(yx)

[0079] Where x is a series of state variables (e.g., attitude, velocity, and rotation), and y is a series of measured values ​​of the state variables. The measured state variables can be determined according to a measurement model (e.g., a PnP algorithm). The corrected state variables are then the sum of the predicted state variables and the state error, as given in the following equation:

[0080]

[0081] in is the updated state variable.

[0082] The computer 105 may update the covariance for the next iteration of the Kalman filter. The covariance may be based on the gain, the measurement model, and the previous covariance, for example, as shown in the following equation:

[0083]

[0084] The computer 105 may also update the covariance based on the state error (eg, the state error of the orientation, ie, the angular component of the attitude), as shown in the following equation:

[0085]

[0086] where J is the following Jacobian:

[0087]

[0088] where δθ is the angular component of the state error δx.

[0089] When the relative motion is below the threshold, the measurement variance has a relatively larger value than when the relative motion is above the threshold. A larger value of the measurement variance results in a smaller value of the gain and a smaller correction of the predicted state variables. In other words, the prediction step is given relatively more weight, thereby causing the corrected state variables to be closer to the predicted state variables, i.e., constrained towards the predicted state variables. The computer 105 may output the corrected state variables (including the posture) as determined state variables.

[0090] Figure 4 4 is a flow chart showing an example process 400 for determining the pose of the subject vehicle 100. The memory of the computer 105 stores executable instructions for performing the steps of the process 400, and / or can be programmed in a structure such as the structure mentioned above. As a general overview of the process 400, the computer 105 receives sensor data, identifies the target vehicle 200 in the current image frame 300, associates the target vehicle 200 in the current image frame 300 with the target vehicle 200 in the previous image frame 300, and determines the relative amount of motion. In response to the relative amount of motion being below a threshold, the computer 105 determines the pose of the subject vehicle 100 subject to the constraints on the motion of the subject vehicle 100. In response to the relative amount of motion being above the threshold, the computer 105 determines the pose of the subject vehicle 100 without the constraints. In either case, the computer 105 actuates components of the subject vehicle 100 based on the pose of the subject vehicle 100. The process 400 can be repeated as each image frame 300 is delivered to the computer 105.

[0091] Process 400 begins in block 405 , where computer 105 receives image frames 300 from camera 145 and motion data from motion sensor 120 .

[0092] Next, in block 410 , the computer 105 identifies the target vehicle 200 in the image frame 300 , for example, identifying a set of pixels that form a corresponding target vehicle 200 in the image frame 300 , as described above.

[0093] Next, in box 415, the computer 105 identifies the target vehicle 200 in the image frame 300 as being the same as the corresponding target vehicle 200 from the previous image frame 300 (e.g., the immediately preceding image frame 300), that is, the target vehicle 200 in the current image frame 300 is associated with the corresponding target vehicle 200 in the previous image frame 300, for example, the pixel set of each target vehicle 200 in the current image frame 300 is associated with the corresponding pixel set of the same target vehicle 200 in the previous image frame 300, as described above.

[0094] Next, in block 420 , computer 105 determines the relative amount of motion of each target vehicle 200 , as described above.

[0095] Next, in decision block 425, computer 105 determines whether the relative motion of any of target vehicles 200 is below a threshold, as described above. In response to the relative motion of at least one of target vehicles 200 being below a threshold, process 400 proceeds to block 430. In response to all relative motions of target vehicles 200 being above a threshold, process 400 proceeds to block 435.

[0096] In block 430 , the computer 105 determines the pose of the host vehicle 100 subject to the constraints on the motion of the host vehicle 100 , as described above. After block 430 , the process 400 proceeds to block 440 .

[0097] In box 435 , the computer 105 determines the posture of the host vehicle 100 without the restraints, as described above. After box 435 , the process 400 proceeds to box 440 .

[0098] In box 440, the computer 105 actuates a component of the host vehicle 100 based on the posture of the host vehicle 100. The component may include, for example, the propulsion system 125, the braking system 130, the steering system 135, and / or the user interface 140. In other words, the computer 105 may actuate at least one of the propulsion system 125, the braking system 130, the steering system 135, or the user interface 140 based on the posture of the host vehicle 100. For example, the computer 105 may actuate the component as part of performing an advanced driver assistance system (ADAS). ADAS is a group of electronic technologies that assist operators in achieving driving functions and parking functions. Examples of ADAS include forward collision detection, lane departure detection, blind spot detection, automatic braking, adaptive cruise control, lane keeping assist, etc. For example, the computer 105 may actuate the steering system 135 or the user interface 140 based on the distance to the lane boundary as part of a lane centering feature, such as steering to prevent the host vehicle 100 from traveling too close to the lane boundary. The computer 105 may identify lane boundaries from map data and / or sensor data, such as from a camera 145 or another environmental sensor 115. If the pose of the host vehicle 100 from blocks 430 or 435 is within a distance threshold of one of the lane boundaries, the computer 105 may direct the steering system 135 to actuate to move the host vehicle 100 away from the lane boundary or direct the user interface 140 to output a message to the operator indicating that the host vehicle 100 is close to the lane boundary. For another example, the computer 105 may autonomously operate the host vehicle 100, i.e., actuate the propulsion system 125, the braking system 130, and the steering system 135 based on the pose, to, for example, navigate the host vehicle 100 through the environment 205. After block 440, the process 400 ends.

[0099] In general, the computing systems and / or devices described may employ any of a variety of computer operating systems, including but not limited to the following versions and / or varieties: Ford Application; AppLink / Smart Device Link middleware; Microsoft Operating system; Microsoft Operating system; Unix operating system (for example, distributed by Oracle Corporation of Redwood Shores, California operating systems); AIX UNIX operating system distributed by International Business Machines Corporation of Armonk, New York; Linux operating system; Mac OSX and iOS operating systems distributed by Apple Inc. of Cupertino, California; BlackBerry operating system distributed by BlackBerry Ltd. of Waterloo, Canada; and Android operating system developed by Google Inc. and the Open Handset Alliance; or provided by QNX Software Systems CAR Infotainment Platform. Examples of computing devices include, but are not limited to, an in-vehicle computer, a computer workstation, a server, a desktop, notebook, laptop or handheld computer, or some other computing system and / or device.

[0100] Computing devices typically include computer-executable instructions, wherein the instructions are executable by one or more computing devices such as those listed above. Computer-executable instructions may be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or technologies, including, alone or in combination, but not limited to Java. TM , C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as Java virtual machines, Dalvik virtual machines, etc. In general, a processor (e.g., a microprocessor) receives instructions from, for example, a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0101] Computer-readable media (also referred to as processor-readable media) include any non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash EEPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0102] The database, data repository or other data storage repository described herein may include various mechanisms for storing, accessing and retrieving various data, including hierarchical databases, file sets in file systems, application databases in special formats, relational database management systems (RDBMS), non-relational databases (NoSQL), graphic databases (GDB), etc. Each such data storage repository is typically included in a computing device using a computer operating system such as one of the above-mentioned, and is accessed via a network in any one or more of a variety of ways. The file system is accessible from the computer operating system and may include files stored in various formats. In addition to languages ​​(such as the above-mentioned PL / SQL language) for creating, storing, editing and executing stored programs, RDBMS typically also uses structured query language (SQL).

[0103] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on computer-readable media associated therewith (e.g., disks, memories, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0104] In the accompanying drawings, the same reference numerals indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps in a sequence different from the sequence described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0105] The present disclosure has been described in an illustrative manner, and it should be understood that the terms that have been used are intended to be of a descriptive nature, rather than restrictive nature. The adjectives "first" and "second" are used throughout this document as identifiers, and are not intended to indicate importance, order, or quantity. The use of "in response to" and "after determining..." indicates a causal relationship, not just a temporal relationship. In view of the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure can be practiced in other ways than those specifically described.

[0106] According to the present invention, a computer is provided, having a processor and a memory, wherein the memory stores instructions, and the instructions can be executed by the processor to: determine the relative motion between a main vehicle and a target vehicle; in response to the relative motion being lower than a threshold, determine the posture of the main vehicle subject to constraints on the motion of the main vehicle; and in response to the relative motion being higher than the threshold, determine the posture of the main vehicle in the absence of the constraints.

[0107] According to an embodiment, the instructions further include instructions for actuating a component of the host vehicle based on the posture of the host vehicle.

[0108] According to an embodiment, the constraint constrains the pose of the host vehicle towards a predicted pose, and the predicted pose is the current pose translated by a certain distance in the current direction of travel of the host vehicle.

[0109] According to an embodiment, the distance is the product of a time step and a current speed of the host vehicle.

[0110] According to an embodiment, the instructions further include instructions for determining the amount of relative movement based on image data depicting the target vehicle.

[0111] According to an embodiment, the image data includes a first image frame depicting the target vehicle and a second image frame depicting the target vehicle, and the relative motion amount includes a measure of the change between the target vehicle as depicted in the first image frame and the target vehicle as depicted in the second image frame.

[0112] According to an embodiment, the instructions for determining the amount of relative motion include instructions for comparing a first set of pixels forming the target vehicle in the first image frame with a second set of pixels forming the target vehicle in the second image frame.

[0113] According to an embodiment, the measure of variation is based on an average pixel shift of the first set of pixels and the second set of pixels, and the instructions for comparing the first set of pixels to the second set of pixels comprise instructions for determining the average pixel shift.

[0114] According to an embodiment, the instructions also include instructions for performing the following operations: extracting a first feature set from the first set of pixels, and extracting a second feature set from the second set of pixels, and the instructions for comparing the first set of pixels with the second set of pixels include instructions for comparing the first feature set with the second feature set.

[0115] According to an embodiment, the instructions further include instructions for: determining a set of feature correspondences between the first set of pixels and the second set of pixels; and determining a pixel-by-pixel shift between the feature correspondences.

[0116] According to an embodiment, the measure of variation is based on an average pixel shift of the first set of pixels and the second set of pixels, and the instructions for comparing the first set of pixels to the second set of pixels include instructions for determining the average pixel shift from the pixel-by-pixel shift.

[0117] According to an embodiment, the instructions also include instructions for performing the following operations: performing semantic segmentation to classify pixels in the first image frame and the second image frame into a first category, the first category indicating a vehicle; and determining the first pixel set and the second pixel set based on the pixels classified into the first category.

[0118] According to an embodiment, the instructions also include instructions for performing the following operations: performing object recognition on the first image frame and the second image frame to identify a vehicle; applying a bounding box around the corresponding identified vehicle; and determining the first set of pixels and the second set of pixels based on the bounding box.

[0119] According to an embodiment, the instructions also include instructions for performing the following operations: performing semantic segmentation to classify pixels in the first image frame and the second image frame into a first category, the first category indicating a vehicle, and the first set of pixels and the second set of pixels are from pixels that are both within the bounding box and classified as the first category.

[0120] According to an embodiment, the instructions further include instructions for identifying the target vehicle in the second image frame as the same vehicle as the target vehicle in the first image frame.

[0121] According to an embodiment, the image data originates from a camera mounted to the host vehicle.

[0122] According to an embodiment, the instructions for determining the pose of the host vehicle comprise instructions for determining the pose of the host vehicle based on a measurement variance of a sensor of the host vehicle, and the constraint is that a value of the measurement variance increases.

[0123] According to an embodiment, the instructions for determining the pose of the host vehicle include instructions for executing a Kalman filter, and a state variable of the Kalman filter includes the pose of the host vehicle.

[0124] According to an embodiment, the instructions further comprise instructions for calculating a gain of the Kalman filter based on a measurement variance of a sensor of the host vehicle, and the constraint is that the value of the measurement variance increases.

[0125] According to the present invention, a method includes: determining a relative motion amount between a main vehicle and a target vehicle; in response to the relative motion amount being lower than a threshold, determining a posture of the main vehicle subject to a constraint on the motion of the main vehicle; and in response to the relative motion amount being higher than the threshold, determining the posture of the main vehicle in the absence of the constraint.

Claims

1. A method comprising: determining the relative motion between the host vehicle and the target vehicle; responsive to the amount of relative motion being below a threshold, determining a posture of the host vehicle subject to a constraint on motion of the host vehicle; as well as In response to the amount of relative motion being above the threshold, the pose of the host vehicle in the absence of the restraint is determined. 2 . The method of claim 1 , further comprising actuating a component of the host vehicle based on the posture of the host vehicle. 3 . The method of claim 1 , wherein the constraint constrains the pose of the host vehicle toward a predicted pose, and the predicted pose is a current pose translated a certain distance in a current direction of travel of the host vehicle. 4 . The method of claim 1 , further comprising determining the amount of relative motion based on image data depicting the target vehicle.

5. The method of claim 4, wherein the image data includes a first image frame depicting the target vehicle and a second image frame depicting the target vehicle, and the relative motion amount includes a measure of the change between the target vehicle as depicted in the first image frame and the target vehicle as depicted in the second image frame.

6. The method of claim 5, wherein determining the amount of relative motion comprises comparing a first set of pixels forming the target vehicle in the first image frame to a second set of pixels forming the target vehicle in the second image frame.

7. The method of claim 6, further comprising extracting a first feature set from the first set of pixels and extracting a second feature set from the second set of pixels, wherein comparing the first set of pixels with the second set of pixels comprises comparing the first feature set with the second feature set.

8. The method of claim 7, further comprising determining a set of feature correspondences between the first set of pixels and the second set of pixels, and determining a pixel-by-pixel shift between the feature correspondences.

9. The method of claim 8, wherein the measure of change is based on an average pixel shift of the first set of pixels and the second set of pixels, and comparing the first set of pixels to the second set of pixels comprises determining the average pixel shift from the pixel-by-pixel shift.

10. The method of claim 6, further comprising: performing semantic segmentation to classify pixels in the first image frame and the second image frame into a first category, the first category being indicative of a vehicle; And determining the first set of pixels and the second set of pixels based on the pixels classified into the first category.

11. The method of claim 6, further comprising: performing object recognition on the first image frame and the second image frame to identify a vehicle; applying a bounding box around said respective identified vehicle; And determining the first set of pixels and the second set of pixels based on the bounding box.

12. The method of claim 11, further comprising: Semantic segmentation is performed to classify pixels in the first image frame and the second image frame into a first category, the first category being indicative of a vehicle, wherein the first set of pixels and the second set of pixels are from pixels that are both within the bounding box and classified into the first category.

13. The method of claim 1, wherein determining the pose of the host vehicle comprises determining the pose of the host vehicle based on a measurement variance of a sensor of the host vehicle, and the constraint is an increasing value of the measurement variance.

14. The method of claim 1 , wherein determining the posture of the host vehicle comprises executing a Kalman filter, the state variables of the Kalman filter comprising the posture of the host vehicle, the method further comprising calculating a gain of the Kalman filter based on a measurement variance of a sensor of the host vehicle, wherein the constraint is an increase in the value of the measurement variance.

15. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method of one of claims 1 to 14.

Images