Drive system, vehicle, and method for automated and / or assisted driving
The integration of inertial measurement units, wheel odometry, and electromagnetic sensors improves vehicle localization accuracy and reliability by enhancing vehicle attitude estimation, addressing the limitations of GPS-dependent systems in urban areas.
Patent Information
- Application Number
- JP2023576216
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-08-11
- Filing Date
- 2022-02-21
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2042-02-21
AI Technical Summary
Existing vehicle localization systems, particularly in urban environments with limited GPS visibility, struggle with accuracy and reliability due to reliance on Global Positioning System (GPS) and the failure of clear line of sight to satellites.
A drive device and method combining inertial measurement units, wheel odometry, and electromagnetic sensors to enhance vehicle attitude estimation through visual odometry, utilizing inertial-related, position-related, and image data to improve accuracy and robustness of vehicle localization.
Enhances the accuracy and reliability of vehicle attitude estimation by integrating multiple sensor data, providing more precise vehicle localization even in urban environments where GPS is unreliable.
Smart Images

Figure 0007594691000008 
Figure 0007594691000009 
Figure 0007594691000010
Abstract
Description
[Technical field]
[0001] The present invention relates to a drive device for automated and / or assisted driving of a vehicle, the drive device comprising a first input port configured to receive inertial-related data measured by an inertial measurement unit, a second input port configured to receive position-related data measured by a wheel odometry unit, and a third input port configured to receive first image data and second image data.
[0002] The present invention further relates to a vehicle comprising an inertial measurement unit configured to output inertial related data, a wheel odometry unit configured to output position related data, and an electromagnetic sensor unit configured to output first image data and second image data, the second image data being image data preceding the first image data.
[0003] The present invention also relates to a computer-implemented method for driving a vehicle in an automatic mode and / or a driver assistance mode, the method comprising the steps of outputting inertial-related data by an inertial measurement unit, outputting position-related data by a wheel odometry unit, and outputting first image data and second image data by an electromagnetic sensor unit. [Background technology]
[0004] A vehicle operating in an autonomous mode (e.g., driverless) or semi-autonomous mode (with driver assistance) relieves the driver from some driving-related tasks. When operating in an autonomous mode, the vehicle can navigate to various locations, allowing the vehicle to travel with minimal human interaction or, in some cases, without an occupant. Similarly, in an assisted driving mode, some of the driver's tasks are performed by (advanced) driver assistance systems.
[0005] Vehicle localization is the position (and orientation) of a vehicle on the Earth. Accurate vehicle localization is a key component for autonomous vehicles. Traditional vehicle localization systems rely primarily on GPS (Global Positioning System), where the vehicle has a clear line of sight to multiple GPS satellites. However, GPS tends to fail in situations such as tunnels and cities with skyscrapers, where there is limited or no line of sight between the satellites and the vehicle.
[0006] US Patent Application Publication No. 2020 / 103664 relates to a method and apparatus for outputting attitude information. US Patent Application Publication No. 2020 / 081134 describes validation of global navigation satellite system position data with other sensor data. US Patent Application Publication No. 2019 / 368879 relates to a vision-aided inertial navigation system for ground vehicle localization. [Prior art documents] [Patent documents]
[0007] [Patent Document 1] US Patent Application Publication No. 2020 / 103664 [Patent Document 2] US Patent Application Publication No. 2020 / 081134 [Patent Document 3] US Patent Application Publication No. 2019 / 368879 Summary of the Invention [Problem to be solved by the invention]
[0008] It is an object of the present invention to provide a drive system, a vehicle, and a computer-implemented method for providing more accurate and more robust estimation of attitude (eg, position and orientation). [Means for solving the problem]
[0009] This object is solved by the subject matter of the independent claims. The dependent claims describe preferred embodiments of the invention.
[0010] A drive device for automated and / or assisted driving of a vehicle includes a first input port, a second input port, a third input port, and a visual odometry unit. The first input port is configured to receive inertial related data measured by an inertial measurement unit. The second input port is configured to receive position related data measured by a wheel odometry unit. The third input port is configured to receive first image data and second image data by an electromagnetic sensor unit, the second image data being image data prior to the first image data. The visual odometry unit is configured to receive the inertial related data, the position related data, the first image data, and the second image data, and is configured to output an estimated attitude of the vehicle calculated by associating first feature points extracted from the first image data with second feature points extracted from the second image data and evaluating the association between the first feature points and the second feature points based on the inertial related data from the inertial measurement unit and the position related data from wheel odometry.
[0011] Optionally, the drive device may include a control unit configured to generate a drive signal for automated and / or assisted driving based on the estimated attitude.
[0012] The vehicle comprises an inertial measurement unit configured to output inertial related data, a wheel odometry unit configured to output position related data, an electromagnetic sensor unit configured to output first image data and second image data, the second image data being image data prior to the first image data, and a visual odometry unit configured to receive the inertial related data, the position related data, the first image data, and the second image data, and configured to output an estimated pose of the vehicle calculated by associating first feature points extracted from the first image data with second feature points extracted from the second image data and evaluating the association between the first feature points and the second feature points based on the inertial related data from the inertial measurement unit and the position related data from wheel odometry.
[0013] Optionally, the vehicle may include a controller configured to operate the vehicle in an automated driving mode and / or an assisted driving mode based on the estimated attitude.
[0014] The present invention is based on the idea of combining multiple sensors to estimate the vehicle attitude in order to obtain a more accurate, more robust and / or more reliable vehicle attitude. The present invention uses information about the estimated attitude of the vehicle received from an inertial measurement unit and a wheel odometry unit to make the determination of the vehicle attitude based on visual odometry more accurate and / or more robust. Thus, the attitude of the vehicle is determined based on more extensive information. In addition, the calculation of the attitude based on inertial related data and / or position related data may be more accurate than the estimated attitude determined using image data. Thus, the higher accuracy of the inertial related data and / or position related data can be used to increase the accuracy of the estimated attitude determined using image data.
[0015] The vehicle and / or the drive of the vehicle may be an autonomous or self-driving vehicle, sometimes called a robocar. Alternatively or additionally, the vehicle and / or the drive may be a semi-autonomous vehicle. The drive may therefore be considered as a control device of an advanced driver assistance system. According to the classification system with six levels published in 2021 by the automotive standardization organization SAE International as J3016_202104 "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles", an autonomous vehicle may be considered as level 4 or level 5, and a semi-autonomous or assisted driving vehicle may be considered as level 1 to level 3.
[0016] The visual odometry unit and / or the control unit may be part of a computer or processor that performs the tasks outlined below. The computer may include the necessary hardware (e.g., processor, memory, storage) and software (e.g., operating system, programs) to perform the tasks outlined below. The tasks performed by the visual odometry unit may be stored as algorithms and / or other forms of computer code on a storage device.
[0017] The storage device may include one or more memories that may be implemented via multiple memory devices to provide a given amount of memory. The storage device may include one or more volatile storage (or memory) devices, such as random access memory (RAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), or other types of storage devices. The storage device may also include a solid-state device (SSD). However, in other embodiments, the storage device may include a hard disk drive (HDD) with or without a smaller capacity SSD storage that serves as an SSD cache to enable non-volatile storage of context state and other such information during a power-down event so that a fast power-up can occur upon resumption of system activity.
[0018] The algorithms stored in the storage device may be updated periodically or intermittently. To this end, the storage device may be electronically and / or communicatively connected or coupled to a communication unit that allows wired or wireless communication with a network, and thus with a server, other types of storage devices external to the vehicle or drive, and / or with other vehicles.
[0019] The communication unit may be considered a network interface device that may include a wireless transceiver and / or a network interface card (NIC). The wireless transceiver may be a WiFi transceiver, an infrared transceiver, a Bluetooth transceiver, a WiMax transceiver, a wireless mobile phone transceiver, or other radio frequency (RF) transceiver, or a combination thereof.
[0020] An inertial measurement unit is a device for measuring forces experienced by the inertial measurement unit, such as acceleration, rotation, yaw, and / or velocity change. The inertial measurement unit may be fixedly mounted to the vehicle and / or drive system, in which case the inertial measurement unit is configured to measure the acceleration and / or velocity change of the vehicle and / or drive system.
[0021] The inertial measurement unit may include one or more sensors, such as an accelerometer, a gyroscope (gyrometer), and / or a magnetometer. Measurements generated by the sensors may be processed by a processor of the inertial measurement unit and / or by a visual odometry unit to determine acceleration and / or velocity changes. In other words, the inertial measurement unit may include sensors whose measurements can be used to determine the acceleration, rotation, yaw, and / or velocity changes of the inertial measurement unit. The gyroscope may include a microelectromechanical system (MEMS) gyroscope.
[0022] The acceleration, rotation, yaw, and / or velocity changes determined by the inertial measurement unit are output as inertial related data. The inertial measurement unit may be communicatively and / or electronically connected / coupled to a first input port, which may be an interface of the drive device for receiving the inertial related data.
[0023] A wheel odometry unit is a device for measuring parameters indicative of the attitude of a vehicle, such as the distance traveled by the vehicle, the speed of the vehicle, and / or the inclination of the wheels of the vehicle. The wheel odometry unit may include one or more sensors, such as a sensor for determining the rotation of the wheels of the vehicle, a sensor for determining the inclination of the wheels of the vehicle, and / or a sensor for determining the position of the steering wheel of the vehicle. Measurements of these parameters collected by the sensors of the wheel odometry unit may be processed by a processor of the wheel odometry unit and / or by a visual odometry section to determine the distance traveled and / or the heading of the vehicle.
[0024] The distance traveled and / or the heading of the vehicle are output by the wheel odometry unit as position related data. The wheel odometry unit may be communicatively and / or electronically connected / coupled to a second input port, which may be an interface of the drive for receiving inertia related data.
[0025] The electromagnetic sensor unit may include sensors capable of determining the vehicle's surroundings in two or three dimensions. The electromagnetic sensor unit may use electromagnetic radiation in various wavelength ranges, such as the optical (visible) wavelength range and / or the radio frequency wavelength range (RF), to sense and / or explore the vehicle's surroundings. The electromagnetic sensor unit may be configured to detect and / or emit electromagnetic radiation in a single wavelength range or multiple wavelength ranges. The electromagnetic sensor unit may be a sensor unit for detecting and / or emitting electromagnetic radiation. The electromagnetic sensor unit may be considered as an electromagnetic radiation sensor unit.
[0026] For example, the electromagnetic sensor unit may include a mono camera and / or a stereo camera, i.e., two cameras spaced apart from each other to obtain a stereo image of the vehicle's surroundings. The cameras may be still cameras and / or video cameras. The cameras may be mechanically movable, for example, by mounting the cameras on a rotating and / or tilting platform.
[0027] However, other imaging or scanning techniques using electromagnetic radiation may be used and thus form part of the electromagnetic sensor unit. The optical sensing device may alternatively or additionally include a radar device and / or a light detection and ranging (LiDAR) device. A LiDAR device may sense objects around the vehicle using lasers. A LiDAR device may include one or more laser sources, a laser scanner, and one or more detectors, among other system components. A radar device may be a system that utilizes radio signals to sense objects in the vehicle's local environment. In some embodiments, the radar unit, in addition to sensing objects, may further sense the speed and / or direction of travel of other objects, such as other vehicles on the road.
[0028] The electromagnetic sensor unit is configured to sense / probe continuously or periodically around the vehicle such that the electromagnetic sensor unit can be used to detect changes in the vehicle's surroundings. For example, the electromagnetic sensor unit is configured to capture a second image of the vehicle's surroundings and, at a later time, a first image of the vehicle's surroundings. The first and second images may have the same orientation and position relative to the vehicle such that differences between the first and second images are due to vehicle motion. Preferably, a continuous stream of the first and second images is generated. The time difference between the first and second images may be in the millisecond range.
[0029] The first image and the second image are output by the electromagnetic sensor unit as first image data and second image data, respectively. The first and / or second image data may include image data from left and right cameras of a stereo camera. More generally, the first and / or second image data includes all data recorded at a particular point in time by a single or multiple sensors of the electromagnetic sensor unit. The first image data refers to image data recorded later compared to the second image data. The first image data can be considered as current image data and the second image data can be considered as previous image data.
[0030] The third input port may be an interface with the driver for receiving first and second image data, the first and second image data being data capable of generating a first and second image, respectively, for example, the first and second image data being a collection of data points that can be associated with pixels of the first and second images, such that the first and second images can be reconstructed from the first and second image data, respectively.
[0031] The visual odometry unit is configured to receive the inertial related data, the position related data, the first image data, and the second image data via the first input port, the second input port, and the third input port. The visual odometry unit is configured to calculate or compute an estimated attitude of the vehicle by analyzing the first image data and the second image data. Generally, a difference between the first image (represented by the first image data) and the second image (represented by the second image data) indicates a motion of the vehicle from which the attitude of the vehicle can be estimated.
[0032] The visual odometry unit may be configured to extract identical feature points from the first image data and the second image data, or in other words, from the first image and the second image. The feature points (sometimes also called key points) relate to distinctive features and / or features of interest in the images, such as edges, corners, and / or intersections. The translation of the feature points on the image plane between the first image (or the first pair of images of the stereo image) and the second image (or the second pair of images of the stereo image) indicates the movement of the vehicle. Thus, the translation (direction and magnitude) of the feature points on the image plane from the first image and the second image may be used to calculate the direction and / or speed of the movement of the vehicle in the real world.
[0033] Preferably, multiple feature points are extracted and used to calculate the pose of the vehicle. The pose may include the rotation (yaw) and translation (direction and / or distance) of the vehicle, both of which may be represented by vectors. In theory, multiple feature points should all result in the determination of the same pose of the vehicle. However, due to measurement inaccuracies and / or artifacts, each pair of first and second feature points may result in a (slightly) different pose. In addition, the visual odometry unit may associate first and second feature points that do not refer to the same real-world feature. In other words, the association of feature points may include errors.
[0034] The visual odometry unit may be configured to use an average of multiple estimated poses to eliminate obviously erroneous feature points or feature point associations (e.g., if they deviate from the average by a predefined threshold) and / or use other mathematical or statistical means to determine the actual value based on a large number of measured points. Optionally, the visual odometry unit may be configured to use Random Sample Consensus (RANSAC). RANSAC uses associated feature points or keypoints to estimate a pose as follows: First, a fixed number of associated keypoints are randomly selected and then a relative pose is estimated. This estimated pose is then used to determine the number of keypoint associations that match (inlier) with this estimated relative pose. This process is repeated until some stopping criterion where the relative pose has a maximum number of inliers. In summary, RANSAC is a consensus-based approach to determine the most likely relative pose based on a maximal subset of keypoint associations (assuming that a maximal subset can be found, and if the maximal subset actually corresponds to the correct relative pose).
[0035] The visual odometry unit may be configured to extract second feature points (images of real word features) from the second image data and to search for identical feature points (images of the exact same real word features) in the first image data, which feature points correspond to the first feature points. The visual odometry unit may then be configured to determine a distance and / or orientation between the two feature points identified in the second image data and the first image data. The visual odometry unit may then be configured to determine the vehicle's motion based on a pre-stored relationship between the distance and / or orientation in the image data and the vehicle's motion. In other words, the distance and / or orientation on the image plane between the two feature points identified in the second image data and the first image data infers that the electromagnetic sensor unit (or the vehicle) has moved (assuming that the corresponding real-world object is fixed). To calculate the relative pose (i.e., the motion of the electromagnetic sensor unit or the vehicle), geometric approaches (e.g., Perspective-n-Point (Pnp) algorithm) and optimization approaches (minimizing reprojection errors) may be used. In both of these cases, the 3D real-word position of the corresponding 2D feature in the image data can be processed, where the first / second 3D point is projected and compared with the second / first feature point.
[0036] The visual odometry unit may be configured to determine the vehicle's pose based on the vehicle's last known position (as determined by the localization unit) and / or to compare the estimated pose with pose estimation-based data received from the localization unit.
[0037] The positioning unit may be a satellite transceiver (e.g., a Global Positioning System (GPS) transceiver) for determining a current position of the vehicle. The current position of the vehicle is processed and / or output by the positioning device as vehicle positioning data. Thus, the current position of the vehicle can be determined based on the positioning data. The positioning unit may be communicatively and / or electronically connected / coupled to the visual odometry portion, for example via a fourth input port of the drive device.
[0038] The visual odometry unit may include a deep learning solution for an autonomous vehicle (e.g., using a deep neural network (DNN) such as a convolutional neural network (CNN)). The neural network or artificial intelligence (AI) is used to determine feature points, vehicle motion, and / or an estimated pose of the vehicle. Alternatively or additionally, the visual odometry unit may be configured to use algorithms to calculate / compute these parameters.
[0039] The visual odometry unit uses the inertial-related data and the position-related data when calculating the estimated pose. For example, the inertial-related data and / or the position-related data may be used to verify / confirm the estimated pose and / or the estimated movement of the vehicle. Thus, the accuracy and / or reliability of the estimated pose may be increased.
[0040] The control unit is not essential to the present invention, since it can be implemented by a known control unit for driving a vehicle autonomously or semi-autonomously. The present invention can be considered to be provided with an estimated attitude based on which the control unit operates.
[0041] The control unit is electronically and / or communicatively connected or coupled to the visual odometry unit to receive the estimated pose. The control unit may be a part or functional unit of the computer mentioned above. The control unit is configured to generate signals to control a steering device, a throttle device (also called an acceleration device), and a braking device for driving the vehicle on a drivable road. The steering device, the throttle device, and the braking device may be part of a control device for navigating the vehicle.
[0042] The steering device may be part of the vehicle for adjusting the heading or direction of the vehicle. The throttle device may also be part of the vehicle for controlling the speed of the motor or engine and thus the speed and acceleration of the vehicle. The brake device may be part of the vehicle for slowing down the vehicle by providing friction to slow down the wheels or tires of the vehicle. The steering device, the throttle device, and / or the breaker device may be controlled based on signals output by the controller.
[0043] The controller may execute an algorithm and / or include such a neural network (AI) for navigating the vehicle based on the estimated pose information.
[0044] The drive unit may be part of the vehicle and is electronically and / or communicatively connected or coupled to the inertial measurement unit, the wheel odometry unit, and the electromagnetic sensor unit by a first input port, a second input port, and a third input port, respectively. In other words, the inertial measurement unit outputs inertial related data that is input to the drive unit via the first input port, the wheel odometry unit outputs position related data that is input to the drive unit via the second input port, and the electromagnetic sensor outputs image data that is input to the drive unit via the third input port.
[0045] In an optional embodiment, the visual odometry unit includes a feature generation and association unit configured to generate the first feature points and the second feature points and associate the first feature points with the second feature points, and optionally the visual odometry unit further includes a feature point removal unit configured to project first projected feature points from the second feature points by using rotation data and translation data calculated based on the inertial related data and the position related data, and configured to calculate pixel distances between the first feature points and the first projected feature points to evaluate certainty of the association between the first feature points and the second feature points.
[0046] The identification of the above-mentioned first and second feature points may be performed by a feature generation and association unit, which may be part or a functional unit of the above-mentioned computer. In particular, the feature generation and association unit is configured to identify closest and / or identical real-world features in both the first image data and the second image data and to associate the first and second feature points in the first image data and the second image data, respectively, in that they refer to the same real-world features.
[0047] The feature generation and association unit may be configured to calculate a pixel distance between the first feature point and the second feature point, the pixel distance being the distance in pixels from a location of the first feature point in the first image to a location of the second feature point at a corresponding pixel location in the second image, the pixel distance being used to evaluate the association between the first feature point and the second feature point.
[0048] The feature point remover may be a part or functional unit of the computer mentioned above. The feature point remover may be configured to calculate the first projected feature points based on the corresponding second feature points, the inertial related data, and the position related data. The feature point remover may be configured to calculate the motion of the vehicle between the time when the second image data was generated and the time when the first image data was generated based on the inertial related data and the position related data. Both types of data are indicative of the motion of the vehicle and based on these two types of data the motion of the vehicle can be determined. The motion of the vehicle can be represented in translation (represented by the translation data) and / or in rotation / yaw (represented by the rotation data).
[0049] The feature remover may be configured to convert the vehicle motion into a pixel distance, and to add the pixel distance to the second feature to obtain the projected first feature. The feature point remover may execute an algorithm and / or include a neural network (AI) for determining / calculating the vehicle motion, the corresponding pixel distances, and / or the projected first feature points.
[0050] TIFF0007594691000001.tif72169
[0051] TIFF0007594691000002.tif40169
[0052] The camera projection matrix K may be vehicle specific and may be determined by calculating the projection using known positions of features in the image data and known positions of corresponding real-world features. If the coordinate system of the electromagnetic sensor unit is different from the coordinate system of the vehicle, then a transformation between these two coordinate systems needs to be added to the above equation (e.g., using an additional transformation matrix). The feature point remover may calculate a pixel distance between the first feature point and the first projected feature point to establish a confidence level of whether the association between the first feature point and the second feature point reflects the true pose of the vehicle. Thus, the pixel distance between the first feature point and the first projected feature point can be used to assess whether the association between the first feature point and the second feature point was correct, i.e., whether the first feature point and the second feature point refer to the same real-world feature.
[0053] For example, the feature point remover may use a statistical method to evaluate whether a pixel distance between the first feature point and the projected first feature point indicates that an association between the first feature point and the second feature point reflects a true pose of the vehicle. Optionally, the feature point remover removes the first feature point if the pixel distance between the first feature point and the projected first feature point is greater than a variance of a statistical distribution of all pixel distances between all the first feature points and the projected first feature points.
[0054] The feature point remover may determine a certainty of the association between the first feature point and the second feature point. The certainty may include assigning a confidence level of how likely it is that the estimated pose calculated based on the association between the first feature point and the second feature point reflects the true pose. For example, the certainty is a confidence level that is a (mathematical) function of pixel distance. The certainty or confidence level may be higher for smaller pixel distances. The certainty may be used to weight a particular association between the first feature point and the second feature point in calculating the estimated pose.
[0055] The certainty or confidence may be used to accept or reject an association between a first feature point and a second feature point, e.g., if the certainty is below a (predetermined) threshold, the association is rejected, otherwise it is accepted.
[0056] In an optional embodiment, the feature point remover is configured to remove the association between the first feature point and the second feature point if the pixel distance is greater than a predetermined value.
[0057] In this optional embodiment, the pixel distance itself is the measure that determines whether to accept or reject an association between a first feature point and a second feature point -- in other words, the certainty is directly related to the pixel distance.
[0058] The unremoved associations between the first feature points and the corresponding second feature points are used in conjunction with the corresponding second feature points to calculate the estimated pose. Removing the associations may include removing both the first feature points and the second feature points, only the first feature points, or only the second feature points.
[0059] The feature point remover may calculate a Euclidean distance between the first feature point and the projected first feature point. The predetermined value may be a fixed value, or may be a predicted translation t WO and / or expected rotation R IMU For example, the expected translation t WO and / or expected rotation R IMU There may be a linear or other form of relationship between
[0060] If the first feature point is removed, the pair of the first and second feature points may not be used to calculate the estimated pose, and other pairs of the first and second feature points that meet a certain threshold requirement may be used to calculate the estimated pose.
[0061] This is a method for removing outliers in pairs of first and second feature points. Thus, the unremoved pairs of first and second feature points provide a more accurate and more reliable prediction of the vehicle's pose. In particular, this method for removing outliers works for real-world objects in the near and far fields of the electromagnetic sensor unit, i.e., near and far from the electromagnetic sensor unit.
[0062] In an optional embodiment, the visual odometry unit further includes a world point removal unit configured to compare a physical distance in depth between the first feature point and the second feature point with an expected physical distance in depth, optionally calculated based on the position related data from the wheel odometry unit.
[0063] The world point remover may be a part or functional unit of the computer mentioned above. The physical distance refers to the real world distance, i.e. the distance that can be measured in the real world. This is in contrast to the pixel distance, which is the distance that can be measured in the image or image data generated by the electromagnetic sensor unit. The depth may correspond to the z coordinate of the world point vector, i.e. the vector connecting two points in the real world. The z coordinate may be a coordinate in the vehicle's coordinate system that points ahead of the vehicle, i.e. in the driving direction if the vehicle is moving straight ahead. As mentioned above, the depth is calculated based on the vehicle translation t when the vehicle is moving straight ahead, i.e. when the vehicle rotation is zero. WO may be parallelized with
[0064] The camera coordinate frame, or typically the coordinate frame of the electromagnetic sensor unit, can be positioned arbitrarily relative to the vehicle coordinate frame. As an example, the camera can be positioned looking at the rear or at the side of the vehicle. In addition, the coordinate system of the electromagnetic sensor unit can be angled relative to the vehicle coordinate system. A transformation of the camera coordinate frame (or the frame of the electromagnetic sensor unit) to the vehicle coordinate frame may be required.
[0065] The x coordinate in the vehicle coordinate system may extend vertically, and the y coordinate in the vehicle coordinate system may extend vertically, for example to the right when looking in the z direction.
[0066] In this embodiment, the electromagnetic sensor unit must include a sensor or sensors capable of determining the depth of the imaged surroundings. This is necessary, for example, to calculate the depth using triangulation. A stereo camera may be used to calculate the set depth using triangulation. Alternatively, the depth may be directly derivable from LiDAR measurements or by using multiple cameras to calculate the depth.
[0067] The world point remover may be configured to convert the association of the first feature point with the second feature point into a physical distance. The world point remover may include an algorithm or a neural network (AI) capable of converting information about the translation of the first feature point relative to the second feature point in the image coordinate system into a physical distance. Alternatively or additionally, the feature point projection unit may be configured to convert the pixel positions of the first feature point and the second feature point (i.e. the positions of the first feature point and the second feature point in the first image data and the second image data) into corresponding positions in the real world, for example in the coordinate system of the vehicle or the electromagnetic sensor unit (e.g. a camera or other sensor). For this purpose, the world point remover may use the inverse of the camera projection matrix K. The world point remover may include an algorithm or a neural network (AI) capable of converting the pixel positions into real world positions. The world point remover may then calculate or compute a real world distance or a physical distance from the calculated real world positions. Finally, the world point remover may calculate a physical distance in depth (or a z-component of the physical distance) from the calculated physical distance or physical translation.
[0068] The expected physical distance may be calculated using position related data from the wheel odometry unit. The world point remover may include an algorithm and / or a neural network (AI) capable of calculating / computing the expected physical distance from the position related data. Finally, the world point remover may calculate the expected physical distance in depth (or the z-component of the expected physical distance) from the calculated expected physical distance or the expected translation or physical translation.
[0069] The comparison between the physical distance at depth and the expected physical distance at depth can be used to remove past outliers of the first and second feature points. The remaining or non-removed pairs of the first and second feature points are used to calculate or compute an estimated pose. Because the outliers are removed, the estimated pose is more accurate and / or more reliable. This method of removing outliers is particularly accurate in the near field of the electromagnetic sensor unit.
[0070] In an optional embodiment, the world point remover is configured to calculate an upper limit of the expected physical distance and a lower limit of the expected physical distance, and to accept the association between the first feature point and the second feature point if the physical distance at the depth is between the upper limit of the expected physical distance and the lower limit of the expected physical distance.
[0071] The world point remover may include an algorithm and / or a neural network (AI) capable of calculating / computing the upper and / or lower bounds. The upper and / or lower bounds may be fixed or predefined values. Alternatively, the upper and / or lower bounds may be variable and may be adjusted according to the translation t WO and / or rotation R IMU Preferably, the upper and / or lower limits can depend on the translation t, which can be calculated based on the position-related data, for example as described above. WO Depends only on
[0072] The lower and / or upper bounds are the translation t WO and / or rotation R IMU , and preferably a function of the translation t WO The function is a linear, quadratic or lower and / or upper translation function. WO and / or rotation R IMU It may also be some other type of function that links to
[0073] If the physical distance at depth is within the boundaries set by the upper and lower bounds, the physical distance is expected to accurately and / or reliably represent the actual physical distance. In this method, the expected physical distance is used to determine the upper and lower bounds that are used to evaluate the accuracy of the physical distance at depth. If the physical distance at depth is deemed accurate, the corresponding pair of the first feature point and the second feature point is used to calculate an estimated pose.
[0074] TIFF0007594691000003.tif69167
[0075] In an optional embodiment, the estimated attitude includes an estimated rotation and an estimated translation, and optionally the visual odometry unit includes a decision rule unit configured to generate an expected rotation and an expected translation calculated based on inertial related data from the inertial measurement unit and position related data from the wheel odometry unit, and to evaluate the estimated rotation and the estimated translation based on the expected rotation and the expected translation.
[0076] The estimated pose may be calculated as described above, i.e., outliers in the pairs of first and second feature points may be removed and are not used to calculate the estimated pose, however, it is also possible to use all pairs of first and second feature points to calculate the estimated pose.
[0077] In this embodiment, the final estimated pose is evaluated taking into account the predicted pose. In the above embodiment, a pair of the first feature point and the second feature point is evaluated to determine the final estimated pose. This means that the above embodiment is related to the selection of data, whereas this embodiment is related to the evaluation of the final estimated pose.
[0078] The visual odometry unit may be configured to calculate an estimated rotation and / or an estimated translation. The estimated rotation and / or the estimated translation may be calculated or computed using both the inertial related data and the position related data. However, it is also possible that the estimated rotation is calculated or computed based on the inertial related data only and / or the estimated translation is calculated or computed based on the position related data only. To this end, the visual odometry unit may execute an algorithm and / or include such a neural network (AI) that is able to determine the estimated rotation and / or the estimated translation based on the inertial related data and / or the position related data.
[0079] The decision rule unit may be configured to evaluate the estimated pose based on the expected pose. Alternatively, the decision rule unit may be configured to evaluate the estimated translation based on the expected translation and / or the estimated rotation based on the expected rotation. The decision rule unit may use statistical methods or any other methods for the evaluation of the estimated pose. The evaluation may involve removal or correction of the estimated pose, estimated translation, and / or estimated rotation taking into account the expected pose, expected translation, and / or expected rotation. Alternatively, the evaluation may include evaluating the accuracy of the estimated pose, for example by associating with the estimated pose a statistical value indicative of how true the estimated pose is believed to be.
[0080] In optional embodiments, the expected translation includes an upper limit of the expected translation and a lower limit of the expected translation, and optionally the expected rotation includes an upper limit of the expected rotation and a lower limit of the expected rotation, and further optionally, the decision rule unit is configured to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation when the vehicle is on a straight road, or to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation and the estimated rotation is between the upper and lower limits of the expected rotation when the vehicle is on a curved road.
[0081] The upper expected translation limit and / or the lower expected translation limit may be a fixed or predetermined value. Alternatively, the upper expected translation limit and / or the lower expected translation limit may be a function of the expected translation and / or expected rotation. Similarly, the upper expected rotation limit and / or the lower expected rotation limit may be a function of the expected translation and / or expected rotation. Preferably, the upper expected translation limit and / or the lower expected translation limit are a function of the expected translation and / or the upper expected rotation limit and / or the lower expected rotation limit are a function of the expected rotation. Instead of a function between the upper / lower limit and the expected translation / rotation, other types of relationships may be used, such as tables.
[0082] If the vehicle is traveling in a straight line, it is detected that the vehicle is traveling on a straight road and / or the steering angle is detected to be equal to zero, and it is assumed that there is no rotation. In this case, the decision rule unit only refers to the translation. In other words, the decision rule unit only evaluates the estimated attitude taking into account the expected translation, such as whether the estimated translation is between the upper and lower limits of the expected translation.
[0083] If the vehicle is not traveling in a straight line, the vehicle is detected to be traveling on a curved road and / or the steering angle is detected to be not equal to zero, and a rotation is assumed to exist. In this case, the decision rule unit refers to both the translation and the rotation. In other words, the decision rule unit evaluates the estimated attitude taking into account both the expected translation and the expected rotation. For example, the decision rule unit evaluates both whether the estimated translation is between the upper and lower bounds of the expected translation and whether the estimated rotation is between the upper and lower bounds of the expected rotation.
[0084] If these conditions are met, the rule determiner accepts the estimated pose, which can then be used to navigate the vehicle. The accepted estimated pose may be forwarded to the drive unit. However, if these conditions are not met, the rule determiner rejects the estimated pose.
[0085] If the electromagnetic sensor unit includes sensors of different types, the above-mentioned method for evaluating the estimated attitude may be extended to all these different types of sensors. This means that the estimated attitude may be calculated based on image data generated by a first type of sensor (e.g., a stereo camera), a second type of sensor (e.g., a LiDAR device), and / or a third type of sensor (e.g., a radar device). Thus, various estimated attitudes may be calculated and evaluated taking into account the expected attitude, as explained above. The various estimated attitudes accepted in the above-mentioned evaluation process may be combined to achieve a single estimated attitude. The combination of the various estimated attitudes may be performed using a fusion module of extended Kalman filtering. Kalman filtering, also known as linear quadratic estimation (LQE), is a known filtering technique for combining different quantities in statistics and control theory.
[0086] A computer-implemented method for operating a vehicle in an automated and / or driver assistance mode comprises the following steps. a) outputting inertial related data by an inertial measurement unit; b) outputting position related data by the wheel odometry unit; c) outputting first image data and second image data by an electromagnetic sensor unit, the second image data being image data before the first image data; and d) receiving, by a visual odometry unit, the inertial related data, the position related data, the first image data, and the second image data, associating first feature points extracted from the first image data with second feature points extracted from the second image data, and outputting an estimated attitude of the vehicle calculated by evaluating the association between the first feature points and the second feature points based on the inertial related data from the inertial measurement unit and the position related data from wheel odometry.
[0087] Optionally, the method may additionally include driving, by a control unit, the vehicle in an automated driving mode and / or an assisted driving mode based on the estimated attitude.
[0088] The above comments, annotations, and optional embodiments regarding the drive unit and the vehicle apply equally to a computer-implemented method for driving a vehicle in an autonomous mode and / or a driver assistance mode, which may be performed by a computer performing the functions of a drivable road detection unit, an image-based drivable road detection unit, a fusion unit, and / or a controller.
[0089] A drive device for automatic and / or assisted driving of a vehicle includes a first input port, a second input port, a third input port, and a visual odometry unit. The first input port is configured to receive inertial-related data measured by an inertial measurement unit. The second input port is configured to receive position-related data measured by a wheel odometry unit. The third input port is configured to receive first image data (or a set of first image data) and second image data (or a set of first image data) by an electromagnetic sensor unit, the second image data being image data prior to the first image data. The visual odometry unit is configured to receive the inertial-related data, the position-related data, the first image data, and the second image data, and is configured to output an estimated attitude of the vehicle calculated based on the inertial-related data and the position-related data, the estimated attitude including an estimated rotation and an estimated translation. The visual odometry unit includes a decision rule unit configured to generate an estimated rotation and an estimated translation calculated based on the inertial data from the inertial measurement unit and the position-related data from the wheel odometry unit, and to evaluate the estimated rotation and the estimated translation based on the estimated rotation and the estimated translation.
[0090] Optionally, the drive device may include a control unit configured to generate a drive signal for automated and / or assisted driving based on the estimated attitude.
[0091] The above comments, remarks and optional embodiments regarding the drive train and vehicle apply equally to the decision rule portion in particular.
[0092] The vehicle comprises an inertial measurement unit configured to output inertial related data, a wheel odometry unit configured to output position related data, an electromagnetic sensor unit configured to output first image data and second image data, the second image data being image data prior to the first image data, and a visual odometry unit. The visual odometry unit is configured to receive the inertial related data, the position related data, the first image data, and the second image data, and is configured to output an estimated attitude of the vehicle calculated based on the inertial related data and the position related data, the estimated attitude including an estimated rotation and an estimated translation. The visual odometry unit includes a decision rule unit configured to create an expected rotation and an expected translation calculated based on the inertial data from the inertial measurement unit and the position related data from the wheel odometry unit, and to evaluate the estimated rotation and the estimated translation based on the expected rotation and the expected translation.
[0093] Optionally, the vehicle may include a controller configured to operate the vehicle in an automated driving mode and / or an assisted driving mode based on the estimated attitude.
[0094] In optional embodiments, the expected translation includes an upper limit of the expected translation and a lower limit of the expected translation, and optionally the expected rotation includes an upper limit of the expected rotation and a lower limit of the expected rotation, and further optionally, the decision rule unit is configured to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation when the vehicle is on a straight road, or to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation and the estimated rotation is between the upper and lower limits of the expected rotation when the vehicle is on a curved road.
[0095] A computer-implemented method for operating a vehicle in an automated and / or driver assistance mode comprises the following steps. a) outputting inertial related data by an inertial measurement unit; b) outputting position related data by the wheel odometry unit; c) outputting first image data and second image data by an electromagnetic sensor unit, the second image data being image data before the first image data; and d) receiving, by a visual odometry unit, the inertial related data, the position related data, the first image data, and the second image data, and outputting an estimated attitude of the vehicle calculated based on the inertial related data and the position related data, wherein the estimated attitude includes an estimated rotation and an estimated translation. The step of outputting the estimated attitude includes generating, by a decision rule unit, a predicted rotation and a predicted translation calculated based on the inertial data from the inertial measurement unit and the position-related data from the wheel odometry unit, and evaluating the estimated rotation and the estimated translation based on the predicted rotation and the predicted translation.
[0096] Optionally, the method additionally includes driving, by a control unit, the vehicle in an automated driving mode and / or an assisted driving mode based on the estimated attitude.
[0097] The invention further relates to a computer program comprising instructions which, when said program is executed by a computer, cause said computer to carry out the steps of the method described above.
[0098] The invention also relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the above-mentioned method.
[0099] Optional embodiments of the present invention will now be described in conjunction with the accompanying drawings. [Brief description of the drawings]
[0100] [Figure 1] FIG. 1 shows a block diagram of a vehicle. [Diagram 2] FIG. 2 shows a block diagram of the drive device of the vehicle according to FIG. [Diagram 3] FIG. 3 shows a flow diagram illustrating the basic functioning of the drive device according to FIG. [Figure 4] FIG. 4 shows a flow diagram illustrating the functioning steps of the drive device according to FIG. [Diagram 5] FIG. 5 shows a flow diagram illustrating the steps of outlier removal in FIG. [Figure 6] FIG. 6 shows a flow diagram illustrating the steps of keypoint or feature point removal of FIG. [Figure 7] FIG. 7 shows a flow diagram illustrating the steps of the decision rule of FIG. [Figure 8] FIG. 8 shows a flow diagram illustrating an expansion of the steps of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0101] 1 shows a vehicle 10, which may be any type of self-propelled motor vehicle, preferably configured to travel on a road. For example, the vehicle 10 includes an engine and / or an electric motor for driving wheels of the vehicle 10.
[0102] The vehicle 10 includes an electromagnetic sensor unit 20, a localization unit 22, a controller 24, an inertial measurement unit 26, a wheel odometry unit 28, and / or a drive unit 30. The electromagnetic sensor unit 20, the localization unit 22, the controller 24, the inertial measurement unit 26, the wheel odometry unit 28, and / or the drive unit 30 are communicatively and / or electronically connected to each other to exchange data or other types of information.
[0103] The electromagnetic sensor unit 20 may include one or more cameras, LiDAR devices, and / or radar devices. The cameras may optionally be stereo cameras. The electromagnetic sensor unit 20 is capable of imaging the surroundings of the vehicle 10. In particular, the electromagnetic sensor unit 20 is configured to provide a 3D representation of the surroundings of the vehicle 10.
[0104] The electromagnetic sensor unit 20 outputs a representation of the surroundings of the vehicle 10 as image data. The electromagnetic sensor unit 20 is optionally configured to periodically and / or continuously image the surroundings of the vehicle 10. Thus, the electromagnetic sensor unit 20 is configured to output a plurality of image data, each image data taken at a different time. The first image data may be taken at a later time compared to the second image data. The electromagnetic sensor unit 20 is configured to image the same (angular) part of the surroundings in the coordinate system of the vehicle 10. The electromagnetic sensor unit 20 may be statically arranged on the vehicle 10.
[0105] The location determination unit 22 may be a device for determining the location of the vehicle 10. The location determination unit 22 may be a GPS (Global Positioning System) transceiver. The location determination unit 22 is configured to output the location of the vehicle 10 as location data.
[0106] The controller 24 includes components of the vehicle 10 that can be controlled in order to drive or navigate the vehicle 10. The controller 24 may include a steering device, a throttle device (also called an acceleration device), and a braking device for driving the vehicle 10 on drivable roads.
[0107] A steering device may be part of the vehicle 10 for adjusting the heading or direction of the vehicle 10. A throttle device may also be part of the vehicle 10 for controlling the speed of the motor or engine and thus the speed and acceleration of the vehicle 10. A brake device may be part of the vehicle 10 for slowing down the vehicle 10 by providing friction to slow down the wheels or tires of the vehicle 10. The steering device, throttle device, and breakaway device may be controlled based on signals output by the controller.
[0108] The inertial measurement unit 26 is a device for measuring forces, such as acceleration, rotation (yaw), and / or change in velocity, that the inertial measurement unit 26 is subjected to. The inertial measurement unit 26 may be fixedly mounted to the vehicle 10. In that case, the inertial measurement unit 26 is configured to measure the acceleration, rotation (yaw), and / or change in velocity of the vehicle 10. The inertial measurement unit 26 outputs the measurements as inertial related data.
[0109] The inertial measurement unit 26 may include one or more sensors, such as an accelerometer, a gyroscope (gyrometer), and / or a magnetometer. The inertial measurement unit 26 includes sensors whose measurements can be used to determine the acceleration, rotation (yaw), and / or speed changes of the vehicle 10. The gyroscope may include a micro-electromechanical system (MEMS) gyroscope.
[0110] The wheel odometry unit 28 is a device for measuring parameters indicative of the motion of the vehicle 10, such as the distance traveled by the vehicle 10, the speed of the vehicle 10, and / or the inclination of the wheels of the vehicle 10. The wheel odometry unit 28 includes one or more sensors, such as a sensor for determining the rotation of the wheels of the vehicle 10, a sensor for determining the inclination of the wheels of the vehicle 10, and / or a sensor for determining the position of the steering wheel of the vehicle 10. The wheel odometry unit 28 outputs the measurements as position-related data.
[0111] The driving unit 30 may be considered a computer or computer system including multiple processors (not shown) and a storage device 33. The driving unit 30 is configured to execute multiple algorithms that may be stored in the storage device 33. The multiple algorithms processed by the driving unit 30 enable the vehicle 10 to navigate autonomously or semi-autonomously. The driving unit 30 may be considered an autopilot and / or driver assistance system for the vehicle 10.
[0112] To this end, the drive system 30 may perform various functions that may be associated with a visual odometry unit 31 and / or a control unit 32. Each of these components may be considered as parts or functional units of the drive system 30 that execute algorithms for achieving autonomous or semi-autonomous navigation of the vehicle 10. Thus, components 31 and / or 32 may be considered as implementations of computer software.
[0113] Algorithms for parts 31 and / or 32 may be stored in a storage device 33. The drive 30 may receive inertial related data from the inertial measurement unit 26 via a first input port 34. Similarly, the drive 30 may receive position related data from the wheel odometry unit 28 via a second input port 35. The drive 30 may receive image data from the electromagnetic sensor unit 20 via a third input port 36. Additionally, the drive 30 may receive localization data from the localization unit 22 via a fourth input port (not shown).
[0114] The first input port 34, the second input port 35, and the third input port 36 can be considered as interfaces that enable communication between the inertial measurement unit 26, the wheel odometry unit 28, and the electromagnetic sensor unit 20, respectively, and the drive unit 30.
[0115] The visual odometry unit 31 may include a feature generation and association unit 31a, a feature point removal unit 31b, a world point removal unit 31c, and / or a decision rule unit 31d. Each of these components may be considered as a part or functional unit of the visual odometry unit 31. Algorithms for the components 31a, 31b, 31c, and / or 31d may be stored in the storage device 33.
[0116] The feature generation and association unit 31a receives the first image data and the second image data, and identifies or extracts a first feature point (or a first key point) in the first image data and a second feature point (or a second key point) in the second feature data. This can be done by identifying or extracting a second feature point in the second image data and subsequently searching for the same feature point in the first image data. Alternatively, the feature generation and association unit 31a searches for feature points in the first image data and feature points in the second feature data and subsequently associates corresponding feature points, i.e. feature points corresponding to the same real-world feature. Preferably, the feature generation and association unit 31a extracts multiple pairs of the first feature point and the second feature point. The feature generation and association unit 31a may include a neural network (AI) or other type of algorithm for associating the first feature point and the second feature point.
[0117] Due to the motion of the vehicle 10, the first feature point may be at a different pixel location compared to the second feature point. This pixel difference corresponds to the motion of the vehicle 10. Thus, the visual odometry unit 31 can extract the motion of the vehicle 10 and / or the pose of the vehicle 10 from the first image data and the second image data. For example, the visual odometry unit 31 includes a neural network (AI) or other type of algorithm for calculating or computing an estimated pose based on the association of multiple pairs of the first feature point and the second feature point. In particular, the visual odometry unit 31 can calculate or compute an estimated translation and an estimated rotation of the vehicle 10 that form the estimated pose of the vehicle 10.
[0118] Theoretically, information about the translation of the first feature point relative to the second feature point in the image coordinate system for each pair of the first and second feature points would result in the same estimate of the motion of the vehicle 10 and / or the pose of the vehicle 10. In practice, the estimate of the motion of the vehicle 10 and / or the pose of the vehicle 10 will be different for each pair of the first and second feature points. One way to increase the accuracy and / or location of the estimated pose is to remove first and second feature point pairs that are not indicative of the actual pose of the vehicle 10.
[0119] In the first embodiment, the feature point remover 31b calculates the pixel distance between the pixel position of the feature point and the pixel position of the second feature point (the reverse argument would also work for the first feature point). For example, the pixel position is calculated using pixel coordinates corresponding to the pixel positions of the first feature point and the second feature point. This step is repeated for each pair of the first feature point and the second feature point.
[0120] The feature point remover 31b may also use the inertial related data and the position related data to project the first projected feature point based on the second feature point. The inertial related data and the position are used to calculate rotation data indicative of the rotation of the vehicle 10 and translation data indicative of the translation of the vehicle 10. To this end, the feature point remover 31b may include a neural network (AI) or other type of algorithm capable of calculating or computing the rotation data and the translation data from the inertial related data and / or the position related data. For example, functions and / or tables link the rotation data and the translation data to the inertial related data and / or the position related data.
[0121] The rotational and translational data represent the motion of the vehicle 10 in a real world coordinate system. The rotational and translational data are then transformed into an image coordinate system by the feature point remover 31b. This can be done by a neural network (AI) or other type of algorithm. For example, the camera projection matrix K can be used. As a result, the motion of the vehicle 10 is transformed into the motion of the feature points in the image. In other words, the real world distances / vectors corresponding to the motion of the vehicle 10 are transformed into pixel distances / vectors. Thus, the motion of the feature points can be predicted. The prediction of the second feature point results in the first projected feature point.
[0122] TIFF0007594691000004.tif72166
[0123] TIFF0007594691000005.tif39165
[0124] If the coordinate system of the electromagnetic sensor unit 20 is different from the coordinate system of the vehicle 10, then the transformation between these two coordinate systems needs to be added to the above equations (eg, using an additional transformation matrix).
[0125] TIFF0007594691000006.tif21166
[0126] If this pixel distance is above a certain threshold, the associated pair of first and second feature points is removed and does not contribute to the calculation of the estimated pose. The associated pairs of first and second feature points, for which the pixel distance between the first feature point in the first projected feature point is below the threshold, are used to calculate the estimated pose. Thus, this method allows for excluding feature point pairs from the calculation of the estimated pose that are unlikely to represent the true movement of the vehicle 10. In addition, it is possible to remove first and second feature point pairs for which the association is incorrect, i.e., the first and second feature points are not associated with the same real-world feature.
[0127] The function of the feature point remover 31b is also illustrated in the flow diagram of Fig. 5. This method uses pixel distances and therefore works in the coordinate system of the image. This method of removing outliers of the first and second feature points works in the near and far fields of the electromagnetic sensor unit 20.
[0128] In the second embodiment, the world point remover 31c removes outliers in pairs of the first and second feature points in a real world coordinate system, for example the coordinate system of the vehicle 10. For this purpose, the world point remover 31c calculates the physical distance in depth between the first and second feature points. This can be done by calculating the distance between the first and second feature points as outlined above and then converting the calculated distance to a real world or physical distance. Alternatively, the world point remover 31c converts the first and second feature points to corresponding real world positions and subsequently calculates the physical distance to the real world positions. The conversion from the pixel coordinate system to the real world coordinate system may be performed by the world point remover 31c using a neural network (AI) or a deterministic algorithm. For example, the inverse of the camera projection matrix K and stereo disparity estimation can be used.
[0129] The physical depth may correspond to a z coordinate that connects two points in the real world. The z coordinate may be a coordinate in the coordinate system of the vehicle 10 that points ahead of the vehicle 10, i.e., in the driving direction if the vehicle 10 is moving straight ahead. The depth is calculated based on the translation t of the vehicle 10 when the vehicle 10 is moving straight ahead, i.e., when the rotation of the vehicle 10 is zero. WO may be parallelized with
[0130] The world point remover 31c also calculates the translation t WO The upper limit UB(t WO ) and the lower bound LB(t WO For example, the translation t WO and the upper limit UB(t WO ) and the lower bound LB(t WO ) may have a linear, exponential, and / or power law relationship. Alternatively, the world point remover 31c may use the translation t WO The upper limit is UB(t WO ) and the lower bound LB(t WO ) to the translation t WOcorresponds to the expected physical distance that the vehicle 10 is expected to travel, calculated using the position-related data from the wheel odometry unit 28. To this end, the world point remover 31c converts the position-related data into an expected physical distance, i.e., a translation t WO This includes neural networks (AI) and / or (deterministic) algorithms that link to the
[0131] The world point removal unit 31c is configured to determine whether the physical distance in depth between the first feature point and the second feature point is within an upper limit UB(t WO ) and the lower bound LB(t WO ), then we remove the pair of first and second feature points. This can be summarized by the following formula: TIFF0007594691000007.tif44161
[0132] The functionality of the world point remover 31c is also illustrated in the flow diagram of Fig. 6. This method uses physical distances and therefore works in the coordinate system of the vehicle 10 or in the real world coordinate system. This method of removing outliers of the first and second feature points works particularly well in the near field of the electromagnetic sensor unit 20.
[0133] The decision rules unit 31d is not concerned with the elimination of pairs of first and second feature points, but with the evaluation of an estimated pose calculated on a plurality of first and second feature points as described above. The decision rules unit 31d is configured to calculate or compute an expected pose which may include an expected rotation and / or an expected translation. The expected pose is determined based on inertial related data and / or position related data, whereas the estimated pose is determined based on image data. The decision rules unit 31d may include a neural network (AI) and / or a (deterministic) algorithm making it possible to calculate an expected rotation and / or an expected translation based on position related data and / or inertial related data. Preferably, the expected rotation is calculated using inertial related data and / or the expected translation is calculated using position related data.
[0134] The decision rule unit 31d determines the predicted rotation R IMU and the expected translation t WO Thus, the decision rule unit 31d is further configured to calculate or compute an upper bound UB(t WO ) and the lower bound LB(t WO ) and the upper limit for rotation, UB(R IMU ) and the lower bound LB(R IMU The upper and lower limits are calculated by the expected rotation R IMU and the expected translation t WO It may be determined using (mathematical) functions and / or tables that link
[0135] When the vehicle 10 is traveling on a straight road or when the steering angle is detected to be zero, the estimated attitude is accepted if the following equation is satisfied: t VO < U B (t WO ) t VO > LB(t WO ) Here, t VO is the estimated translation, which can be calculated based on the image data as described above.
[0136] When the vehicle 10 is traveling on a curved road or when the steering angle is detected to be not equal to zero, the estimated attitude is accepted if the following equation is satisfied: t VO < U B (t WO ) t VO > LB(t WO ) R VO < UB(R IMU ) R VO > LB(R IMU ) Here, R VO is the estimated rotation, which can be calculated based on the image data as described above.
[0137] Decision rule unit 31d thus evaluates an estimated pose rather than each individual pair of first and second feature points.The flow diagram of Figure 7 also illustrates how decision rule unit 31d operates.
[0138] The decision rule unit 31d evaluates the estimated pose determined using image data from the different sensors of the electromagnetic sensor unit 20. This means that each sensor of the electromagnetic sensor unit 20 produces image data that is used to calculate the estimated pose using the method outlined above. The decision rule unit 31d accepts the respective estimated poses from the different sensors of the electromagnetic sensor unit 20 if they satisfy the equations outlined above.
[0139] The filtering fusion module 32a combines or fuses multiple estimated poses accepted by the decision rule unit 31. Thus, the filtering fusion module 32a generates a single estimated pose that includes information of all estimated poses accepted by the decision rule unit 31. The filtering fusion module 32a may include a Kalman filter for fusing multiple estimated poses accepted by the decision rule unit 31. This method is also illustrated in the flow diagram of FIG.
[0140] The control unit 32 may include a known neural network or other type of known artificial intelligence (AI) to generate drive signals for navigating the vehicle 10 based on the estimated pose output by the visual odometry unit 31 or by the filtering fusion module 32a. The drive system may be used for autonomous and / or assisted driving (semi-autonomous driving). The control unit 32 may include output ports for outputting drive signals to the control device 24, which controls the steering device, the throttle device, and / or the braking device based on the drive signals.
[0141] A method for navigating the vehicle 10 autonomously and / or semi-autonomously will now be described in conjunction with FIG.
[0142] The electromagnetic sensor unit 20 outputs image data that the visual odometry unit 31 processes to determine an estimated pose of the vehicle 10, including an estimated translation and an estimated rotation. Details of this processing have been described above and are known in the art. To increase the accuracy and precision of the estimated pose, the visual odometry unit 31 also processes the inertial related data output by the inertial measurement unit 26 and the position related data output by the wheel odometry unit 28. The position related data and the inertial related data are used to remove feature points or key features in the image data. The removed feature points are considered not to represent the estimated pose. This means that the feature points used to determine the estimated pose are evaluated taking into account other means for determining the pose of the vehicle 10, namely the data generated by the inertial measurement unit 26 and the wheel odometry unit 28. As mentioned above, the removal of outlier feature points can be done in both the image data coordinate system and the real world coordinate system. Furthermore, localization data from the localization unit 22 can also be used to evaluate the estimated pose. For example, the localization data may be used to calculate predicted translations and rotations that may be fed into an evaluation process similar to the evaluation of an estimated attitude based on predicted translations and rotations generated using inertial-related data and position-related data.
[0143] In a subsequent step, the estimated attitude is accepted or rejected based on the expected attitude. In detail, as described above, a range within which the estimated attitude is accepted is determined taking into account the expected attitude. In contrast to the above steps, the estimated attitude itself is accepted or rejected, not the values that contribute to the calculation of the estimated attitude. Here too, localization data from the localization unit 22 may be used to accept or reject the estimated attitude, i.e. the expected attitude may be calculated based on the localization data. The acceptance or rejection of the estimated attitude can be done for each estimated attitude calculated from the different sensors of the electromagnetic sensor unit 20. In other words, an estimated attitude is calculated and accepted for each sensor of the electromagnetic sensor unit 20 if the estimated attitude is within a boundary set taking into account the expected attitude.
[0144] In a final step, the accepted estimated poses are combined using a Kalman filter. The expected rotation and translation as well as the localization data may also be inputs to the Kalman filter. A single estimated pose is the result of the Kalman filtering process.
[0145] In a final optional step, the vehicle 10 is navigated based on a single estimated pose.
Claims
1. A drive system for automated and / or assisted driving of a vehicle (10), comprising: a first input port (34) configured to receive inertial related data measured by the inertial measurement unit (26); a second input port (35) configured to receive position related data measured by the wheel odometry unit (28); a third input port (36) configured to receive the first image data and the second image data by the electromagnetic sensor unit (20), the second image data being image data prior to the first image data; a visual odometry unit (31) configured to receive the inertial related data, the position related data, the first image data, and the second image data, and configured to associate first feature points extracted from the first image data with second feature points extracted from the second image data, and output an estimated pose of the vehicle (10) calculated by evaluating the association between the first feature points and the second feature points based on the inertial related data from the inertial measurement unit (26) and the position related data from wheel odometry; A control unit (32) configured to generate a drive signal for automatic driving and / or assisted driving based on the estimated attitude.
2. an inertial measurement unit (26) configured to output inertial related data; a wheel odometry unit (28) configured to output position related data; an electromagnetic sensor unit (20) configured to output first image data and second image data, the second image data being image data prior to the first image data; a visual odometry unit (31) configured to receive the inertial related data, the position related data, the first image data, and the second image data, and configured to associate first feature points extracted from the first image data with second feature points extracted from the second image data, and output an estimated pose of the vehicle (10) calculated by evaluating the association between the first feature points and the second feature points based on the inertial related data from the inertial measurement unit (26) and the position related data from the wheel odometry unit (28); and a control unit (32) configured to drive the vehicle (10) in an automated driving mode and / or an assisted driving mode based on the estimated attitude.
3. The visual odometry unit (31) a feature generation and association unit (31 a) configured to generate the first feature points and the second feature points and associate the first feature points with the second feature points; a feature point removal unit (31 b) configured to project a first projected feature point from the second feature point by using rotation data and translation data calculated based on the inertial related data and the position related data, and configured to calculate a pixel distance between the first feature point and the first projected feature point to evaluate certainty of association between the first feature point and the second feature point.
4. 4. The drive apparatus or vehicle according to claim 3, wherein the feature point removal unit (31b) is configured to remove the association between the first feature point and the second feature point when the pixel distance is greater than a predetermined value.
5. 5. The drive or vehicle according to claim 1, wherein the visual odometry unit (31) further comprises a world point removal unit (31c) configured to compare a physical distance in depth between the first feature point and the second feature point with an expected physical distance in depth, the expected physical distance being calculated based on the position-related data from the wheel odometry unit (28).
6. 6. The drive or vehicle of claim 5, wherein the world point removal unit (31c) is configured to calculate an upper limit of the expected physical distance and a lower limit of the expected physical distance, and to accept the association between the first feature point and the second feature point if the physical distance at the depth is between the upper limit of the expected physical distance and the lower limit of the expected physical distance.
7. the estimated pose includes an estimated rotation and an estimated translation; 7. The drive unit or vehicle according to claim 1, wherein the visual odometry unit (31) includes a decision rule unit (31 d) configured to generate an estimated rotation and an estimated translation calculated based on the inertial related data from the inertial measurement unit (26) and the position related data from the wheel odometry unit (28), and configured to evaluate the estimated rotation and the estimated translation based on the estimated rotation and the estimated translation.
8. the expected translation includes an upper bound of the expected translation and a lower bound of the expected translation; the expected rotation includes an upper limit of the expected rotation and a lower limit of the expected rotation; 8. The drive unit or vehicle of claim 7, wherein the decision rule unit is configured to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation when the vehicle is on a straight road, or to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation and the estimated rotation is between the upper and lower limits of the expected rotation when the vehicle is on a curved road.
9. 1. A computer-implemented method for operating a vehicle (10) in an automated and / or driver-assisted mode, comprising: outputting inertial related data by an inertial measurement unit (26); outputting position related data by a wheel odometry unit (28); outputting first image data and second image data by an electromagnetic sensor unit (20), the second image data being image data prior to the first image data; receiving, by a visual odometry unit (31), the inertial related data, the position related data, the first image data, and the second image data, associating first feature points extracted from the first image data with second feature points extracted from the second image data, and outputting an estimated pose of the vehicle (10) calculated by evaluating the association between the first feature points and the second feature points based on the inertial related data from the inertial measurement unit (26) and the position related data from wheel odometry; and driving, by a control unit (32), the vehicle (10) in an automated driving mode and / or an assisted driving mode based on the estimated attitude.
10. A drive system for automated and / or assisted driving of a vehicle (10), comprising: a first input port (34) configured to receive inertial related data measured by the inertial measurement unit (26); a second input port (35) configured to receive position related data measured by the wheel odometry unit (28); a third input port (36) configured to receive the first image data by the electromagnetic sensor unit (20) and second image data that is image data prior to the first image data, the second image data being image data prior to the first image data; a visual odometry unit (31) configured to receive the inertial related data, the position related data, the first image data, and the second image data, and configured to output an estimated attitude of the vehicle (10) calculated based on the inertial related data and the position related data, the estimated attitude including an estimated rotation and an estimated translation; and a control unit (32) configured to generate a drive signal for automatic driving and / or assisted driving based on the estimated attitude, The visual odometry unit (31) includes a decision rule unit (31 d) configured to generate an estimated rotation and an estimated translation calculated based on the inertial data from the inertial measurement unit (26) and the position-related data from the wheel odometry unit (28), and to evaluate the estimated rotation and the estimated translation based on the estimated rotation and the estimated translation.
11. an inertial measurement unit (26) configured to output inertial related data; a wheel odometry unit (28) configured to output position related data; an electromagnetic sensor unit (20) configured to output first image data and second image data, the second image data being image data prior to the first image data; a visual odometry unit (31) configured to receive the inertial related data, the position related data, the first image data, and the second image data, and configured to output an estimated attitude of the vehicle (10) calculated based on the inertial related data and the position related data, the estimated attitude including an estimated rotation and an estimated translation; a control unit (32) configured to drive the vehicle (10) in an automated driving mode and / or an assisted driving mode based on the estimated attitude; The visual odometry unit (31) includes a decision rule unit (31 d) configured to generate an estimated rotation and an estimated translation calculated based on the inertial data from the inertial measurement unit (26) and the position-related data from the wheel odometry unit (28), and to evaluate the estimated rotation and the estimated translation based on the estimated rotation and the estimated translation.
12. the expected translation includes an upper bound of the expected translation and a lower bound of the expected translation; the expected rotation includes an upper limit of the expected rotation and a lower limit of the expected rotation; 12. The drive unit or vehicle of claim 10 or 11, wherein the decision rule unit (31 d) is configured to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation when the vehicle (10) is on a straight road, or to accept the estimated attitude if the estimated translation is between the upper and lower limits of the expected translation and the estimated rotation is between the upper and lower limits of the expected rotation when the vehicle (10) is on a curved road.
13. 1. A computer-implemented method for operating a vehicle (10) in an automated and / or driver-assisted mode, comprising: outputting inertial related data by an inertial measurement unit (26); outputting position related data by a wheel odometry unit (28); outputting first image data and second image data by an electromagnetic sensor unit (20), the second image data being image data prior to the first image data; receiving the inertial-related data, the position-related data, the first image data, and the second image data by a visual odometry unit (31), and outputting an estimated attitude of the vehicle (10) calculated based on the inertial-related data and the position-related data, the estimated attitude including an estimated rotation and an estimated translation; and driving the vehicle (10) in an automatic driving mode and / or an assisted driving mode based on the estimated attitude by a control unit (32). the step of outputting the estimated attitude includes generating, by a decision rule unit (31 d), a predicted rotation and a predicted translation calculated based on the inertial data from the inertial measurement unit (26) and the position-related data from the wheel odometry unit (28), and evaluating the estimated rotation and the estimated translation based on the predicted rotation and the predicted translation.
Citation Information
Patent Citations
Drive support device
JP2015214282A
Vision-aided inertial navigation system for ground vehicle localization
US20190368879A1
Validation of global navigation satellite system location data with other sensor data
US20200081134A1
Method and apparatus for outputting pose information
US20200103664A1
Local location computation device and local location computation method
WO2015125299A1
