Visual ranging method for operating movable device

By combining Superpoint, SOLD2 and attn/GNN, the feature point detection problem of stereoscopic video ranging method under low contrast and low light conditions is solved, and high-precision position measurement in harsh environments is achieved, ensuring that the system is in harsh environments high resolution pose data under .

CN119022876APending Publication Date: 2024-11-26FORD GLOBAL TECH LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410637086.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-26
Filing Date
2024-05-22
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Under low contrast and low light conditions, the existing stereoscopic video ranging method is difficult to accurately and repetitively determine feature points, affecting the system's position measurement accuracy in harsh environments.

Method used

Using the feature point detector Superpoint and the occlusion-perceptual line feature detection algorithm SOLD2, combined with the attention/graph neural network (attn/GNN) and the differentiable optimal transport processor, the point features and lines are determined through self-supervised machine learning. Features, enhance feature point detection capabilities under low contrast and low light conditions, and determine the 3D stereo camera posture through perspective n-point and line algorithm.

Benefits of technology

Improve the reliability and repeatability of feature point detection in low contrast and low light conditions, ensure the accuracy of high-resolution position and orientation data in harsh environments, and support the system's position resolution within 10cm range and a rotation resolution of one degree.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119022876A_ABST
    Figure CN119022876A_ABST
Patent Text Reader

Abstract

The invention provides a visual ranging method for operating a movable device. First side stereoscopic images of a first pair of stereoscopic images and a second pair of stereoscopic images acquired at a first time step and a second time step respectively are selected, and point features and line features in the masked first first side stereoscopic image and second first side stereoscopic image are determined. Matching point features in the masked first and second first side stereoscopic images are determined using a first attention / graph neural network (attn / GNN) according to the key points determined by the line features. Three-dimensional (3D) positions in the scene are determined based on matching point features included in a first first side stereoscopic image and point features determined in a first second side stereoscopic image of the first pair of stereoscopic images and the second pair of stereoscopic images and a perspective n-point and line algorithm is determined thereon to determine a 3D stereoscopic camera pose.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to visual odometry for operating a mobile device. Background Art

[0002] The computer may operate a system and / or device, including a vehicle, a robot, a drone, and / or an object tracking system. Data, including images, may be acquired by sensors and processed using a computer to determine the position of the system relative to objects in an environment surrounding the system. The computer may use the position data to determine a trajectory for moving the system in the environment. The computer may then determine control data to be transmitted to system components to control the system components to move the system based on the determined trajectory. Summary of the invention

[0003] Systems (including vehicles, robots, drones, etc.) can be operated by acquiring sensor data about the environment surrounding the system and processing the sensor data to determine a path on which to operate the system or a portion of the system. The sensor data can be processed to determine the location of the real-world coordinates of the system in the environment. The system can then be operated using the determined location. For example, a mobile robot can determine its location in a warehouse or workplace. A vehicle can determine its location on a road. The mobile robot or vehicle can use the determined location to determine a path on which to operate to move the mobile robot or vehicle from the determined location to a planned location.

[0004] A movable system may rely on sensor data acquired by sensors included in the system to determine the position of the movable system and / or a movable element of the system (e.g., a vehicle, a robotic arm, etc.) relative to the environment. Some sensors (such as lidar or radar) return mileage or distance data directly. Other sensors (such as visible light or infrared video sensors) may rely on one or more two-dimensional (2D) images to determine the distance to a location in the environment. Stereoscopic vision relies on two cameras arranged to observe overlapping fields of view. The change in position of the same point in two images acquired from a pair of stereo images (called stereo disparity) can be used to determine the distance of the point to a point within the stereo camera.

[0005] Stereo video odometry is a technique for determining the position of a vehicle relative to the environment using stereo camera data. Stereo video odometry matches feature points in a pair of stereo images acquired by a stereo camera and determines stereo disparity based on the matched feature points. Feature points are pixel locations in an image that are determined by local configurations of pixel intensities, such as edges and corners. Stereo disparity is the difference in pixel positions of two feature points corresponding to the same location in a scene included in two stereo images. Stereo disparity can be combined with intrinsic camera data including image scale and focal length to determine the distance between the stereo camera and a location in the scene.

[0006] The ability of stereo video ranging techniques to determine the distance to a location in a scene depends on the ability of the system to accurately and repeatably determine feature points corresponding to the same location in two stereo images. Accurate and repeatable determination of feature points means determining the same feature points at the same location through repeated trials. Low contrast and low light conditions (such as bad weather and darkness) can make it difficult to achieve accurate stereo video ranging. In addition to point features, the techniques for stereo video ranging described herein also use line features to increase the system's ability to determine accurate and repeatable feature points in low contrast and low light conditions.

[0007] A computing device in the system may operate the system using stereo video odometry. Stereo video odometry can determine the position of a stereo camera relative to a scene viewed by the stereo camera by determining the distance to a point in the scene. The distance to a point in the scene can be compared to map data such as The system position and orientation in real-world coordinates are determined by combining data from a GPS (Global Positioning System) (GPS) data, a GPS map, a GPS (Global Navigation Satellite System) (GNSS) data, and / or an Inertial Measurement Unit (IMU) data to determine the system position and orientation in real-world coordinates. An IMU uses an accelerometer to determine the system position and orientation in real-world coordinates. Real-world coordinates may be defined in terms of global x, y, and z position coordinates relative to latitude, longitude, and altitude, and roll, pitch, and yaw rotation coordinates relative to the x-, y-, and z-axes, respectively.

[0008] Map data and GPS data include positions in real-world coordinates with a resolution measured in units of one or more meters. IMU data may include higher resolution position data, but only relative measurements are returned rather than real-world coordinates. An operating system may require a position resolution in the range of 10 centimeters (cm) or less and a rotation resolution of one degree or less. Stereo video odometry can be combined with map data, GPS data, and IMU data to provide high-resolution position and orientation data, which is defined as a position resolution in the range of less than 10 cm and a rotation resolution of one degree or less.

[0009] The high-resolution position and orientation, defined herein as high-resolution pose data, can be used by a computing device in a system to operate a system, such as a vehicle, a robotic arm, and the like. One technique for operating a system includes determining a path polynomial according to which the system can travel from a current position to a planned position. The path polynomial can be a polynomial function determined to maintain lower and upper limits for lateral and longitudinal accelerations as the system travels according to the path polynomial. The computing device in the system can control system actuators to cause the system to travel along the path polynomial. In an example where the system is a vehicle, the computing device can control vehicle propulsion, steering, and braking to cause the vehicle to travel along the path polynomial.

[0010] A method is disclosed herein, the method comprising selecting a first first side stereo image and a second first side stereo image of a first pair of stereo images and a second pair of stereo images acquired at a first time step and a second time step, respectively, and masking the first first side stereo image and the second first side stereo image. Point features in the masked first first side stereo image and the second first side stereo image may be determined, and line features in the masked first first side stereo image and the second first side stereo image may be determined. Matching point features in the masked first first side stereo image and the second first side stereo image may be determined using a first attention / graph neural network (attn / GNN) based on key points determined from the line features. Matching line features may be determined using a second attn / GNN in the masked first first side stereo image and the second first side stereo image based on key points determined from the line features. A three-dimensional (3D) position in a scene may be determined by determining a stereo disparity based on the matched point features included in the first first side stereo image and the point features determined in the first second side stereo images of the first pair of stereo images and the second pair of stereo images. A 3D stereo camera pose may be determined by determining a perspective n-point and line algorithm on the 3D position.

[0011] The first first side stereo image and the second first side stereo image may be masked by determining a first segmented image and a second segmented image of the first first side stereo image and the second first side stereo image by an image segmenter and masking the first first side stereo image and the second first side stereo image based on the first segmented image and the second segmented image, respectively. Masking the first first side stereo image and the second first side stereo image may remove dynamic objects and the sky. The point features in the masked first first side stereo image and the second first side stereo image may be determined by performing self-supervised machine learning using the first attn / GNN followed by a differentiable optimal transport processor. The line features in the masked first first side stereo image and the second first side stereo image may be determined by performing self-supervised machine learning using the second attn / GNN followed by a differentiable optimal transport processor. The point features may be determined based on a Superpoint algorithm. The line features may be determined based on a SOLD2 algorithm. The key points may be determined by sampling the line features determined in the masked first first side stereo image and the second first side stereo image. Point features in the first second side stereo images of the first pair of stereo images and the second pair of stereo images can be determined based on correlating pixel neighborhoods around the point features in the first left stereo image. Stereo disparity can be determined by determining the distance between the point features in the first second side stereo image and the masked first first side stereo image and based on one or more of the baseline, focal length and pixel scale between stereo cameras. The 3D stereo camera pose can be determined using one or more of global navigation satellite system (GNSS) data or inertial measurement units (IMUs) to determine the 3D vehicle pose based on an extended Kalman filter. A path polynomial can be determined based on the 3D vehicle pose, and a vehicle can be operated based on the path polynomial. Superpoint can be a feature point detector based on a neural network. SOLD2 can be an occlusion-aware line feature detection and description algorithm.

[0012] A computer-readable medium is also disclosed, the computer-readable medium storing program instructions for executing some or all of the above method steps. A computer is also disclosed, the computer being programmed to execute some or all of the above method steps, the computer comprising a computer device, the computer device being programmed to: select a first first side stereo image and a second first side stereo image of a first pair of stereo images and a second pair of stereo images acquired at a first time step and a second time step, respectively; and mask the first first side stereo image and the second first side stereo image. Point features in the masked first first side stereo image and the second first side stereo image may be determined, and line features in the masked first first side stereo image and the second first side stereo image may be determined. Matching point features in the masked first first side stereo image and the second first side stereo image may be determined using a first attention / graph neural network (attn / GNN) based on key points determined according to the line features. Matching line features may be determined using a second attn / GNN in the masked first first side stereo image and the second first side stereo image based on key points determined according to the line features. A three-dimensional (3D) position in a scene may be determined by determining a stereo disparity based on the matched point features included in the first first side stereo image and the point features determined in the first second side stereo images of the first pair of stereo images and the second pair of stereo images. A 3D stereo camera pose may be determined by determining a perspective n-point and line algorithm on the 3D position.

[0013] The instructions may include further instructions for performing the following operations: masking the first first side stereo image and the second first side stereo image by determining a first segmented image and a second segmented image of the first first side stereo image and the second first side stereo image by an image segmenter and masking the first first side stereo image and the second first side stereo image based on the first segmented image and the second segmented image, respectively. Masking the first first side stereo image and the second first side stereo image may remove dynamic objects and the sky. The point features in the masked first first side stereo image and the second first side stereo image may be determined by performing self-supervised machine learning using the first attn / GNN followed by a differentiable optimal transport processor. The line features in the masked first first side stereo image and the second first side stereo image may be determined by performing self-supervised machine learning using the second attn / GNN followed by a differentiable optimal transport processor. The point features may be determined based on a Superpoint algorithm. The line features may be determined based on a SOLD2 algorithm. The key points may be determined by sampling the line features determined in the masked first first side stereo image and the second first side stereo image. Point features in the first second side stereo images of the first pair of stereo images and the second pair of stereo images can be determined based on correlating pixel neighborhoods around the point features in the first left stereo image. Stereo disparity can be determined by determining the distance between the point features in the first second side stereo image and the masked first first side stereo image and based on one or more of the baseline, focal length and pixel scale between stereo cameras. The 3D stereo camera pose can be determined using one or more of global navigation satellite system (GNSS) data or inertial measurement units (IMUs) to determine the 3D vehicle pose based on an extended Kalman filter. A path polynomial can be determined based on the 3D vehicle pose, and a vehicle can be operated based on the path polynomial. Superpoint can be a feature point detector based on a neural network. SOLD2 can be an occlusion-aware line feature detection and description algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of an example vehicle system.

[0015] Figure 2 is an illustration of an example image of a scene.

[0016] Figure 3 is an illustration of an example stereo camera.

[0017] Figure 4 is an illustration of an example scene including feature points.

[0018] Figure 5is an illustration of an example scene including feature lines.

[0019] Figure 6 is an illustration of an example segmented image of a scene.

[0020] Figure 7 is a diagram of an example stereo video ranging system.

[0021] Figure 8 is an illustration of an example image of a scene including point matching and line matching.

[0022] Fig. 9 is a diagram of an example positioning system.

[0023] Fig.10 is a flow chart of an example stereo vision ranging system.

[0024] Fig.11 is a flow chart of an example process for positioning a system to operate a vehicle. DETAILED DESCRIPTION

[0025] Figure 1 1 is an illustration of a vehicle computing system 100. The vehicle computing system 100 includes a vehicle 110, a computing device 115 included in the vehicle 110, and a server computer 120 remote from the vehicle 110. The computing device 115 of one or more vehicles 110 may receive data regarding the operation of the vehicle 110 from sensors 116. The computing device 115 may operate the vehicle 110 based on the data received from the sensors 116 and / or the data received from the remote server computer 120. The server computer 120 may communicate with the vehicle 110 via a network 130.

[0026] The computing device 115 includes a processor and memory such as are known. In addition, the memory includes one or more forms of computer-readable media and stores instructions that can be executed by the processor to perform various operations including as disclosed herein. For example, the computing device 115 may include programming to operate one or more of vehicle braking, propulsion (i.e., controlling the acceleration of the vehicle 110 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior lights and / or exterior lights, etc., and determine whether and when the computing device 115 (rather than a human operator) controls such operations.

[0027] The computing device 115 may include, or be communicatively coupled to (i.e., via a vehicle communication bus as further described below) more than one computing device (i.e., controllers included in the vehicle 110 for monitoring and / or controlling various vehicle components, etc. (i.e., propulsion controller 112, brake controller 113, steering controller 114, etc.)). The computing device 115 is typically arranged to communicate over a vehicle communication network (i.e., including a bus in the vehicle 110, such as a controller area network (CAN), etc.); the vehicle 110 network may additionally or alternatively include, such as known wired or wireless communication mechanisms, i.e., Ethernet or other communication protocols.

[0028] The computing device 115 may transmit and / or receive messages to and from various devices in the vehicle (i.e., controllers, actuators, sensors (including sensor 116), etc.) via the vehicle network. Alternatively or additionally, where the computing device 115 actually includes multiple devices, the vehicle communication network may be used to communicate between the devices represented in this disclosure as the computing device 115. Furthermore, as mentioned below, various controllers or sensing elements (such as sensor 116) may provide data to the computing device 115 via the vehicle communication network.

[0029] Additionally, computing device 115 may be configured to communicate with remote server computer 120 (i.e., cloud server) via network 130 through vehicle-to-infrastructure (V2X) interface 111, as described below, which includes hardware, firmware, and software that permit computing device 115 to communicate with remote server computer 120 (i.e., cloud server) via a vehicle-to-infrastructure (V2X) interface 111, such as a wireless Internet connection. The V2X interface 111 may include a network 130 configured to communicate with the remote server computer 120 using various wired and / or wireless networking technologies (i.e., cellular, The computing device 115 may be configured to communicate with other vehicles 110 through a V2X (vehicle to the outside world) interface 111 using a vehicle-to-vehicle (V2V) network formed between neighboring vehicles 110 on the basis of a mobile ad hoc network or through an infrastructure-based network (i.e., based on cellular communication (C-V2X) wireless communication cellular, dedicated short-range communication (DSRC) and / or similar communications). The computing device 115 also includes a non-volatile memory such as known. The computing device 115 can record data by storing the data in the non-volatile memory for later retrieval and transmission to the server computer 120 or the user mobile device 160 via the vehicle communication network and the vehicle to infrastructure (V2X) interface 111.

[0030] As already mentioned, programming for operating one or more vehicle 110 components (i.e., braking, steering, propulsion, etc.) without human operator intervention is typically included in instructions stored in memory and executable by a processor of the computing device 115. Using data received in the computing device 115 (i.e., sensor data from sensors 116, data from server computer 120, etc.), the computing device 115 can make various determinations and / or control various vehicle 110 components and / or operations. For example, the computing device 115 may include programming to adjust vehicle 110 operating behavior (i.e., the physical manifestation of vehicle 110 operation), such as speed, acceleration, deceleration, steering, etc., as well as strategic behavior (i.e., controlling operating behavior in a manner that is generally intended to achieve efficient traversal of a route), such as the distance between vehicles and / or the amount of time between vehicles, lane changes, minimum gaps between vehicles, left turn crossing path minimums, arrival times at specific locations, and the minimum time from arrival at an intersection to crossing an intersection (without a signal light).

[0031] The term controller as used herein includes a computing device that is typically programmed to monitor and / or control a particular vehicle subsystem. Examples include a propulsion controller 112, a brake controller 113, and a steering controller 114. The controller may be an electronic control unit (ECU) such as is known, possibly including additional programming as described herein. The controller may be communicatively connected to a computing device 115 and receive instructions from the computing device to actuate a subsystem in accordance with the instructions. For example, a brake controller 113 may receive instructions from a computing device 115 to operate the brakes of the vehicle 110.

[0032] One or more controllers 112, 113, 114 for the vehicle 110 may include known electronic control units (ECUs) and the like, including, as non-limiting examples, one or more propulsion controllers 112, one or more brake controllers 113, and one or more steering controllers 114. Each of the controllers 112, 113, 114 may include a corresponding processor and memory and one or more actuators. The controllers 112, 113, 114 may be programmed and connected to a vehicle 110 communication bus, such as a controller area network (CAN) bus or a local interconnect network (LIN) bus, to receive instructions from a computing device 115 and control actuators based on the instructions.

[0033] Sensors 116 may include a variety of devices known to provide data via a vehicle communication bus. For example, a radar affixed to a front bumper (not shown) of vehicle 110 may provide a distance from vehicle 110 to the next vehicle in front of vehicle 110, or a global positioning system (GPS) sensor disposed in vehicle 110 may provide geographic coordinates of vehicle 110. For example, the distance provided by radar and / or other sensors 116 and / or the geographic coordinates provided by a GPS sensor may be used by computing device 115 to operate vehicle 110 autonomously or semi-autonomously.

[0034] The vehicle 110 is typically a ground-based vehicle 110 (i.e., a passenger car, a light truck, etc.) capable of autonomous and / or semi-autonomous operation and having three or more wheels. The vehicle 110 includes one or more sensors 116, a V2X interface 111, a computing device 115, and one or more controllers 112, 113, 114. The sensor 116 may collect data related to the vehicle 110 and the operating environment of the vehicle 110. By way of example and not limitation, the sensor 116 may include, i.e., an altimeter, a camera, a laser radar (LIDAR), a radar, an ultrasonic sensor, an infrared sensor, a pressure sensor, an accelerometer, a gyroscope, a temperature sensor, a pressure sensor, a Hall sensor, an optical sensor, a voltage sensor, a current sensor, a mechanical sensor (such as a switch), etc. The sensor 116 may be used to sense the operating environment of the vehicle 110, i.e., the sensor 116 may detect phenomena such as weather conditions (rainfall, outside temperature, etc.), road slope, road position (i.e., using road edges, lane markings, etc.), or the position of a target object (such as a neighboring vehicle 110). The sensors 116 may also be used to collect data, including dynamic vehicle 110 data related to the operation of the vehicle 110 , such as speed, yaw rate, steering angle, engine speed, brake pressure, oil pressure, power levels applied to controllers 112 , 113 , 114 in the vehicle 110 , connectivity between components, and accurate and timely performance of components of the vehicle 110 .

[0035] The server computer 120 generally has features in common with the V2X interface 111 and computing device 115 of the vehicle 110, such as a computer processor and memory and configuration for communicating via the network 130, and therefore these features will not be further described to reduce redundancy. The server computer 120 can be used to develop and train software that can be transmitted to the computing device 115 in the vehicle 110.

[0036] Figure 22 is an illustration of an image 200 of a scene 202. The image 200 includes a road 204, with a vehicle 206 traveling on the road 204. For example, the image 200 may be acquired by a camera included in a pair of stereo cameras included in the vehicle 110. The computing device 115 included in the vehicle 110 may input the image 200 as one of a pair of stereo images into a stereo video odometry system included in the computing device 115 to determine a high-resolution pose of the vehicle 110 relative to the scene 202. The high-resolution pose may be used by the computing device 115 to determine a path polynomial to operate the vehicle 110 through the scene 202.

[0037] Figure 3 300. The stereo camera 300 includes a left camera 302 and a right camera 304, both of which are configured to observe a scene 306. The left camera 302 and the right camera 304 each include an optical center 308, 310, respectively. The left camera 302 and the right camera 304 can be configured to observe the scene 306 along parallel optical axes 324, 326, respectively. The optical axes 324, 326 and the sensor plane 318 can be configured to be perpendicular, thereby forming a Z axis and an X axis. The Y axis is in a direction perpendicular to the page. The sensor plane 318 includes a left image sensor 314 and a right image sensor 316 included in the left camera 302 and the right camera 304, respectively. The optical centers 308, 310 are separated by a baseline distance b. The left sensor 314 and the right sensor 316 are located in the sensor plane 318 at focal lengths f320, 322 from the optical centers 308, 310, respectively.

[0038] The stereo camera 300 can be used to determine the depth z of a point P in the scene 306 from a plane defined by the optical centers 308, 310 of the left camera 302 and the right camera 304, respectively. The point P is a distance x from the right optical axis 326. p 328. Assuming the pinhole optical model of the left stereo camera 302 and the right stereo camera 304, the image of point P is respectively at point x l and x r The value x is projected onto the left sensor 314 and the right sensor 316. l and x r Indicates the distance from the right optical axis 324 and the left optical axis 326. The value (x l -x r )=d is called stereo disparity d. Stereo disparity d can be converted from pixel distance to real world distance by multiplying by a scaling factor s. Depth z can be determined from stereo disparity d, focal length f, pixel scale s, and baseline b by the following equation:

[0039]

[0040] In this example, the stereoscopic image pair is displaced from each other only in the x-direction along the X-axis, which means that the stereo disparity d is determined only along the X-axis.

[0041] Figure 4 4 is an illustration of an image 400 of a scene 402. Image 400 may be one of a pair of stereo images acquired by stereo camera 300. Image 400 may be used as one of a pair of stereo images to determine a depth z of a point P (a small dot) in scene 402, as described above with respect to Figure 3 As described. The image 400 shown in the figure is acquired under foggy conditions 406, resulting in low contrast conditions in the image 400. Low contrast conditions pose a challenge to stereo visual odometry because point feature detectors rely on local image contrast to generate feature points P. The stereo video odometry technique described herein uses a neural network-based feature point P detector to reliably and repeatedly obtain feature points P in low contrast and low light images 400.

[0042] A neural network-based feature point P detector is the Superpoint algorithm. Superpoint is a neural network-based feature point P detector described in DeTone, D., Malisewicz, T., and Rabinovich, A., "Superpoint: Self-supervised interest point detection and description", Proceedings of the IEEE Symposium on Computer Vision and Pattern Recognition, 2018. Superpoint uses a convolutional neural network for an encoder to generate interest points from an input image, and uses two separate decoder heads to process the interest points to generate interest points and descriptors of the interest points. Although Superpoint has enhanced performance in generating feature points P, low contrast and low light conditions may still cause Superpoint to perform unreliably. To remedy this situation, the stereo video technique discussed in this article detects line features in addition to feature points P.

[0043] Figure 5 500 is an illustration of an image 500 of a scene 502. Image 500 may be one of a pair of stereo images acquired by stereo camera 300. Similar to image 400, image 500 includes low contrast due to foggy conditions. Image 500 may be acquired simultaneously with image 400, above image 400, or at a different time. In this example, the time step between acquiring image 400 and image 500 may be small, so that despite the motion of stereo camera 300 used to acquire images 400 and 500, scenes 402 and 502 included in images 400 and 500, respectively, will not change significantly. Image 500 may be used to enhance feature points P determined in image 400 by determining line features 504.

[0044] The SOLD2 algorithm can be used to determine line features 504 in low contrast or low light images. SOLD2 is an occlusion-aware line feature detection and description algorithm. The stereo video ranging technique described herein uses only the line detection portion of SOLD2 and ignores the line matching portion of the algorithm. SOLD2 is configured to perform well in low contrast and low light conditions by training the algorithm on synthetic data generated to simulate low contrast and low light conditions, and SOLD2 is described in Pautrat, R., Lin, JT, Larsson, V., Oswald, MR, and Pollefeys, M., "SOLD2: Self supervised occlusion-aware line description and detection", IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021.

[0045] Figure 6 200 is an illustration of a segmented image 600 based on an image 200 of a scene 202. The segmented image 600 is generated by receiving an input image 200 at an image segmentor. For example, the image segmentor is a software program executable on a computing device 115. The segmented image 600 can be determined by receiving the input image 200 at an image segmentor, which groups pixels into continuous regions based on edges and pixel values. For example, the segmentor can use an edge detection routine (such as a Canny edge detector) to determine the edges. The average color value and variance of the pixels within the pixel region bounded by the edge can be determined. Adjacent regions with average values ​​within a user-selected variance can be merged to determine the segmented image 600 by determining image segments that group regions of continuous pixels by color similarity.

[0046] An example of image processing software that performs image segmentation is the IntelligentScissorsMB routine in the OpenCV image processing library located at OpenCV.org as of the filing date of this application. IntelligentScissorsMB uses a watershed algorithm (a known technique for image segmentation) to group pixels by color within edges located using a Canny edge detector. In some examples, an image may be segmented using a trained neural network to generate a segmented image 600 in a manner similar to the OpenCV image processing library. Based on color, size, shape, and position, the segmentor may identify individual segments or regions within the segmented image as specific types of objects. For example, the image segmentor may identify regions as dynamic objects 602, sky 604, 606, 608, or background 610 regions, where the dynamic object 602 is, for example, a vehicle. The segmented image 600 may be used as a mask to guide stereo video odometry processing, as described below with respect to Figure 7 By masking out the dynamic object 602 and the sky 604, 606, 608, processing based on the segmented image 600 may allow the stereo video ranging system to generate more reliable and repeatable point features P.

[0047] Figure 7 700 is a block diagram of a stereo video ranging system 700. The stereo video ranging system 700 receives a left stereo image 704 and a right stereo image 706 from a stereo camera 702 as input and outputs a 3D camera pose estimate. The stereo video ranging system 700 operates by selecting a pair of right stereo images or left stereo images that are adjacent in time. For the convenience of explanation herein, a pair of left stereo images that are adjacent in time are described as selected. That is, the left image is an image that can be referred to as a "first side" image. For generality, reference may be made to a first side image and a second side image, where the first side is one of the right or left, and the second side is the other of the right or left. For ease of explanation, the left image is the first side image and the right image is the second side image, and the examples herein. Without loss of generality, a pair of right stereo images that are adjacent in time may also be used, that is, the right image may be the first side image and the left image may be the second side image. The stereo video ranging system 700 inputs two left stereo images 704 acquired at different times separated by a smaller time step. The time step may be a single frame time, for example, 16.7 milliseconds for 60 frames per second video. The stereo video ranging system 700 processes the two left stereo images 704 to determine feature points P that appear in the two left stereo images 704 to increase the reliability and repeatability of the detected feature points P.

[0048] The segmenter 708 may receive the two left stereo images 704 to segment the two consecutive left stereo images 704 to remove dynamic objects, such as vehicles and the sky, as described above with respect to Figure 6Then, two consecutive left stereo images 704 are received at both the point feature processor 710 and the line feature processor 712. Figure 4 As described, the point feature processor 710 uses the Superpoint algorithm to detect feature points P in two consecutive frames of the left stereo image 704. In parallel with the point feature processor 710, the line feature processor 712 detects the line features 504 in the consecutive left stereo images 704. The detected line features 504 are sampled and the sampled points are processed using the Superpoint algorithm to obtain key point features. After the line feature processor 712, the key point detector 714 detects key points that are point features appearing on the line features 504.

[0049] After point feature P detection and line / point feature detection, point feature P and keypoint features are received by self-supervised machine learning based on attention / graph neural network (attn / GNN) 716, 718, respectively. attn / GNN 716, 718 is based on Superglue, i.e., point matching graph neural network described in Sarlin, P.E., DeTone, D., Malisewicz, T., and Rabinovich, A., 2020, "Superglue: Learning feature matching with graph neural networks", IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2020. Processing point feature P and keypoint features with attn / GNN 716, 718 includes processing the output with a differentiable optimal transport processor 720.

[0050] The attn / GNN 716, 718 and the differentiable optimal transport processor 720 are combined to match two sets of point features P and keypoint features by jointly finding correspondences and rejecting unmatched points by solving a differentiable optimal transport problem, the computational cost of which is predicted by the attn / GNN 716, 718. In this context, matching means that a first feature point in the first image determined by a pattern in the pixel values ​​of surrounding pixels can be moved to the position of a second feature point in the second image determined by the same pattern of surrounding pixel values ​​by a single transformation that moves similar matching feature points in the first and second images. The attn / GNN 716, 718 processor can determine the pattern of feature points in the first and second images, and determine a transformation that moves the pattern of feature points in the first image to cover the pattern of feature points in the second image. In examples where there is more than one transformation to move the pattern of feature points, the optimal transport process can determine the transformation with the minimum computational cost (e.g., most efficient) to move feature points from the first image to feature points from the second image. Pairs of feature points that are moved to cover by a transformation are called matching feature points.

[0051] The optimal transport process includes an algorithm for minimizing the computational cost of transporting items in the presence of multiple item sources and multiple destinations. In this example, moving point features P and keypoint features between adjacent images is a transportation problem. The combination of attn / GNN 716, 718 and the differentiable optimal transport processor 720 can perform robust line matching in poorly textured image data (such as low contrast or low light level images).

[0052] The attn / GNN 716, 718 generates visual appearance descriptors and considers the position of the keypoints relative to other keypoints in the same image and in adjacent images to perform robust feature matching. The keypoint features serve as anchors that enable lines to have positional constraints when passing through the attn / GNN 716, 718. These constraints enable robust line matching in repetitive scenes that contain little or no texture, such as repetitive windows in a building, which results in no matches or erroneous matches in existing solutions. Figure 8 The result of processing two adjacent frames of the left stereoscopic image 704 for both the point feature P and the line feature 504 is shown in FIG.

[0053] Return to Figure 7, the point feature P matches and line feature 504 matches of the temporally adjacent left stereo images 704 are received by the 3D processing 722. The point feature P and line feature 504 matches between the temporally adjacent left stereo images 704 are converted into 3D points by triangulation using the stereo disparity d. The stereo disparity d may be obtained from the right stereo image 706 received from the stereo camera 702. The point feature P and the key point features received from the differentiable optimal transport processor 720 may be used to determine the position of the point feature P included in the right stereo image 706. Then, the stereo disparity d may be used to determine the 3D position of the point feature P and the line feature 504 in the scene 202 using the above equation (1).

[0054] By selecting a point feature P in one of the left stereo images that matches a point feature P in an adjacent left stereo image, the point features P and the line features 504 of the adjacent frames of the left stereo data 704 can be reduced to a set of point features P. In this way, point features P that appear only in two adjacent left stereo images and are supported by key points derived from the line features 504 are included in the output point features P. This provides a reliable and repeatable point feature P for determining a 3D position using the stereo disparity d.

[0055] A point feature P in the left stereo image can be used to determine a matching point feature P in the right stereo image by correlating a neighborhood of pixels around the point feature in the left stereo image. For each point feature P in the left stereo image, a small neighborhood of pixel values ​​around the point feature P (e.g., 7X7 or 11X11 pixels) can be used to form a correlation kernel. The correlation kernel can be used to determine the position in the right stereo image where the correlation kernel has the highest correlation coefficient, for example, at this position, the neighborhood of pixels around the feature point P in the left stereo image has the same value and position as the neighborhood of pixels around the right stereo image position. This position is a match for the left stereo point feature P and can be used to determine the stereo disparity d, as described above with respect to Figure 3 described.

[0056] The 3D positions of the point features P and the line features 504 may be received by the pose solver 724, where the 3D pose of the stereo camera 702 is determined by a perspective n-point and line solver algorithm. The perspective n-point and line solver algorithm is described in "CvxPnPL: A unified convex solution to the absolute pose estimation problem from point and line correspondences" by Agotinho, S., Gomes, J., and Del Bue, A., arXiv.com, arXiv:1907.10545, 2019 as of the filing of this application. The perspective n-point and line solver algorithm iteratively determines the minimum error 3D pose of the camera based on the set of 3D points output by the 3D processing 722.

[0057] Figure 8 is an illustration of two temporally adjacent left stereo images 802 , 804 including a match indicated by a line 806 between point features P and two temporally adjacent right stereo images 808 , 810 including a match indicated by a line 812 between points determined based on line features 504 . Figure 8 Output from the differentiable optimal transport processor 720 is shown indicating matches received by the 3D processing 722. The differentiable optimal transport processor 720 processes potential matches between point features P and points based on line features 504 to eliminate inconsistent matches. Consistent matches are matches that produce parallel lines 806, 812 of the same length within a user-determined tolerance, indicating that the same points in each of the image pairs 802, 804 and 808, 810 have been selected as matching pairs.

[0058] Fig. 9 9 is an illustration of a vehicle positioning system 900 based on stereo video odometry. The vehicle positioning system 900 determines a high-resolution real-world position 912 of the vehicle 110 based on stereo image data received from a stereo camera 902 and GNSS / IMU 904 data received from sensors 116 included in the vehicle 110. As defined above, the high-resolution real-world position has a resolution of approximately 10 cm. As described above with respect to Figure 7As described, the stereo camera 902 data is processed by the stereo video odometry system 700 to determine the 3D pose of the stereo camera 902. The GNSS / IMU 904 data is processed by the odometry fusion processor 906 to produce a coarse position of the vehicle 110. As defined above, the coarse position has a resolution greater than one meter. The 3D camera pose and the coarse position of the vehicle 110 are received by an extended Kalman filter processor, which generates a best fit between the 3D camera pose and the coarse vehicle position. Determining the best fit between the high-resolution 3D camera pose and the coarse vehicle position generates a high-resolution real-world position 912 of the vehicle 110.

[0059] Fig.10 1 is a flow chart of a process 1000 for determining a 3D camera pose using a stereoscopic video system 700. For example, the process 1000 may be implemented in a computing device 115. The process 1000 includes a number of blocks that may be performed in the order shown. Alternatively or in addition, the process 1000 may include fewer blocks, or may include blocks that are performed in a different order.

[0060] Process 1000 begins at block 1002, where computing device 115 in vehicle 110 acquires two pairs of temporally adjacent left stereo images 704 and right stereo images 706 from stereo sensor 116, which may be a camera included in vehicle 110. The two pairs of temporally adjacent left stereo images 704 and right stereo images 706 include data about the environment surrounding vehicle 110.

[0061] At block 1004, two pairs of temporally adjacent left and right stereo images 704, 706 are input to the stereo video ranging system 700, which segments the temporally adjacent left stereo images 704 to detect the sky and dynamic objects using a segmenter 708. The segmented temporally adjacent left stereo images 704 are segmented to eliminate the dynamic objects 602 and the sky 604, 606, 608, leaving only the background 610 for further processing.

[0062] At block 1006, the segmented temporally adjacent pair of left stereo images 704 is received by the point feature processor 710 and the line feature processor 712 to determine the point features P and the line features 504, as described above with respect to Figure 7 described.

[0063] At block 1008, the line feature 504 is received by the key point detector 714 to determine the key points indicated by the line feature 504, as described above with respect to Figure 7 described.

[0064] At block 1010, the point feature P and the line feature 504 are received by attn / GNN 716, 718, respectively, to determine a match of the point feature P and the line feature 504 supported by the keypoints received from the keypoint detector 714. The outputs from attn / GNN 716, 718 are received by the optimal transport processor 720 that matches the point feature P and the line feature 504.

[0065] At block 1012, a pair of temporally adjacent left stereo images 704 including matching point features P and a pair of temporally adjacent right stereo images 704 including line feature matching are input to a 3D processor 722 to determine a 3D position in the scene 202 based on: a stereo disparity d determined by matching point features P from the left stereo image 704 with matching point features P determined in the right stereo image 706, as described above with respect to Figure 3 and Figure 7 described.

[0066] At block 1014, the 3D position is received by the pose solver 724, which uses a perspective n-point and line algorithm to determine the 3D camera pose of the stereo camera 300, as described above with respect to Figure 7 After block 1014, process 1000 ends.

[0067] Fig.11 1 is a flow chart of a process 1100 for operating a vehicle 110 based on a 3D camera pose determined by a stereoscopic video system 700. As a non-limiting example, the process 1100 is described in terms of operating a vehicle. The process 1100 may be more generally applicable to mobile systems. For example, the process 1100 may provide high-resolution pose data to mobile systems such as mobile robots and drones. The process 1100 may also be applied to systems that include mobile components, such as fixed robots, package sorting systems, and security systems. The process 1100 may be implemented by a computing device 115 included in the vehicle 110. The process 1100 includes a plurality of blocks that may be executed in the order shown. Alternatively or in addition, the process 1100 may include fewer blocks, or may include blocks that are executed in a different order.

[0068] Process 1100 begins at block 1102 , where computing device 115 in vehicle 110 acquires two pairs of temporally adjacent left stereo images 704 and right stereo images 706 from stereo camera 702 .

[0069] At block 1104, computing device 115 determines the 3D camera pose of stereo camera 702 based on two pairs of temporally adjacent left stereo images 704 and right stereo images 702, as described above with respect to Figure 7 and Fig.10 described.

[0070] At block 1106, computing device 115 determines a high-resolution vehicle pose based on the 3D camera pose received from block 1104 and sensor data acquired from sensors 116 included in vehicle 110. For example, the sensors may include GPS and IMU.

[0071] At box 1108, the computing device 115 operates the vehicle 110 based on the high-resolution vehicle pose. For example, the computing device 115 may determine a path polynomial that directs vehicle motion from a current position based on the high-resolution vehicle pose to a planned position. The vehicle 110 may be operated by determining a path polynomial function that maintains minimum and maximum limits on lateral and longitudinal accelerations. The vehicle 110 may be operated along the vehicle path by transmitting commands to the controllers 112, 113, 114 to control vehicle propulsion, steering, and brakes. After box 1008, the process 1100 ends.

[0072] Computing devices such as those described herein typically each include commands that can be executed by one or more computing devices such as those identified above and used to implement the blocks or steps of the processes described above. For example, the process blocks described above may be embodied as computer executable commands.

[0073] The computer executable instructions may be compiled or interpreted by a computer program created using a variety of programming languages ​​and / or techniques, including but not limited to the following in single or combined form: Java TM , C, C++, Python, Julia, SCALA, Visual Basic, Java Script, Perl, HTML, etc. Typically, a processor (i.e., a microprocessor) receives commands from a memory, a computer-readable medium, etc., and executes these commands to perform one or more processes including one or more of the processes described herein. Such commands and other data may be stored in files and transmitted using a variety of computer-readable media. Files in a computing device are typically collections of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0074] Computer-readable media (also referred to as processor-readable media) include any non-transitory (i.e., tangible) media that participate in providing data (i.e., instructions) that can be read by a computer (i.e., by a processor of a computer). Such media may take many forms, including, but not limited to, non-volatile media and volatile media. Instructions may be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0075] Unless otherwise expressly indicated herein, all terms used in the claims are intended to be given their ordinary and customary meanings as understood by those skilled in the art. Specifically, unless a claim recites an express limitation to the contrary, the use of singular articles such as "a," "an," "the," and "said" should be interpreted as reciting one or more of the indicated elements.

[0076] The term "exemplary" is used herein in the sense of referring to an example, ie, reference to "exemplary widget" should be interpreted as referring merely to an example of a widget.

[0077] The adverb “approximately” modifying a value or result means that the shape, structure, measurement, value, determination, calculation, etc. may vary from the exactly described geometry, distance, measurement, value, determination, calculation, etc. due to imperfections in materials, machining, manufacturing, sensor measurement, calculation, processing time, communication time, etc.

[0078] In the accompanying drawings, the same candidate marks indicate the same elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, etc. described herein, it should be understood that although the steps or frames of such processes, etc. have been described as occurring in a sequence in a particular order, such processes may be practiced by the described steps performed in an order other than the order described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. In other words, the description of the process herein is provided for the purpose of illustrating certain embodiments and should never be interpreted as limiting the claimed invention.

[0079] According to the present invention, a system is provided, the system comprising: a computer, the computer comprising a processor and a memory, the memory comprising instructions, the instructions being executable by the processor to: select a first first side stereo image and a second first side stereo image of a first pair of stereo images and a second pair of stereo images acquired by a first side camera of a stereo camera at a first time step and a second time step respectively; mask the first first side stereo image and the second first side stereo image; determine point features in the masked first first side stereo image and the second first side stereo image; determine line features in the masked first first side stereo image and the second first side stereo image; and use a first An attention / graph neural network (attn / GNN) is provided to determine matching point features in the masked first first side stereo image and the second first side stereo image; a second attn / GNN is used to determine matching line features in the masked first first side stereo image and the second first side stereo image based on key points determined according to the line features; a three-dimensional (3D) position in a scene is determined by determining stereo disparity based on the matched point features included in the first first side stereo image and point features determined in the first second side stereo images of the first pair of stereo images and the second pair of stereo images; and a 3D stereo camera pose is determined by determining a perspective n-point and line algorithm at the 3D position.

[0080] According to an embodiment, the instructions include further instructions for performing the following operations: masking the first first side stereo image and the second first side stereo image by determining a first segmented image and a second segmented image of the first first side stereo image and the second first side stereo image by an image segmentor and masking the first first side stereo image and the second first side stereo image based on the first segmented image and the second segmented image, respectively.

[0081] According to an embodiment, masking the first first side stereoscopic image and the second first side stereoscopic image removes dynamic objects and the sky.

[0082] According to an embodiment, the instructions include further instructions for determining the point features in the masked first first side stereo image and the second first side stereo image by performing self-supervised machine learning using the first attn / GNN followed by a differentiable optimal transport processor.

[0083] According to an embodiment, the instructions include further instructions for determining the line features in the masked first first side stereo image and the second first side stereo image by performing self-supervised machine learning using the second attn / GNN followed by a differentiable optimal transport processor.

[0084] According to an embodiment, the instructions comprise further instructions for determining the point features based on a Superpoint algorithm.

[0085] According to an embodiment, the instructions comprise further instructions for determining the line characteristics based on a SOLD2 algorithm.

[0086] According to an embodiment, the instructions comprise further instructions for determining the key points by sampling the line features determined in the masked first first side stereo image and the second first side stereo image.

[0087] According to an embodiment, the instructions comprise further instructions for determining a point feature in the first second side stereo images of the first and second pair of stereo images based on correlating a pixel neighborhood around the point feature in the first left stereo image.

[0088] According to an embodiment, the instructions include further instructions for performing the following operations: determining the stereo disparity by determining the distance between point features in the first second side stereo image and the masked first first side stereo image and based on one or more of the baseline, focal length and pixel scale between the stereo cameras.

[0089] According to an embodiment, the instructions include further instructions for combining the 3D stereo camera pose with one or more of Global Navigation Satellite System (GNSS) data or an Inertial Measurement Unit (IMU) to determine a 3D vehicle pose based on an Extended Kalman Filter.

[0090] According to an embodiment, the instructions include further instructions for determining a path polynomial based on the 3D vehicle pose, and operating the vehicle based on the path polynomial.

[0091] According to the present invention, a method is provided, comprising: selecting a first first side stereo image and a second first side stereo image of a first pair of stereo images and a second pair of stereo images acquired at a first time step and a second time step, respectively; masking the first first side stereo image and the second first side stereo image; determining point features in the masked first first side stereo image and the second first side stereo image; determining line features in the masked first first side stereo image and the second first side stereo image; using a first attention / graph neural network (attn / GNN) based on key points determined according to the line features to determine matching point features in the masked first first side stereo image and the second first side stereo image; using a second attn / GNN based on the key points determined according to the line features to determine matching line features in the masked first first side stereo image and the second first side stereo image; determining a three-dimensional (3D) position in a scene by determining stereo disparity based on the matched point features included in the first first side stereo image and the point features determined in the first second side stereo images of the first pair of stereo images and the second pair of stereo images; and determining a 3D stereo camera pose by determining a perspective n-point and line algorithm at the 3D position.

[0092] In one aspect of the present invention, the method includes masking the first first side stereo image and the second first side stereo image by determining a first segmented image and a second segmented image of the first first side stereo image and the second first side stereo image by an image segmentor and masking the first first side stereo image and the second first side stereo image based on the first segmented image and the second segmented image, respectively.

[0093] In one aspect of the present invention, masking the first first side stereoscopic image and the second first side stereoscopic image removes dynamic objects and the sky.

[0094] In one aspect of the invention, the method comprises determining the point features in the masked first first side stereo image and the second first side stereo image by performing self-supervised machine learning using the first attn / GNN followed by a differentiable optimal transport processor.

[0095] In one aspect of the invention, the method comprises determining the line features in the masked first first side stereo image and the second first side stereo image by performing self-supervised machine learning using the second attn / GNN followed by a differentiable optimal transport processor.

[0096] In one aspect of the invention, the method comprises determining the point features based on a Superpoint algorithm.

[0097] In one aspect of the invention, the method comprises determining the line characteristics based on a SOLD2 algorithm.

[0098] In an aspect of the present invention, the method comprises determining the key points by sampling the line features determined in the masked first first side stereo image and the second first side stereo image.

Claims

1. A method, comprising: selecting a first first side stereo image and a second first side stereo image of a first pair of stereo images and a second pair of stereo images acquired at a first time step and a second time step, respectively; masking the first first side stereoscopic image and the second first side stereoscopic image; determining point features in the masked first first side stereo image and the second first side stereo image; determining line features in the masked first first side stereo image and the second first side stereo image; determining matching point features in the masked first first side stereo image and the second first side stereo image using a first attention / graph neural network (attn / GNN) based on key points determined from the line features; determining matching line features in the masked first first side stereo image and the second first side stereo image using a second attn / GNN based on key points determined from the line features; determining a three-dimensional (3D) position in a scene by determining a stereo disparity based on the matched point features included in the first first side stereo image and point features determined in the first second side stereo images of the first pair of stereo images and the second pair of stereo images; as well as The 3D stereo camera pose is determined by determining a perspective n-point and line algorithm at the 3D position.

2. The method of claim 1 , further comprising masking the first first side stereo image and the second first side stereo image by determining, by an image segmenter, a first segmented image and a second segmented image of the first first side stereo image and the second first side stereo image and masking the first first side stereo image and the second first side stereo image based on the first segmented image and the second segmented image, respectively. The method of claim 1 , wherein masking the first first side stereo image and the second first side stereo image removes dynamic objects and sky.

4. The method of claim 1, further comprising determining the point features in the masked first first side stereo image and the second first side stereo image by performing self-supervised machine learning using the first attn / GNN followed by a differentiable optimal transport processor.

5. The method of claim 1, further comprising determining the line features in the masked first first side stereo image and the second first side stereo image by performing self-supervised machine learning using the second attn / GNN followed by a differentiable optimal transport processor. The method of claim 1 , further comprising determining the point features based on a Superpoint algorithm.

7. The method of claim 1, further comprising determining the line features based on a SOLD2 algorithm. 8 . The method of claim 1 , further comprising determining the key points by sampling the line features determined in the masked first first side stereo image and the second first side stereo image.

9. The method of claim 1, further comprising determining a point feature in the first and second side stereo images of the first and second pair of stereo images based on correlating a pixel neighborhood around the point feature in the first left stereo image.

10. The method of claim 1, further comprising determining stereo disparity by determining distances between point features in the first second side stereo image and the masked first first side stereo image and based on one or more of a baseline, focal length, and pixel scale between stereo cameras.

11. The method of claim 1 , further comprising combining the 3D stereo camera pose with one or more of Global Navigation Satellite System (GNSS) data or an Inertial Measurement Unit (IMU) to determine a 3D vehicle pose based on an Extended Kalman Filter.

12. The method of claim 1 further comprising determining a path polynomial based on the 3D vehicle pose, and operating a vehicle based on the path polynomial.

13. The method of claim 6, wherein Superpoint is a feature point detector based on a neural network.

14. The method of claim 7, wherein SOLD2 is an occlusion-aware line feature detection and description algorithm.

15. A system comprising a computer programmed to perform the method of any one of claims 1 to 14.