Data-driven dynamically reconfigured disparity map

Abstract

In some examples, the system can receive at least one image including a road from at least one camera of the vehicle. The system can further receive vehicle location information including an indication of the vehicle location. Additionally, the system can receive at least one of historical information from a history database or road anomaly information. The road anomaly information is determined from at least one of a road anomaly database or real-time road anomaly detection. Based on the at least one image, the indication of the vehicle location, and at least one of the historical information or the road anomaly information, the system can generate at least one of a disparity map or a disparity image.

Description

Technical Field

[0001] Some implementations herein can be used to efficiently and dynamically generate a disparity map or disparity image that can be used to detect obstacles, road anomalies, and other features with high depth accuracy in various scenarios without significantly increasing the processing load on the in-vehicle processor.

Background Art

[0002] Advanced Driver Assistance Systems (ADAS), as well as semi-autonomous vehicle systems, fully autonomous vehicle systems, or other Autonomous Driving (AD) systems, are systems that automate or enhance vehicle control for purposes such as improving safety and automating navigation. These systems typically use many types of sensors to recognize lanes, other vehicles, obstacles, pedestrians, etc. and avoid them. In particular, a camera is a reliable sensor for imaging the visual representation of the environment around the vehicle, for example, for performing on-road recognition, sign recognition, obstacle avoidance, etc. (For example, Patent Document 1) 。

[0003] Such systems may at least partially rely on the generation of a disparity map to perform recognition functions based on the received camera images. To provide higher accuracy, a denser disparity map is preferred. However, generating a denser disparity map may significantly increase the processing load on the in-vehicle processor.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] In some implementations, the system can receive at least one image including a road from at least one camera of the vehicle. The system can further receive position information of the vehicle including a display of the vehicle position. Further, the system can receive at least one of historical information from a historical database or abnormal information of the road. The abnormal information of the road is determined from at least one of a road abnormal database or real-time road anomaly detection. Based on the at least one image, the display of the vehicle position, and at least one of the historical information or the abnormal information of the road, the system can generate at least one of a disparity map or a disparity image.

Brief Description of the Drawings

[0006] A detailed description will be given with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference numeral identifies the drawing in which the reference numeral first appears. The same reference numerals used in different drawings indicate similar or identical items or features.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

[0007] Some implementations herein can be used to efficiently and dynamically generate a disparity map or disparity image that can be used to detect obstacles, road anomalies, and other features with high depth accuracy in various scenarios without significantly increasing the processing load on an in-vehicle processor. Examples herein may include generating a high-precision disparity map that can be used to detect features and their respective distances with higher accuracy. In some cases, a historical map database can be developed from driver monitoring data and other data received from multiple vehicles and can be used to improve the generation efficiency of the disparity maps herein. Thus, the techniques herein enable the generation of high-precision disparity maps using lower computing requirements, thereby improving the safety of ADAS systems and AD systems.

[0008] As an example, a processor mounted in a vehicle may receive at least one image that images a road or other travel path from at least one camera of the vehicle. The processor may also receive vehicle position information including a display of the current position of the vehicle that can display the current lane in which the vehicle is moving, based on, for example, GPS (Global Positioning System) information. The processor may also receive at least one of historical information from a historical database or road anomaly information from a road anomaly database.

[0009] The processor may generate a disparity map or a disparity image based on at least one image, a display of the vehicle's position, as well as historical information and / or road anomaly information. For example, the disparity map or the disparity image may have some regions of higher disparity density and some regions of lower disparity density dynamically determined based on the received information. In particular, regions determined to include potential obstacles, road anomalies, or other regions determined to be of interest are processed to have denser disparity information, while other regions of the disparity map or the disparity image may be processed to have lower density disparity information, e.g., sparse disparity information. In this way, by dynamically processing only selected portions of the disparity map or the disparity image to have dense disparity information while processing the remaining portions of the disparity map or the disparity image to have lower density or sparser disparity information, a significant amount of computing resources can be saved.

[0010] In some implementations, the system herein may include at least one monocular camera system or a stereo camera system mountable on the vehicle such that under normal conditions, the wide-angle field of view (FOV) of the camera is typically parallel to the road surface. The FOV of the camera may be wide enough to image the front road and roadside objects. Further, the system herein is not limited to imaging only the front road of the vehicle. For example, in some instances, the camera may additionally or alternatively be mounted at the rear and / or the side of the vehicle to detect the entire area around the vehicle.

[0011] In some examples, the accuracy of the disparity map for detecting obstacles, road anomalies, and other features has been improved. For example, by accurately detecting features and the distance to the features, the safety of the vehicle can be significantly improved. There are few conventional techniques available for improving the accuracy of the disparity image generated based on the images captured by the camera. The implementations herein can use data-driven information to efficiently utilize computing resources to achieve a more accurate disparity map. For example, the data-driven information can be used to select the pixel-level bit depth (image quality), the optimal image compression level and method, and / or to dynamically estimate the disparity parameters to generate the improved dynamically configurable disparity map herein under various real-world scenarios. Therefore, the data-driven, dynamically reconfigurable disparity map techniques and arrangements herein provide significant advantages in improving the safety and comfort for vehicle occupants.

[0012] Some examples may rely only on the images received from the camera system and the information received via the network to select the priority areas in the received images, select the image compression method to use, dynamically select the bit depth of the pixels, and dynamically determine the optimal search range for calculating the disparity range. For example, some implementations may use Internet of Things (IoT) information to select the priority areas, determine the image quality, and determine the image compression method to use for calculating the disparity map.

[0013] The implementation forms in this specification can select regions of interest in an image using a history map, and while generating dense disparity information for regions with higher priority, it can select regions with higher priority to generate sparser disparity information for regions outside the regions with higher priority. Therefore, the implementation forms in this specification are not limited to determining priority regions based on detected obstacles, road anomalies, or other features. Instead, the history map can be used to determine the resolution of various regions of the disparity map in advance. In some examples, the determination of regions with higher priority may be further based on Internet of Things (IoT) information, such as the outputs of current and past driver monitoring systems that may all be included in the history map, road anomaly data stored in and / or real-time detected road anomalies in a road anomaly database, and information such as current and past traffic movements, directions, congestion situations, etc.

[0014] Furthermore, based on the history map and the selected regions with higher priority, the bit depth and image quality of appropriate pixels can be selected. Furthermore, based on the history map and the selected regions with higher priority, the appropriate level and algorithm of image compression can be selected prior to generating a disparity map from the image. At the same time, the disparity search range for the selected regions with higher priority can be dynamically estimated in real time, enabling the efficient generation of denser disparity information for regions with higher priority.

[0015] Some of the examples in this specification can provide a great advantage of estimating a very accurate disparity map for current and future semi-autonomous and fully autonomous vehicles, thereby detecting features and the distances of features with high precision. For example, the techniques in this specification can be implemented using various types of processors, such as embedded processors (e.g., AD / ADAS electronic control units (ECUs) or other types of ECUs), general-purpose CPUs, dedicated processors, etc., as well as processors of mobile devices such as smartphones, tablet computing devices, etc. Further, the implementations in this specification are not limited to passenger vehicles and can be applied to any vehicle with built-in semi-autonomous or fully autonomous capabilities, such as shuttle buses, trucks, trains, ships, etc.

[0016] For illustrative purposes, some exemplary implementations are described in an environment that generates a disparity image having a dense disparity region dynamically determined based on the detection and recognition of potential obstacles or other features of interest. However, the implementations in this specification are not limited to the specific examples provided and can be extended to other types of cameras, other types of vehicles, other types of roads and features, other weather conditions, etc., as will be apparent to those skilled in the art in light of the disclosure herein.

[0017] FIG. 1 shows an example of a recognition system 100 configured to perform recognition around a vehicle 102 according to some implementation forms. In this example, it is assumed that the vehicle 102 moves on a road or other movement route 104 in the direction indicated by the arrow 106. The recognition system 100 in this specification can include at least one camera system 108 that can be mounted on the vehicle 102. In the illustrated example, the camera system 108 can include one or more cameras 110. In this example, the camera 110 includes a stereo camera. In other examples, the camera system 108 may include one or more monocameras instead of or in addition to the stereo camera as the camera 110. At the same time, in this example, although the camera 110 of the camera system 108 is illustrated as being installed on the roof of the vehicle, in other examples, the camera 110 may be installed at any of various different positions on the vehicle 102.

[0018] In the illustrated example, the field of view (FOV) 114 of the camera 110 is wide enough to image the road or other movement route 104 in front of the vehicle 102, and can include other lanes 116, other vehicles 118, roadside lane markings 120, roadside areas 122, lane markings 124, etc. In some cases, the camera system 108 can continuously capture images corresponding to the FOV 114 at a rate sufficient to provide images, for example, at 10 frames per second, 15 frames per second, 30 frames per second, 60 frames per second, or any other preferred frequency, while the vehicle 102 is in operation, so as to take avoidance actions or recognize any obstacles or road anomalies in the movement route 104 and within the movement route without delay in order to adapt to the recognition information. For example, the image capture frequency (sampling frequency) of the camera system 108 may be increased as the vehicle speed increases.

[0019] Vehicle 102 can include one or more vehicle computing devices 126, as further described below. The vehicle computing device 126 can execute a recognition program 128 and a vehicle control program 130. In some cases, the recognition program 128 can receive an image captured by a camera of the camera system 108, process the image, and recognize the current travel route 104 of the vehicle 102 based on one or more dynamically reconfigured disparity maps 132. For example, the dynamically reconfigured disparity map 132 can be dynamically reconfigured to have some regions of interest with denser disparity information and other regions with sparser disparity information. The recognition program 128 can provide recognition information about detected and recognized obstacles, road anomalies, and other features to the vehicle control program 130, and the control program can, based on the disparity information, initiate one or more actions such as issuing a warning to alert the occupant, braking the vehicle 102, accelerating the vehicle, and steering one or more wheels of the vehicle 102.

[0020] In some cases, the camera system 108 may include at least one vehicle computing device 126 that executes the recognition program 128. In other cases, the vehicle computing device 126 may be separated from the camera system 108 and installed in another location of the vehicle 102 to execute the recognition program 128. In either case, the vehicle computing device 126 can receive an image from the camera system 108 and process the image to detect features such as roads, road characteristics, road anomalies, signs, obstacles, other vehicles, etc.

[0021] In some examples, the recognition program 128 can generate a disparity map from the received image using, for example, stereo camera images, monocular camera images, or images taken from multiple monocular cameras. When using a monocular camera, a trained machine learning model (not shown in FIG. 1) can be used to calculate a depth map. For example, first, a set of monocular images and their corresponding ground truth parallax maps can be captured and used to train the machine learning model. Subsequently, the machine learning model can be used to predict an approximation of the parallax map according to the newly captured image.

[0022] Alternatively, in the case of a stereo camera or multiple cameras, the image can be captured by two or more cameras. Using the captured image, parallax can be calculated using a block matching technique such as semi-global block matching or any other suitable technique. The parallax information can be used to generate a disparity map and / or a disparity image. In some examples herein, a stereo camera system is used as an exemplary system for explaining some exemplary implementations, but those skilled in the art will also understand that similar arrangements and techniques can be applied using a single monocular camera or a system having multiple monocular cameras in addition.

[0023] In a stereo camera system, left and right camera images synchronized using an image acquisition system can be captured, and they may have distortions caused by displacement of the cameras during installation or in an unfavorable driving state. Further, the distortion may be caused by the lenses attached to the stereo camera. After the stereo images are acquired, in some examples, calibration matrix image correction is performed to remove the distortion and re-project the images onto a projection plane parallel to the line connecting the two cameras. This enables the epipolar lines of the left and right images to be aligned. According to an example in this specification, by aligning two cameras on the same plane, the search for generating a disparity map can be simplified to one dimension, for example, a horizontal line parallel to the baseline between the left and right cameras. From the corrected stereo images, depending on the capabilities of the ECU, a disparity map or a disparity image can be calculated using a stereo block matching algorithm.

[0024] After stereo images are captured and corrected based on calibration information, stereo matching can be performed on the corrected images using, for example, a predefined region of interest (ROI). For example, the disparity with a predefined resolution for each pixel can be estimated by a block matching technique. This may include searching for the region in one image of the image pair that best corresponds to a template in the other image. The template can be shifted along the epipolar line within a predefined fixed disparity range. The process can be repeated until the disparity is estimated for all pixels in the right image. In the prior art, for example, based on the processing capabilities of the ECU, either a sparse disparity or a dense disparity can be calculated for the entire image.

[0025] Depth estimation of features based on an image captured by a stereo camera may depend on the density of a disparity map created using stereo images obtained from the left and right cameras of the stereo camera. A dense disparity map or a higher-resolution disparity map may provide accurate depth information with high reliability for all pixels in the image. On the other hand, a sparse disparity map or a lower-resolution disparity map may also contain high-precision depth information, but only for a limited number of pixels. In some conventional systems, a fixed ROI can be used for the left and right camera images of the stereo camera, and either a sparse disparity map or a dense disparity map can be calculated for the entire image. However, calculating a dense disparity map for a fixed ROI may often be computationally expensive and inefficient. On the other hand, a sparse disparity map of a fixed ROI may have low accuracy and thus may not be suitable for all AD / ADAS applications.

[0026] Furthermore, the quality of pixels may refer to the bit depth of the stereo image (e.g., 8 bits, 10 bits, 12 bits, 16 bits, etc.), and the image compression method used may also affect the reliability and accuracy of disparity map generation. For example, using a low-bit-depth image (e.g., 8 bits) with irreversible image compression may result in a low memory requirement but may generate a sparse disparity map. On the other hand, increasing the bit depth with a reversible image compression method may be able to generate a dense disparity map, but this may increase the memory requirement and as a result, may increase the processing and memory capabilities of the entire system.

[0027] Furthermore, parallax parameters such as the search range may also be considered when generating a disparity map or a disparity image. For example, the search range can define the minimum and maximum disparity values in a given stereo image pair. In the prior art, a fixed search range for the entire image (e.g., search range 0 to 250) may be used to calculate the disparity map without considering various different scenarios and / or types of obstacles, road anomalies, or other features. As a result, the processing time may increase, and a more high-performance processor may be required for use in a real-time system.

[0028] To address these problems, the system in this specification can generate a disparity map that can provide reliable feature detection and high-precision depth estimation, and can meet the requirements of vehicle manufacturers to lower system costs. For example, in the implementation forms in this specification, a disparity map or a disparity image can be generated by dynamically determining a priority area within the image based on data-driven information obtained from, for example, a history database and determined via a history map. Furthermore, in the implementation forms in this specification, when determining the bit depth (e.g., pixel quality) for each pixel and the image compression method, the priority area can be considered to reduce memory requirements and improve depth estimation for all obstacles, road anomalies, and features in all other expected scenarios.

[0029] Furthermore, in some examples in this specification, the parallax parameters can be dynamically estimated to calculate a dense disparity map and a sparse disparity map using, for example, edge and disparity information, thereby improving depth estimation with lower system requirements. Furthermore, in some examples, a disparity map can be generated and reconstructed using stereo camera images, but the same technology can be applied to images captured by one or more monocular cameras.

[0030] Figure 2 shows an exemplary architecture of a recognition system 200 that can be included within vehicle 102 according to some implementations. Each vehicle computing device 126 can include one or more processors 202, one or more computer-readable media 204, and one or more communication interfaces 206. In some examples, vehicle computing device 126 may include one or more ECUs or any other various types of computing devices. For example, computing device 126 may include one or more ADAS / AD ECUs for controlling critical vehicle systems and may perform ADAS and / or AD tasks such as navigation, braking, steering, accelerating, decelerating, etc. Computing device 126 may also include other ECUs for controlling other vehicle systems.

[0031] ECU is a general term for any embedded system that controls one or more systems, subsystems, or components within a vehicle. Software such as recognition program 128 and vehicle control program 130 is executed by one or more ECUs and stored in a part of the computer-readable media 204 (e.g., program ROM) associated with each ECU to make the ECU operable as an embedded system. ECUs can generally communicate with each other via vehicle bus 208 according to a vehicle bus protocol. As an example, the Controller Area Network bus (CAN bus) protocol is a vehicle bus protocol that enables ECUs and other vehicle devices and systems to communicate with each other without a host computer. The CAN bus may include at least two different types. For example, high-speed CAN may be used in applications where the bus is laid from one end of the environment to the other, while fault-tolerant CAN is often used when a group of nodes are connected together.

[0032] Each ECU or other vehicle computing device 126 may include one or more processors 202, which may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a microcomputer, a microcontroller, a digital signal processor, a state machine, one or more of logic circuits, and / or any device that operates signals based on processing instructions. As an example, the processor 202 may include one or more hardware processors and / or logic circuits of any suitable type that are specially programmed or configured to execute the algorithms and other processes described herein. The processor 202 can be configured to fetch and execute computer-readable instructions stored on a computer-readable medium 204, and these instructions can program the processor 202 to perform the functions described herein.

[0033] The computer-readable medium 204 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information such as computer-readable instructions, data structures, programs, program modules, and other code or data. For example, the computer-readable medium 204 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, optical storage devices, solid state storage devices, magnetic disks, cloud storage, or any other medium that can be used to store desired information and is accessible by a computing device. Depending on the configuration of the vehicle computing device 126, the computer-readable medium 204, when mentioned, may be a tangible non-transitory medium to the extent that it excludes media such as energy, carrier signals, electromagnetic waves, and / or signals themselves. In some cases, the computer-readable medium 204 may be in the same location as the vehicle computing device 126, while in other examples, the computer-readable medium 204 may be partially remote from the vehicle computing device 126 and accessible via, for example, a wireless network.

[0034] The computer-readable medium 204 can be used to store any number of functional components executable by the processor 202. In many implementations, these functional components include instructions or programs executable by the processor 202, which, when executed, specifically program the processor 202 to perform operations attributable to the vehicle computing device 126 herein. The functional components stored on the computer-readable medium 204 can include the recognition program 128 and the vehicle control program 130, each of which can include one or more computer programs, applications, executable code, or portions thereof. Further, although these programs are illustrated together in this example, in use, some or all of these programs may be executed on separate vehicle computing devices 126.

[0035] Furthermore, the computer-readable medium 204 can store data, data structures, and other information used to perform the functions and services described herein. For example, the computer-readable medium 204 can store a history map 210, recognition information 212, vehicle data 214, image data 216, one or more machine learning models (MLMs) 218, other sensor data 220, and the like. As will be further described below, the recognition information 212 can include the dynamically reconfigured disparity map 132 described above. Further, although these data and data structures are shown together in this example, in use, some or all of these data and / or data structures may be stored by a separate computing device 126 or together with a separate computing device 126. The computing device 126 can also include or maintain other functional components and data, which can include programs, drivers, and the like, and data used or generated by the functional components. Further, the computing device 126 can include many other logical, programmatic, and physical components, and those described above are merely exemplary in relation to the discussion herein.

[0036] As described above, when a monocular camera is used as the camera 110, a depth map can be calculated using one or more trained machine learning models 218. For example, first, a set of monocular images and their corresponding ground truth parallax maps can be captured and used to train the machine learning model 218. Subsequently, the machine learning model 218 can be used to predict an approximation of the parallax map according to the newly captured image.

[0037] One or more communication interfaces 206 can include one or more software and hardware components to enable communication with various other devices, for example, via vehicle bus 208 and / or one or more networks (not shown in FIG. 2). For example, communication interface 206 can enable communication via one or more of, for example, LAN, Internet, cable network, cellular network, wireless network (e.g., Wi-Fi), and wired network (e.g., CAN, fiber channel, fiber optic, Ethernet), direct connection, and short-range communication such as BLUETOOTH (registered trademark), as further enumerated elsewhere in this specification.

[0038] In this example, camera 110 of camera system 108 includes stereo camera 219. In other examples, camera 110 may include one or more monocular cameras. Computing device 126 may communicate with camera system 108 via vehicle bus 208, direct connection, or any other type of connection to receive images 223 (e.g., left image and right image) from camera system 108. For example, as will be described in detail below, recognition program 128 can receive images 223 from camera system 108 and perform recognition of features within images 223. Image data 216 can include images 223 received from camera system 108, as well as other image information such as metadata or processed images. For example, some or all of images 223 may be received as unprocessed images without any substantial processing. Alternatively, in other examples, camera system 108 may perform image processing and recognition of images 223, rather than sending unprocessed images 223 to vehicle computing device 126, as will be further described below in connection with FIG. 3.

[0039] Furthermore, computing device 126 can receive vehicle data 214 from other systems and / or other sensors within the vehicle. For example, in addition to camera system 108, the vehicle may include a plurality of other sensors 225 that can provide sensor information used by vehicle control program 130. Some non-exhaustive examples of other sensors 225 can include radar, LiDAR, ultrasonic, global positioning system (GPS) receivers, for example, other cameras facing other directions, and the like. Further, vehicle data 214 used by vehicle control program 130 can include information received from or associated with various vehicle systems, such as suspension controller 224 associated with the suspension system, steering controller 226 associated with the steering system, vehicle speed controller 228 associated with the braking and acceleration systems, and the like.

[0040] As an example, recognition program 128 may continuously receive image 223 from camera system 108 as camera system 108 captures an image 223 of the moving path or the surrounding situation of other vehicles, for example, while the vehicle is in motion. Further, recognition program 128 may process the received image 223 to recognize road anomalies, obstacles, and other features. As will be further described below, recognition information 212 may include one or more dynamically reconfigured disparity maps 132 having one or more denser disparity regions and one or more less dense disparity regions.

[0041] The recognition program 128 can provide recognition information 212 about any recognized obstacles, road anomalies, other features, etc. to the vehicle control program 130, and the vehicle control program can take one or more actions according to the recognition information 212. In some examples, the vehicle control program 130 and / or the recognition program 128 can fuse, combine, and integrate the recognition information 212 determined from the image 223 with other sensor data 220 to provide additional available information to the vehicle control program 130 for controlling the vehicle.

[0042] As an example, the vehicle control program 130 may determine parameters for vehicle control using rule-based and / or artificial intelligence-based control algorithms. For example, the vehicle control program 130 may apply one or more machine learning models to determine appropriate actions such as braking, steering, decelerating, accelerating, etc. Further, the vehicle control program 130 may transmit one or more control signals 238 to one or more vehicle systems according to the recognition information 212. For example, the vehicle control program 130 may transmit the control signal 238 to the suspension controller 224, the steering controller 226, and / or the vehicle speed controller 228. For example, the control signal 238 may include specified spring coefficients and / or damping control information transmitted to the suspension controller 224, specified steering angles transmitted to the steering controller 226 to steer one or more wheels, and / or specified braking or acceleration control information transmitted to the vehicle speed controller 228.

[0043] Further, or alternatively, for example when the vehicle is under the control of a human driver, the vehicle control program 130 may transmit a control signal 238 to the display unit 240 to present a warning, and / or may transmit it to one or more warning devices 242 such as an audible warning device or a visual warning device. Examples of the warning device 242 include a speaker capable of generating an audible warning, a tactile device capable of generating vibration or other types of tactile warnings (e.g., within the seat or within the steering wheel), and / or a visual signal transmitter capable of generating a visual warning.

[0044] In some examples, the vehicle may include a driver monitoring system 244 that provides driver monitoring data 246 to the vehicle computing device 126. For example, the driver monitoring system 244 can monitor the line-of-sight direction, head position, steering wheel position, brake light operation, turn signal operation, accelerator operation, and other information regarding the driver of the vehicle and / or actions performed by the driver of the vehicle in order to provide the driver monitoring data 246 to the vehicle computing device 126. For example, the driver monitoring data 246 can be received by the recognition program 128 for use, at least in part, in generating the history map 210.

[0045] Furthermore, the vehicle computing device 126 may be in communication with a history database 250 that can provide additional information to the vehicle computing device 126 for generating the history map 210. For example, the history database 250 may include a database 252 of driver monitoring data, a vehicle light operation database 254, a steering wheel position database 255, and a past and current traffic direction and presence database 256. The information within the history database 250 can also be received from the vehicle 102 and from a plurality of other vehicles to establish a history associated with each position on each road or other travel path on which the vehicle moves.

[0046] Furthermore, the vehicle computing device 126 may be in communication with a road anomaly database 260 that may include information about road anomalies such as potholes, irregular surfaces, speed bumps, etc. For example, the road anomaly database 260 can be compiled using road anomaly information provided by a number of vehicles and can provide information about road anomalies encountered at each location on each road or other travel route that the vehicle moves on. Additionally, as described above, when determining the higher priority regions of the disparity images, real-time obstacle detection, road anomaly detection, and detection of other features by the recognition program 212 can also be taken into account.

[0047] In some examples, the road anomaly database 260 and / or the history database 250 may be stored locally on a computer-readable medium of the vehicle itself. In other examples, the history database 250 and / or the road anomaly database 260 may be stored, for example, in the cloud on one or more network computing devices 262 and be accessible via a network. For example, the network computing device 262 may be accessed in real-time by the vehicle computing device 126 while the vehicle is operating and may be accessed to obtain information for moving to a destination when the destination is determined. When the history database 250 and / or the road anomaly database 260 are stored at least partially locally on the vehicle 102, these databases may be updated periodically by accessing updated information from a primary database stored on the network computing device 262. Additionally, as the vehicle traverses a route, new information regarding road anomalies, detected features, detected traffic patterns, etc. may be uploaded to the network computing device and the history database 250 and / or the road anomaly database 260 may be updated and expanded for future use by the vehicle 102 and other vehicles.

[0048] Figure 3 shows an exemplary architecture of a recognition and vehicle control system 300 that may be included in vehicle 102 according to some implementations. In this example, camera system 108 can have processing capabilities to determine recognition information 212 independently of vehicle computing device 126. Accordingly, camera system 108 includes one or more processors 302, one or more computer-readable media 304, and one or more communication interfaces 306. The one or more processors 302 may be any of the processors 202 described above in connection with FIG. 2, or other suitable processors for performing the processing described herein, or may include them. Further, the one or more computer-readable media 304 may be any of the computer-readable media 204 described above in connection with FIG. 2, or other suitable computer-readable media, or may include them. Similarly, communication interface 306 may be any of the communication interfaces 206 described above in connection with FIG. 2, or other suitable communication interfaces, or may include them.

[0049] Further, camera system 108 includes a camera 110 that may include one or more lenses, one or more focusing systems, and one or more image sensors, which are well known in the art. In this example, camera 110 includes a stereo camera 219. In other examples, camera 110 may include one or more monocameras. Camera system 108 may execute a recognition program 128 on one or more processors 302. Accordingly, stereo camera 219 can capture an image 223 within its FOV and store image 223 on computer-readable media 304 as part of image data 216.

[0050] In some examples, as each image 223 is captured, the recognition program 128 can continuously receive the image 223 from the camera 110 into a buffer within the computer-readable medium 304. Further, the recognition program 128 can perform recognition processing on the received image 223, which processing can include, as will be described in further detail below, the generation of a dynamically reconfigured disparity map 132 based at least in part on the history map 210. For example, the history map 210 can be generated based on information such as current and past driver monitoring data 246, real-time road anomaly information detected by the recognition program, past road anomaly information received from the road anomaly database 260, current and past traffic movement, direction, and other information, as will be described further below. Thereafter, the recognition program 128 can transmit the recognition information 212 to the vehicle computing device 126 in real time (e.g., while the vehicle is moving and in time to perform the navigation function in response to detected obstacles, road anomalies, or other features). Thereafter, the vehicle control program 130 can process the recognition information 212 to control the vehicle, as described above in connection with FIG. 2 and as will be described further below.

[0051] In some examples, the vehicle computing device 126 can execute an integration program 308 that can first receive the recognition information 212 and can also receive other sensor data 220. For example, the integration program 308 can compare and reconcile the recognition information 212 with the other sensor data 220 to provide a more comprehensive indication of the vehicle's surroundings to the vehicle control program 130. Similar to the example of FIG. 2 described above, the vehicle control program 130 can transmit one or more control signals 238 to one or more vehicle systems 224, 226, 228, 240, and / or 242 based on the received recognition information 212 and other sensor data 220.

[0052] Furthermore, similar to the example of FIG. 2 described above, the recognition program 128 can receive information from, for example, the driver monitoring system 244, the history database 250, and / or the road abnormality database 260, and receive information for formulating the history map 210. For further details on formulating and using the history map 210, see the following description in relation to FIG. 6, for example.

[0053] FIG. 4 shows an example 400 of image capture performed by the camera system 108 according to some implementations. In this example, the camera system 108 includes a stereo camera 219. For example, the stereo camera 219 can include a right lens 402 and a left lens 404. The right lens 402 can capture a right image 406 within a right field of view (FOV) 407, and the left lens 404 can capture a left image 408 within a left FOV 409. In the illustrated example, the stereo camera 219 includes a right image sensor 410 for capturing an image through the right lens 402 and a left image sensor 412 for capturing an image through the left lens 404. In other examples, the stereo camera 219 may include a single image sensor (not shown in FIG. 4) that alternately captures images through the right lens 402 and the left lens 404.

[0054] The system (e.g., systems 100, 200, and / or 300) can use the right image 406 and the left image 408 respectively to determine a parallax image called a disparity image. For example, the system can calculate a disparity image using the stereo right image 406 and left image 408 based on, for example, a block matching method. As an example, as is well known in the art, for a point P L =(u1, v1) in the left image 408, the corresponding point P R =(u2, v2) in the right image 406 can be at the same height as P L when measured from a common baseline and v1 = v2. Thus, the measurement of parallax can be determined using a simple stereo camera theory where parallax can be defined as follows.

Equation

[0055] Based on the determined parallax, in the implementations herein, since the parallax is inversely proportional to the corresponding parallax, a parallax image can be determined by determining the depth information of the 3D points from the parallax. Therefore, the depth can be calculated, for example, as follows using the left image, the right image, and the actual parallax.

Equation

[0056] At least a portion of FIGS. 5, 8, 9, 16, and 20 includes flow diagrams illustrating exemplary algorithms or other processes according to some implementations. The processes are shown as a collection of blocks in a logical flow diagram representing a series of operations, some or all of which can be implemented in hardware, software, or a combination thereof. In a software context, the blocks can represent computer-executable instructions stored on one or more computer-readable media, which, when executed by one or more processors, program the processors to perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the blocks are described should not be construed as limiting. Any number of the described blocks can be combined in any order and / or in parallel to implement the process or an alternative process, and it is not necessary to execute all the blocks. For purposes of illustration, the processes are described with reference to the environments, frameworks, and systems described in the examples herein, but the processes can be implemented in a wide variety of other environments, frameworks, and systems.

[0057] FIG. 5 is a flowchart showing an exemplary process 500 for generating a dynamically reconfigured disparity map according to some implementations. In some examples, process 500 can be executed by the systems 100, 200, and / or 300 described above by executing the recognition program 128. For example, the disparity reconstruction techniques herein can include determining priority regions in an image using data-driven historical map information. Further, the system can select the pixel bit depth and the image compression algorithm to use for various different regions in the image being processed. Further, the system can determine a search range for disparity using the disparity map and edge information, and then apply dynamic stereo matching techniques to generate a disparity map for each region identified in the image.

[0058] As an example, after the engine is started and the destination is determined, the recognition program can be executed to start capturing left and right camera images. Using the current position of the vehicle in addition to the destination information, a historical map can be generated to determine higher priority regions in the captured images. In some examples, the higher priority regions may be determined based on road anomaly information received from a road anomaly database. Based on the identified higher priority regions and the current weather information, the system can determine the optimal image quality (bit depth) and the optimal image compression algorithm for each pixel and use them for each region of the image. Further, the system can determine a search range for disparity and perform stereo matching to generate a disparity map in which regions of denser disparity information are dynamically determined based at least in part on the historical map and / or road anomaly information. Based at least in part on the generated disparity map, recognition can be performed, and additional disparity maps can be repeatedly generated until the vehicle reaches the destination. An exemplary flowchart of this process is shown in FIG. 5.

[0059] At 502, the system can determine that the vehicle has started and a destination has been indicated.

[0060] At 504, the system may start acquiring an image from a camera. In some examples, the camera may be a stereo camera, while in other examples the camera may be one or more monocular cameras.

[0061] At 506, the system may determine the current position of the vehicle, including the current lane. For example, the system may access a GPS receiver or other position indication information to determine the current position of the vehicle.

[0062] At 508, the system may receive weather information. For example, the system may receive weather information via a network, or alternatively may receive weather information from local sensors mounted on the vehicle.

[0063] At 510, the system may determine historical map information. An example of determining historical map information will be described below in connection with FIG. 6.

[0064] At 512, the system may determine a parallax reconstruction by performing blocks 514, 516, and 518.

[0065] At 514, the system may select a region for the received image. Exemplary algorithms for selecting a region will be described below, for example, in connection with FIGS. 6-8.

[0066] At 516, the system may select an image quality and compression parameters. For example, as will be described below in connection with FIGS. 9-11, the system may determine the image quality and compression level / method to use for various different regions of the image based on the selected region.

[0067] At 518, the system may estimate parallax parameters and use them to generate a parallax map. For example, the system may determine an optimal search range for generating parallax information using the techniques described below, for example, in connection with FIGS. 15-19.

[0068] At 520, the system can perform dynamic stereo block matching on the received image and generate a disparity map or disparity image based on the estimated disparity parameters. Some regions of the disparity map may be generated with denser disparity information, and the remaining regions of the disparity map may be generated with sparser disparity information.

[0069] At 522, the system can perform detection and recognition using the generated disparity map.

[0070] At 524, the system can determine whether it has arrived at the destination. If it has not arrived at the destination, the process can return to 504 to receive the next pair of stereo images from the camera. If it has arrived at the destination, the process can return to 502 and wait until a new destination is indicated.

[0071] FIG. 6 shows an example of generating a history map 210 according to some implementations. In some cases, obstacle and / or road anomaly detection may be used to determine which regions of the image to select for denser disparity processing. However, some problems with using only obstacle and road anomaly detection include that the reasons for generating regions with higher priority are not limited to obstacles and road anomalies. For example, when the vehicle is approaching an exit ramp or entrance ramp of a highway, it may be desirable to utilize dense disparity information for these regions of interest. As another example, if only detected obstacles are used to determine high-density regions in a high-traffic situation, this may result in a dense disparity map being generated for substantially the entire image, potentially significantly increasing the processing load on the processor. Therefore, rather than relying solely on obstacle and road anomaly detection, in the implementations herein, the history map 210 can be used to determine regions with higher priority for selection within the image to generate denser disparity information.

[0072] In the example of FIG. 6, the history map 210 can be generated based at least in part on information received from the history database 250 described above in connection with FIGS. 2 and 3. For example, as described above, the history database 250 can be generated from information collected from a plurality of vehicles that have traveled on the same road as the road on which the vehicle is currently traveling. In this example, the history database 250 includes a database 252 of driver monitoring data, a vehicle light operation database 254, a steering wheel position database 255, and a past and current traffic direction and presence database 256. For example, the database 252 of driver monitoring data can include areas of the road that another driver may have looked at at a particular point in time while traveling on the road.

[0073] The vehicle light operation database 254 can indicate whether a brake was used and / or whether a turn signal was used while another driver was operating a vehicle on the road. The steering wheel position database 255 can indicate whether the steering wheel was turned at a particular position while another driver was operating a vehicle on the road. Additionally, the real-time actions of the driver can also be monitored and used for decision-making. For example, the movement of the steering wheel of the host vehicle and / or the operation of the vehicle lights can be monitored and received as real-time input. The past and current traffic direction and presence database 256 can indicate traffic patterns, traffic flow, traffic jams, etc., encountered by other drivers while traveling on the road, as well as the current traffic position. Further, a road anomaly database 260 can indicate any road anomalies detected by a plurality of vehicles in the past on the road on which the vehicle 102 is currently traveling. Examples of road anomalies can include potholes, speed bumps, other bumps, and other types of irregular surfaces. Additionally, any currently detected obstacles and currently detected road anomalies can be tracked and taken into account in real-time by a recognition program.

[0074] Furthermore, when generating the history map 210, the recognition program may also be configured to take into account inputs received from a driver monitoring system installed in the vehicle that can indicate the current line of sight of the driver of the vehicle 102 or other real-time characteristics, as well as the current steering wheel position information, current vehicle light operation information, etc. Accordingly, the recognition program 128 can take into account road anomaly database information 602, road anomaly information 603 detected in real time, current and past driver monitoring information 604, current and past vehicle light operation information 606, current and past steering wheel position information 607, current and past traffic data 608, weather information 610, obstacles tracked in real time 612, current position information 614, and destination information 616. By taking into account all of this information, the recognition program 128 can calculate the regions and parameters to be used for creating the disparity map, as shown at 618.

[0075] FIG. 7 shows an example of area selection 700 based at least on a history map according to some implementations. In this example, assume that the left image 408 and the right image 406 described above in connection with FIG. 4 are received for processing. Further, the recognition program can receive information from the history map 210 described above in connection with FIG. 6 as input in order to determine a priority area in at least one of the received images. For example, each of the databases 252, 254, 255, 256 within the history database 250 and the road anomaly database 260 includes a list of detected / recognized features along with the position information of each corresponding feature. Based on the current position of the vehicle, for each frame (i.e., each received image), the content of each database can be accessed with respect to that position. For example, if there is a road anomaly indicated by the road anomaly database 260 and / or real-time road anomaly detection, subsequently, the distance from the current vehicle position to the road anomaly can be determined. For example, the size of the road anomaly can also be determined from the road anomaly database and / or recognition information, along with the priority. The distance and size information can be converted into image coordinates. The area corresponding to the road anomaly can be labeled according to the value of the priority level, as will be further described below in connection with FIG. 8, for example.

[0076] In the illustrated example, the history map information may include current position information 614, road anomaly information 602 and 603, past and current driver monitoring information 608, vehicle light operation 606, steering wheel position information 607, weather information 610, tracked obstacles 612, and destination information 616. Thus, the history map information can indicate the presence of a depression as shown at 702, a right line of sight and a left barrier as shown at 704, the presence of a tunnel as shown at 706, and one or more tracked vehicles 708. Based on this information, the recognition program can configure the system to select some higher-priority regions in at least one of the images. Thus, the system can select the region 710 corresponding to the depression 712 as a higher-priority region. Further, the system can select the region 714 corresponding to the tunnel 716 as a higher-priority region. Further, the system can select the region 718 corresponding to the tracked vehicle 720 as another higher-priority region. Further, the system can select the left region 722 corresponding to the concrete road barrier 724 as another lower-priority region (shown by cross-hatching), and can also select the right region 726 as another lower-priority region (similarly shown by cross-hatching). The non-selected regions of the remaining images can also be shown as lower-priority regions. Additionally, in this example, the priority regions 710, 714, and 718 are shown as rectangles, but the implementations herein are not limited to any particular shape for the priority regions.

[0077] FIG. 8 includes a flowchart showing an exemplary process 800 for determining selected regions according to some implementations. In some examples, process 800 can be performed by the systems 100, 200, and / or 300 described above, for example, by execution of the recognition program 128. In some cases, process 800 can correspond in part to block 514 of FIG. 5 described above.

[0078] At 802, the system can receive region, feature display, and other parameters from the history map.

[0079] At 804, the system can receive position information indicating the current position of the vehicle based on information received from, for example, a GPS receiver.

[0080] At 806, the system can obtain information from the road anomaly database based on the position information.

[0081] At 808, the system can determine distance and dimension information regarding various features identified by the history map and road anomaly information.

[0082] At 810, the system can perform a conversion from 3D to image coordinates to determine the distance from the vehicle to various identified features within the image.

[0083] At 812, the system can label the regions corresponding to the identified features. For example, in some cases, the system may optionally label each of the regions, for example, according to the type of feature. To perform the labeling, the system may refer to a priority level table 811 that can indicate the type of features such as vehicles, small obstacles, traffic lights, traffic signs, tunnels / low illumination areas, etc. Table 811 may further indicate the proximity of each feature, for example, near, intermediate, or far, and the corresponding priority level for each feature with respect to the proximity as a decimal value, for example, 1 - 15.

[0084] At 814, the system can provide the region information to the following algorithm to perform selection of image quality and selection of compression as further described below.

[0085] FIG. 9 is a flow diagram showing an exemplary process 900 for determining bit depth and compression level according to some implementations. In some examples, process 900 can be performed by the systems 100, 200, and / or 300 described above by executing the recognition program 128. For example, after a priority area is selected, the system can determine an optimal bit depth and corresponding image compression level for each selected priority area. In particular, in a camera system, an image sensor such as a charge-coupled device (CCD) or an active pixel sensor (CMOS) can receive light through a lens or other optical components. The sensor can transfer information about the received light to the next stage either as a voltage signal or a digital signal. For example, a CMOS sensor can use an analog-to-digital converter (ADC) to convert photons to electrons, then to a voltage, and subsequently to a digital value. The digital value can be stored at a specified bit depth.

[0086] Conventionally, the bit depth of pixels in an ADC can be predefined (i.e., constant). For example, either a higher bit depth (e.g., 16-bit) can be selected for the entire image, or in some cases, a lower bit depth (e.g., 8-bit) can be selected for the entire image. In images with a lower bit depth, the memory requirements are lower, and the density of the generated disparity map may also be lower. On the other hand, a denser disparity map can be generated using an image with a higher bit depth, but more memory is also required for each image.

[0087] Furthermore, the images in this specification may be compressed. However, depending on the compression algorithm, information may be lost, which may not be desirable for regions with higher priorities. Therefore, the system in this specification can determine the optimal compression level and the corresponding compression algorithm based on at least the priority levels indicated for each region and use them for individual selected regions. In some cases, a compression priority table may be used to determine the optimal compression algorithm for each selected region.

[0088] In 902, the system may receive a selected region and information indicating the priority of each corresponding selected region determined based on, for example, the process 800 of FIG. 8.

[0089] In 904, the system may select and process one of the selected regions.

[0090] In 906, the system may determine the bit depth, for example, by referring to a bit depth priority table, and use it for the selected region being processed. An exemplary bit depth priority table will be described below in relation to FIG. 10.

[0091] In 908, the system may perform analog-to-digital conversion on the selected region using pixel-level analog-to-digital conversion. An example will be described below in relation to FIG. 11.

[0092] In 910, the system may determine the image compression level and the corresponding compression algorithm to be used for the selected region and compress the selected region.

[0093] In 912, the system may determine whether all selected regions have been compressed. If they have been compressed, the process proceeds to 912. Otherwise, the process returns to 904 to select the next selected region for processing.

[0094] In 914, when all regions of an image are compressed, the compressed image can be provided to a disparity and parallax estimation algorithm that is further described below in connection with, for example, FIGS. 15 - 22.

[0095] FIG. 10 shows an exemplary bit - depth priority table 1000 according to some implementations. For example, the system can select a bit depth for each pixel using the selected region and the bit - depth priority table 1000. In this example, the bit - depth priority table 1000 includes information indicating the bit depth of each selected region that is selected depending on, for example, the type of feature and the distance of the feature from vehicle 102 based on one or more current scenarios. For example, in a shadow or low - illumination scenario (e.g., a tunnel), since only a narrower range of brightness and darkness is available for disparity estimation, it can be extremely difficult to generate an accurate disparity map using a low pixel bit - depth such as 8 bits. To address this, by adding a range of brightness and darkness with an image of a high pixel bit - depth (10 - 12 bits), it may be possible to correct deep brightness and / or highlight displays to generate an accurate disparity map.

[0096] Similarly, for features far away (i.e., distant regions) such as small obstacles, a higher pixel bit - depth such as 10 bits to 12 bits may be used. Further, although the examples in this specification are described using bit - depths of 8 bits, 10 bits, and 12 bits, the implementations in this specification are not limited to any specific value for the bit - depth, and any of a variety of bit - depth values can be applied to other pixel bit - depth settings. For example, the pixel bit - depth can be changed based on the capabilities of the ECU, for example, to increase disparity accuracy and efficiently use available resources. In the example of FIG. 10, the bit - depth priority table includes the type of feature 1002 and the position 1004 of features such as distant 1006, intermediate 1008, and near 1010.

[0097] FIG. 11 shows an example 1100 of pixel-level analog-to-digital conversion according to some implementation forms. Some examples in this specification may employ an image sensor 1102 having a pixel-level analog-to-digital converter (ADC). The image sensor 1102 includes a plurality of pixel blocks (PB) 1104 arranged in rows and columns and communicating with a row selector 1106 and a column selector 1108.

[0098] As shown in an enlarged example 1110 of one of the pixel blocks 1104, each pixel block 1104 within the image sensor 1102 includes its own ADC 1112, which converts an analog signal value (e.g., voltage) into a digital signal value. For example, a pixel 1114 of the pixel block 1104 can receive light, and an amplifier 1116 associated with the pixel 1114 can amplify the electrical signal generated based on the received light. An analog correlated double sampling (CDS) circuit 1118 can sample the signal twice and reduce noise to send the signal to the ADC 1112.

[0099] The bit depth in each ADC 1112 for each pixel 1114 can be controlled by a multiplexer 1120 based on an input 1122 of the received pixel bit depth. For example, as described in connection with FIGS. 9 and 10, the recognition program 128 can determine a priority level for each pixel within each region of the image sensor, and based on the bit depth priority table 1000 described above in connection with FIG. 10, dynamically set the bit depth for each pixel block 1104 within the image sensor 1002. Thus, the bit depth can be set such that the next acquired image has a specific pixel-level bit depth for each pixel with respect to the region of the image sensor corresponding to the region of the image where the priority level has been determined. For example, not only the bit depth of the grayscale but also the color channels can be dynamically specified, for example, based on the type, distance, and position of the features.

[0100] By using a lower bit depth for regions of an image with a lower priority and a higher bit depth for regions of the same image with a higher priority, it is possible for regions of lower priority, and thus the entire image, to consume less memory, while regions of higher priority still have a large amount of image information for generating more detailed disparity information. After analog-to-digital conversion in ADC1112, the signal can be sent to digital CDS1124 and subsequently output to memory. After the values of all pixel blocks 1104 have been converted to digital values, the image can be stored in memory and appropriate image compression can be applied to the image based on the selected priority regions, as will be further described below.

[0101] FIG. 12 shows an exemplary image 1200 with different bit depths assigned to regions with different priorities, according to some implementations. In this example, assume that regions with a lower priority are assigned an 8-bit bit depth, regions with an intermediate priority are assigned a 10-bit bit depth, and regions with a higher priority are assigned a 12-bit bit depth. Further, the image includes displays of the assigned distances, namely, far 1202, middle 1204, and near 1206. Thus, as shown at 1208, regions of the image with a lower priority are given an 8-bit bit depth, as indicated by the first cross-hatching style. Further, as shown at 1210, regions of the image 1200 with an intermediate priority are given a 10-bit bit depth, as indicated by the second cross-hatching style. Further, as shown at 1212, regions of the image 1200 with a higher priority are given a 12-bit bit depth, as indicated by the third cross-hatching style.

[0102] For example, the depression 1214 previously identified as being associated with a region of higher priority is included in a portion of the image 1200 generated with a 12-bit bit depth. Further, in this example, 8-bit, 10-bit, and 12-bit are used as the relative bit depths, but in other examples, higher or lower bit depths may be used for various different levels of priority. Accordingly, the implementations herein are not limited to any particular bit depth for any particular level of priority.

[0103] FIG. 13 shows an exemplary compression priority table 1300 according to some implementations. For example, the compression priority table 1300 may include a type of feature 1302 and a location 1304 of a feature that may include a far 1306, middle 1308, and near 1310. Each distance level, far 1306, middle 1308, and near 1310, may include a priority level 1312 and a corresponding image compression quality level 1314.

[0104] Examples of features can include vehicles, small obstacles, traffic lights, traffic signs, and / or tunnels or other low-light areas. For example, portions of an image having features that are far away and thus more difficult to detect can be compressed using a low-lossy or lossless compression algorithm, while portions of an image having features that are close by and thus easier to detect within the image can be compressed using, for example, a medium-lossy or lossy compression algorithm based on the type and distance of the feature.

[0105] An example of a compression algorithm is shown in 1316. For example, the reversible compression algorithm may be a PNG (Portable Network Graphics) reversible algorithm, while various levels of JPEG (Joint Photographic Experts Group) compression algorithms can be used to provide low-lossy compression levels, medium-lossy compression levels, or lossy compression levels. Further, although some exemplary compression algorithms are described herein, the limitations herein are not limited to any particular compression algorithm.

[0106] FIG. 14 shows an exemplary image 1400 including various regions selected for compression using various compression levels according to some implementations. Once the image is generated, image compression may be applied to make it easier to store and transmit the image. For example, depending on the selected compression level, image compression can reduce the data size and obtain a controllable effect on the image quality. As an example, in color space conversion, when one color is input, it can be decomposed into red, green, and blue components or YCbCr (luminance, chrominance blue, chrominance red) components and stored individually. The mapper can convert the pixel values of the input image into inter-pixel coefficients. Using quantization can reduce the number of possible values of the quantity, thereby reducing the number of bits required to represent the quantity. Further, entropy encoding can be used to represent the quantized coefficients as compactly as possible.

[0107] In some examples herein, the above-described compression priority table 1300 can be used to select an image compression algorithm and a compression level. For example, the compression priority table 1300 can indicate the compression algorithm and compression level to be used to compress an image based on the type of feature, the distance from the vehicle to the feature, and the current traffic scenario.

[0108] The image compression rate may have some impact on the image and the quality of the colors in the image, and then it may affect the accuracy of the disparity map. For example, the detected features located in the distant region may have very few pixels that can be used to calculate the disparity map with high accuracy. Therefore, when lossy image compression is used to reduce memory storage, the image quality and subsequent disparity information may also be affected. For example, as shown in 1402, the vehicle in the distant region may be diffused and blend into the background, and an inaccurate disparity map may be generated. On the other hand, the implementation forms herein can use lossless image compression for such regions of the image 1400, while a high compression rate can be used for larger features in the near-distance region and parts of the image without regions of interest. Therefore, based on the compression priority table, an appropriate image compression level can be selected to improve system performance without increasing the storage memory.

[0109] In the example of FIG. 14, as described above, low lossy compression or lossless compression can be used for the selected region B shown at 1402, and can also be used for the selected region A shown at 1404 including the depression. Further, a medium lossy compression level can be used for the selected region D shown at 1406 including another vehicle. A high lossy compression level can be used for the rest of the image including regions E and C.

[0110] FIG. 15 is a flow diagram illustrating an exemplary process 1500 for determining an optimal search range according to some implementations. After the image is compressed, a disparity parameter can be determined to generate a disparity map. The disparity information can be determined using either a global method or a local method. An example of a local method includes block matching, in which the processor searches for the region in one image of a pair of images that best corresponds to a template in the other image of the pair. The template can be continuously shifted along the epipolar line (i.e., the scan line) within a specified disparity search range.

[0111] The disparity search range is a parameter that can affect both the disparity accuracy and the processing load. Conventionally, the disparity search range may be fixed for the entire image or a pre-defined region and may be constant regardless of the traffic scenario, detected features, etc. For example, in a pre-defined disparity search, accurate disparity information for near and mid-range features may be obtained, but the accuracy may decrease in the far-distance region, and it may not be possible to find a good block match in the search process.

[0112] To provide a solution to the above problems, in some examples herein, a dense disparity map and edge map information determined from the previous frame image can be used to use a dynamic and optimal search range in which the search range is dynamically determined for each selected region. This technique not only achieves a reduction in processing time but also frees up more resources that can be used for, for example, disparity estimation. In some examples, process 1500 can be executed by the systems 100, 200, and / or 300 described above, for example, by the execution of recognition program 128, and can correspond to block 518 of FIG. 5.

[0113] At 1502, the system may receive an edge map output of an image (e.g., frame N-1) that preceded the currently processed image. For example, the previous frame may precede the current image within a threshold period.

[0114] At 1504, the system may receive detected feature information of an image (e.g., frame N-1) preceding the current image.

[0115] At 1506, the system may select an area within the previous image to determine the optimal disparity search range.

[0116] At 1508, the system may access dynamically dense disparity map information regarding the selected area within an image (e.g., frame N-1) preceding the current image, or otherwise determine that information.

[0117] At 1510, in some examples, the system may downsample the selected disparity map area.

[0118] At 1512, the system may calculate a histogram for the selected disparity area determined based on the edge map information, detected feature information, and dense disparity map information of the image preceding the current image.

[0119] At 1514, the system may determine the minimum and maximum search ranges based on the histogram.

[0120] At 1516, the system may determine the optimal range for the selected area within the current image based on the minimum and maximum search ranges determined for the previous image.

[0121] FIG. 16 shows an example 1600 for determining dynamically dense parallax information regarding a selected region of an image according to some implementations. For example, FIG. 16 can correspond to at least blocks 1502-1508 of FIG. 15. In particular, the dynamic parallax map 1602 of the image of frame N-1, the edge map 1604 of frame N-1, and the detected feature information image 1606 of frame N-1 can be used to determine the parallax map of the detected vehicle within the selected region of the image of frame N-1 as shown at 1608. For example, the selected region can correspond to region 1610 shown within the image 1606 of frame N-1.

[0122] In some cases, for frame N-1, a fixed parallax range can be selected for the entire image, and the parallax map 1602 can be estimated. Next, based on the information regarding the previously detected features in the previous image 1606, and based on the edge map 1604 of the previous image and the parallax map of the previous image, a dynamic optimal search range estimation for the current image can be performed. For example, the edge map 1604 can be used to finely adjust the selected region to make the selected region very compact. Once the region is selected, the parallax map 1608 of the selected region may, in some cases, be downsampled, while in other cases, the parallax map 1608 may be sent to the next stage for histogram generation without being downsampled. The decision regarding whether to downsample can depend at least in part on the processing capabilities of the processor performing the processing.

[0123] FIG. 17 shows an example of generating a histogram 1700 from a disparity map of a selected region of frame N-1 according to some implementations. For example, a histogram 1700 can be generated using a selected portion of the previous image, for example, the disparity map 1608 determined in FIG. 16 above. The histogram 1700 can represent the number of pixels corresponding to each disparity (parallax) value within the disparity range 1702. The histogram 1700 can be used to determine an optimal search range and can be used to determine disparity information regarding the current image N following the previous image N-1.

[0124] FIG. 18 shows an example 1800 for determining an optimal search range for a plurality of regions according to some implementations. For example, a plurality of histograms 1700, for example, one histogram may be generated for each region of interest within the current image being processed. Once the histogram is generated, the maximum (end) and minimum (start) ranges can be estimated and can be shown to be the optimal search range (OSR) for the corresponding region of the current image 1802. In the exemplary histogram 1700, based on the pixel number information of the histogram, the start search range 1804 is determined to be 4 and the end search range 1806 is shown to be 40. Thus, the OSR 1808 determined from the histogram 1700 is 4 to 40, which is based on frame N-1.

[0125] The OSR 1808 is determined with respect to the previous frame N-1. Thus, predicted feature information can be used to estimate the OSR for the current frame (N). For example, if the relative speed of the detected vehicle is greater than zero, the end search range can be reduced, or if the relative speed of the detected vehicle is less than zero, it can be expanded. In this example, assuming that the relative speed of the detected vehicle is less than zero, the OSR for the current frame is not 4 to 40 but 4 to 42. Next, when the OSR is estimated using the predicted region, as shown in FIG. 27, it is used to calculate the disparity map for the current frame.

[0126] Similarly, for the remaining regions of interest in the image, the OSR can be estimated based on calculating an additional histogram for the selected region of frame N-1. If the selected region (row * column = m × n) is larger than a predefined threshold "Th", a default "m × n" block is used (e.g., a road region). For example, m × n is selected based on the capabilities of the ECU. Thus, in image 1802, region 1812 is determined to have an OSR of 4 to 42 as described above. Further, region 1814 has an OSR of 1 to 5, region 1816 has an OSR of 1 to 18, region 1818 has an OSR of 60 to 80, and region 1820 has an OSR of 70 to 80.

[0127] Figure 19 is a flowchart showing an exemplary process 1900 for adaptive window-based stereo matching according to some implementations. In some examples, process 1100 can be executed by the systems 100, 200, and / or 300 described above by executing a recognition program 128.

[0128] Once the optimal disparity search range is estimated for each region of interest in the image, various cost metrics can then be selected according to the processing capabilities of the vehicle computing device to find the best match. As an example, the determination of the sum of absolute differences (SAD) may require relatively less processing time, but the accuracy of the similarity measurement value determined using this technique may also be low. Thus, in some examples herein, the sum of zero-mean absolute differences (ZSAD) may be used. ZSAD is a technique improved for similarity matching and may use more processing time than SAD, but can provide a more accurate similarity measurement value compared to the similarity measurement value of SAD.

[0129] Furthermore, the accuracy of the similarity measurement values as a whole may also depend on the size of the matching window used. For example, if the window size selected for block matching is too small, the process may generate a noisy disparity map. On the other hand, if the window size is too large, the resulting disparity map may be too smooth, for example, due to edges spreading into the background, and provide little useful information.

[0130] To overcome the above problems, some examples in this specification include a novel adaptive window-based ZSAD stereo matching (disparity map) technique. For example, first, a color similarity threshold and an arm length threshold can be selected, and the arm length threshold can play a role based on the priority of the selected region. Using the above parameters, an adaptive window can be generated separately for the right and left images. Next, the ZSAD value can be calculated using the estimated OSR range for each region. Once the ZSAD is calculated over the entire disparity optimal search range (OSR), the minimum value can be calculated and used as the low-resolution disparity value. Next, in some cases, sub-pixel disparity can also be estimated according to the desired resolution.

[0131] In 1902, the system may receive information about the selected region regarding the region of interest previously selected within the image being processed.

[0132] In 1904, the system may estimate the arm length and color similarity based on each of the arm length threshold and the color similarity threshold.

[0133] In 1906, the system may select adaptive window parameters for the selected region.

[0134] In 1908, the system may generate an adaptive window.

[0135] In 1910, the system may receive an estimated parallax parameter.

[0136] In 1912, the system may estimate a matching cost.

[0137] In 1914, the system may select a disparity value, in some examples, a sub-pixel disparity estimate.

[0138] Figure 20 shows an exemplary matching window size table 2000 according to some implementations. For example, after a priority region is determined based on the region selection process described above in connection with FIGS. 5-8, a matching window arm length threshold (window size) can be selected based on, for example, the type of feature and the distance of the feature. For example, an appropriate matching window size may be selected using the matching window size table 2000. In this example, the matching window size table 2000 includes the type of feature 2002, the position of the feature 2004, and the horizontal matching window size 2008 for various different distances such as far 2008, middle 2010, and near 2012.

[0139] Figure 21 shows an exemplary color difference threshold table 2100 according to some implementations. In this example, for example, after a priority region is determined based on the region selection process described above in connection with FIGS. 5-8, the shape of the matching window can be determined using the color difference threshold table 2100, such as based on the position of the feature. In this example, the color difference threshold table 2100 may include positions of features such as maximum and minimum color differences 2102, far 2106, middle 2108, or near 2110. For each position, a minimum value 2112 and a maximum value 2114 can be specified.

[0140] FIG. 22 shows an example 2200 of an adaptive window 2202 according to some implementations. For example, the adaptive window 2202 can be defined based on pixels included in or excluded from the adaptive window 2202. For example, given a plurality of pixels 2204, the adaptive window can have a horizontal arm 2206 and a vertical arm 2208 each having an arm length. The adaptive window 2202 further includes a boundary 2210 corresponding to the boundary of the pixels included in the adaptive window 2202. For example, the size of the horizontal arm 2206 can be defined based on a number of pixels from p to p n in the horizontal direction, and the vertical arm can be defined similarly. In some cases, the plurality of pixels 2204 may correspond to a specified size of a block matching window, while the adaptive window 2202 can be defined within the block matching window.

[0141] FIG. 23 shows an example 2300 of adaptive window generation according to some implementations. This example shows a schematic diagram of generating a matching window in a matrix format of 2302 and a corresponding image 2304 having some exemplary adaptive windows 2306(1), 2306(2), and 2306(3) at several different positions within the image 2304. For example, the matching window 2308 represented by the matrix 2302 can be, in some examples, 9 pixels high × 11 pixels wide where the values within the matrix 2302 represent gray levels or intensity values. In some cases, to simplify the calculation, the vertical dimension of the matching window may be fixed. However, in other examples, the vertical dimension can also be made variable using the same procedure used to calculate the size of the horizontal window. For example, as described above in connection with FIGS. 19-21, once the window size and color difference threshold are selected, an adaptive window 2310 having an adaptive shape and adaptive size can be estimated for one of the images in an image pair, such as the left image, using equations (3), (4), and (5) as follows.

Mathematics

Mathematics

Mathematics

[0142] As shown by 2312, the pixel under consideration (the pixel for which parallax needs to be calculated) is located at column n th and row m th Using the above equations, the length of the horizontal arm (e.g., left and right) of row m th can be calculated. For example, each pixel under consideration can be compared with adjacent pixels. If all equations are true, the arm length can be increased. This process can be repeated for each row in both the up and down directions as long as each row satisfies equations (3), (4), and (5).

[0143] FIG. 24 shows an example 2400 of using an adaptive window to determine the ZSAD value according to some implementations. In this example, ZSAD can be used as a sample cost metric technique for measuring similarity or matching cost. Additionally, or alternatively, other matching cost techniques may be applied. Different from the conventional systems that use a fixed-size matching window, in some examples herein, the above-described adaptive window can be used to estimate the matching cost regardless of the situation or AD / ADAS application.

[0144] As an example, after the adaptive window is calculated, the same adaptive window can be used for both the left image and the right image, and the ZSAD value can be calculated using the following equation (6) until the estimated OSR is reached.

Equation

[0145] Thus, in the illustrated example, the left image 2402 corresponds to a matching window 2404 represented as a 9×11 matrix, and the right image 2406 of the same scene may correspond to a matching window 2408 represented as another 9×11 matrix. The adaptive window 2410 corresponding to the reference block 2412 can be used, for example, for block matching along the scan line 2414 to determine the disparity information within the optimal search range 2416 as described above. For example, the adaptive block 2410 may correspond to the pixels within the matrix 2404, while the reference block 2412 may correspond to the pixels within the matrix 2408.

[0146] FIG. 25 shows an example 2500 of determining the matching cost over an optimal search range according to some implementations. In this example, after the ZSAD value is calculated over the optimal search range, the minimum value can be estimated and used as the sparse disparity value of the optimal search range. For example, assume that the minimum value of the optimal search range is 4 and the maximum value is 42, as shown by 2502 and 2504 respectively. For example, as described above, a plurality of ZSAD values 2506 can be calculated over the optimal search range. In this example, assume that the value shown by 2508 is the smallest value calculated over the search range, i.e., C2 in this example, and C1 and C3 are adjacent values. Therefore, the minimum ZSAD value 2508 may indicate the minimum matching cost value.

[0147] FIG. 26 shows an example of the selected resolution determined for the region within the image 2600 according to some implementations. In this example, after the minimum matching value is determined, sub-pixel information can also be calculated using Equation (7) according to the resolution.

Number

[0148] In some examples, the resolution of the disparity map can be selected for the selected regions A to E described above in connection with FIGS. 5 to 8, for example, using a disparity resolution table that includes the resolution specified for the type and distance of the feature as described below in connection with FIG. 27. Once the resolution is selected, the disparity map can be calculated using the estimated disparity parameters described above, such as the matching window size, color difference threshold, optimal search range, and matching cost method (e.g., ZSAD or SAD using an adaptive window).

[0149] In this example, the selected disparity resolution for each of the selected regions A to E is shown in 2602. For example, regions A and B have a dense disparity of 1×1 pixel as regions with a higher priority, region E has a sparse disparity of 1 / 4 downsampled, and regions C and D have a sparse disparity of 1 / 2 downsampled in the middle.

[0150] FIG. 27 shows an exemplary disparity resolution table 2700 according to some implementations. For example, the disparity resolution table 2700 can be referred to by the recognition program 128 to determine the disparity resolution to be applied to each of the selected regions of the image. In this example, the disparity resolution table 2700 includes the feature type 2702, the feature position 2704, and the disparity resolution 2706, and is applied to various different distances such as far 2708, middle 2710, and near 2712.

[0151] FIG. 28 shows an exemplary dynamically reconfigured disparity map or disparity image 2800 according to some implementations. In this example, the system can generate a disparity map or disparity image that includes denser disparity regions and less dense disparity regions determined in real time based on the techniques described above. For example, the system can calculate denser disparity information only for regions with higher priority (e.g., regions A and B in this example), while in other regions with lower priority, the disparity information may be calculated at a lower density (lower resolution).

[0152] In this example, the disparity resolution for each selected region A - E is shown at 2802. For example, regions A and B, as regions with higher priority, have a dense disparity of 1×1 pixels, while region E has a sparse disparity that is 1 / 4 downsampled, and regions C and D have a sparse disparity that is 1 / 2 downsampled in the middle. Thus, in the implementations herein, higher precision recognition can be provided at higher priority positions while reducing the amount of processing resources required to do so.

[0153] The generated dynamically reconfigured disparity map 2800 can be used to detect obstacles and other features, as well as the respective distances and / or positions of the features. The detected information and any results can be uploaded to a storage location on the network, such as the network computing device 262 described above in connection with FIGS. 2 and 3, and can be used to update a database of historical data and / or a database of road anomalies. This updated information can be continuously used by the vehicle and other vehicles in future historical maps.

[0154] The exemplary processes described herein are merely process examples provided for purposes of discussion. In light of the disclosure herein, numerous other variations will be apparent to those skilled in the art. Further, although the disclosure herein describes some examples of suitable frameworks, architectures, and environments for executing a process, the implementations herein are not limited to the specific examples illustrated and described. Additionally, the disclosure provides various exemplary implementations as described herein and as shown in the drawings. However, the disclosure is not limited to the implementations described and illustrated herein and can be extended to other implementations as would be known to or become known to those skilled in the art.

[0155] The various instructions, processes, and techniques described herein can be considered in the general context of computer-executable instructions, such as computer programs and applications stored on a computer-readable medium and executed by a processor herein. Generally, the terms program and application may be used synonymously and may include instructions, routines, modules, objects, components, data structures, executable code, etc. for performing a particular task or implementing a particular data type. These programs, applications, etc. may be executed as native code or may be downloaded and executed in a virtual machine or other just-in-time compilation execution environment, etc. Generally, the functions of the programs and applications may be combined or distributed as desired in various implementations. Implementations of these programs, applications, and techniques may be stored on a computer storage medium or transmitted via some form of communication medium.

[0156] The subject matter is described in a language specific to structural features and / or methodological operations, but it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or operations described. Rather, the specific features and operations are disclosed as exemplary forms of carrying out the claims.

Claims

1. Comprising one or more processors configured to perform processing by executable instructions, The processing is as follows, Receiving, by the one or more processors, at least one image including a road from at least one camera of a vehicle, Receiving, by the one or more processors, vehicle position information including an indicator of the position of the vehicle, Receiving, by the one or more processors, at least one of historical information from a historical database, or abnormal information of a road determined from at least one of a road abnormal database or detected road abnormal information, Selecting, in the at least one image, at least one region with a higher priority and at least one region with a lower priority based at least in part on at least one of the historical information or the road abnormal information, Making the bit depth of the at least one region with a higher priority higher than the bit depth of the at least one region with a lower priority, or compressing the at least one region with a lower priority at a compression rate higher than the compression rate used to compress the at least one region with a higher priority, and Generating, by the one or more processors, at least one of a disparity map or a disparity image based on the at least one image, and an indicator of the position of the vehicle, and further based on the at least one region with a higher priority and the at least one region with a lower priority selected at least in part on at least one of the historical information or the road abnormal information, The historical information includes a system including at least one of current detected driver monitoring information, past driver monitoring information, current vehicle light operation, past vehicle light operation, past steering wheel position information, current steering wheel position information, past traffic information, or current traffic information.

2. The processing segments, prior to the at least one image, one previous image of one frame received previously into at least one region with a higher priority and at least one region with a lower priority, based at least in part on at least one of a disparity map from the previous image or a disparity image from the previous image, and an edge map of the previous image. Determining histograms for the at least one region with a higher priority and the at least one region with a lower priority, and The system according to claim 1, further comprising determining a search range for searching the at least one image based at least in part on the histogram.

3. The processing further includes determining disparity information for the at least one image by applying a first disparity resolution to the at least one region with a higher priority and applying a second different disparity resolution to the at least one region with a lower priority.

4. The processing further includes determining an adaptive window for use in block matching to determine disparity information for the at least one image, Determining the adaptive window is based at least in part on comparing at least one of the grayscale values or color values of the pixels within the adaptive window with at least one of the grayscale values or color values of adjacent pixels. The system according to claim 1.

5. The processing includes recognizing one or more features within at least one of the disparity map or the disparity image to determine recognition information, and Further includes transmitting at least one control signal based on the recognition information, The at least one control signal is An instruction for controlling the vehicle to accelerate or decelerate the vehicle, An instruction for controlling the vehicle to steer the wheels of the vehicle, or The system according to claim 1, including at least one of an instruction for presenting a warning.

6. Receiving, by one or more processors, at least one image including a road from at least one camera of a vehicle. Receiving, by the one or more processors, vehicle position information including an indicator of the position of the vehicle; Receiving, by the one or more processors, at least one of historical information from a historical database, or at least one of road anomaly information determined from at least one of a road anomaly database or detected road anomaly information; Selecting, in the at least one image, at least one region of higher priority and at least one region of lower priority based at least in part on at least one of the historical information or the road anomaly information; Performing at least one of making the bit depth of the at least one region of higher priority higher than the bit depth of the at least one region of lower priority, or compressing the at least one region of lower priority at a compression rate higher than the compression rate used to compress the at least one region of higher priority, and Generating, by the one or more processors, at least one of a disparity map or a disparity image based on the at least one image, and an indicator of the position of the vehicle, and further based on the at least one region of higher priority and the at least one region of lower priority selected at least in part on at least one of the historical information or the road anomaly information; The historical information includes at least one of current detected driver monitoring information, past driver monitoring information, current vehicle light operation, past vehicle light operation, past steering wheel position information, current steering wheel position information, past traffic information, or current traffic information. Segmenting, prior to the at least one image, a previous image of one frame received into at least one region of higher priority and at least one region of lower priority based at least in part on at least one of a disparity map from the previous image or a disparity image from the previous image, and an edge map of the previous image; Determining a histogram for the at least one region of higher priority and the at least one region of lower priority, and The method according to claim 6, further comprising determining a search range for searching for the at least one image based at least in part on the histogram.

8. The method according to claim 6, further comprising determining disparity information for the at least one image by applying a first disparity resolution to a region of higher priority of the at least one priority and applying a second different disparity resolution to a region of lower priority of the at least one priority.

9. The method according to claim 6, further comprising determining an adaptive window to be used for block matching to determine disparity information for the at least one image, wherein determining the adaptive window is at least partially based on comparing at least one of the grayscale values or color values of the pixels within the adaptive window with at least one of the grayscale values or color values of adjacent pixels. The method according to claim 6.

10. One or more non-transitory computer-readable media storing instructions executable by the one or more processors to configure the one or more processors to perform processing, the processing comprising receiving at least one image including a road from at least one camera of a vehicle, receiving vehicle position information including an indication of the position of the vehicle, receiving at least one of historical information from a historical database, or road anomaly information determined from at least one of a road anomaly database or detected road anomaly information, selecting, within the at least one image, at least one region of higher priority and at least one region of lower priority based at least in part on at least one of the historical information or the road anomaly information, increasing the bit depth of the at least one region of higher priority to be higher than the bit depth of the at least one region of lower priority, or compressing the at least one region of lower priority at a compression rate higher than the compression rate used to compress the at least one region of higher priority, and generating at least one of a disparity map or a disparity image based on the at least one image, and an indicator of the position of the vehicle, and further based on at least one of the history information or the road anomaly information, and on a higher priority region and a lower priority region of the at least one priority selected at least partially based thereon, the history information includes at least one of current detected driver monitoring information, past driver monitoring information, current vehicle light operation, past vehicle light operation, past steering wheel position information, current steering wheel position information, past traffic information, or current traffic information, on one or more non-transitory computer-readable media. **Claim 11** the processing includes segmenting one previous image of one frame received prior to the at least one image into a higher priority region of at least one priority and a lower priority region of at least one priority, based at least in part on at least one of a disparity map from the previous image or a disparity image from the previous image, and an edge map of the previous image, determining a histogram for the higher priority region of the at least one priority and the lower priority region of the at least one priority, and further including determining a search range for searching the at least one image based at least in part on the histogram, for the one or more non-transitory computer-readable media of claim 10. **Claim 12** the processing further includes determining disparity information for the at least one image by applying a first disparity resolution to the higher priority region of the at least one priority and a second different disparity resolution to the lower priority region of the at least one priority, for the one or more non-transitory computer-readable media of claim 10. **Claim 13** the processing further includes determining an adaptive window to be used for block matching to determine disparity information for the at least one image, Determining the adaptive window is at least partially based on comparing at least one of the grayscale values or color values of the pixels within the adaptive window with at least one of the grayscale values or color values of adjacent pixels, one or more non-transitory computer-readable media according to claim 10.

14. The processing includes performing recognition of one or more features within at least one of the disparity map or the disparity image to determine recognition information, and further includes transmitting at least one control signal based on the recognition information, the at least one control signal is an instruction for controlling the vehicle to accelerate or decelerate the vehicle, an instruction for controlling the vehicle to steer the wheels of the vehicle, or one or more non-transitory computer-readable media according to claim 10, including at least one of an instruction for presenting a warning.

15. Performing recognition of one or more features within at least one of the disparity map or the disparity image to determine recognition information, and further comprising transmitting at least one control signal based on the recognition information, the at least one control signal is an instruction for controlling the vehicle to accelerate or decelerate the vehicle, an instruction for controlling the vehicle to steer the wheels of the vehicle, or the method according to claim 6, including at least one of an instruction for presenting a warning.

Images