Using landmarks to enhance navigation instructions in difficult driving conditions
Through the onboard cameras, real-time identification and comparison of images, modification of navigation routes and providing enhanced instructions, the problem of drivers' difficulty in following navigation in severe weather and road conditions is solved, and navigation accuracy and safety is improved.
Patent Information
- Application Number
- CN201980026974.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-11-20
- Filing Date
- 2019-01-09
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2039-01-09
AI Technical Summary
In severe weather or road conditions, it is difficult for drivers to follow the navigation direction, resulting in difficulty in driving, and the existing technology is difficult to effectively adapt to real-time environmental changes.
Get real-time images with onboard cameras, compare with previously captured images, identify landmark visibility, modify navigation routes, and provide enhanced visual, audible or tactile instructions to ensure the driver can reach his destination correctly.
Improve the driver's navigation accuracy and safety under harsh conditions, and reduce driving difficulties caused by environmental factors by adjusting navigation routes and prompts in real time.
Smart Images

Figure CN112055806B_ABST
Abstract
Description
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS
[0002] This application claims the benefit of the filing date of U.S. Provisional Application No. 62 / 769,646, filed on November 20, 2018, entitled Enhanced Navigation Instructions with Landmarks Under Difficult Driving Conditions, the disclosure of which is incorporated herein by reference. Background Art
[0003] Map navigation systems are used in many different situations, including when providing driving or walking directions to a user. In the case of driving, the navigation system may adapt to current traffic conditions to change routes based on knowledge of traffic jams or accidents. This information may be received from third-party reports, e.g., crowdsourced from other drivers, local transportation authorities, etc. However, there may be real-time environmental and other factors that may also affect the journey. For example, this may include, for example, weather, road construction / lane work, low light conditions due to time of day, and other factors. Conditions related to these factors may limit the ability of a user, such as a driver, to properly follow the set of driving directions. Summary of the invention
[0004] Various aspects of the technology relate to factors that trigger enhanced driving directions, such as adverse weather conditions. After determining one or more relevant factors, the system modifies or otherwise enhances the directions in real time, enabling the driver to quickly and easily identify different landmarks and navigate to the desired destination using the enhanced information. For example, as discussed in detail below, the onboard camera is configured to detect objects and conditions in the environment adjacent to the vehicle or otherwise in the vicinity of the vehicle, such as within the line of sight in front of the vehicle. This real-time environmental information is analyzed, and when it is determined that the environmental information may adversely affect the driver's ability to follow the existing directions, enhanced directions are provided to more easily enable the driver to correctly follow a particular route.
[0005] According to one aspect, a method includes: obtaining initial directions for a driving route for a vehicle driver, the directions being used to plan a route for the vehicle from a starting point to a destination; obtaining previously captured images of one or more locations along the route; identifying a first landmark located in the previously captured images; receiving, by one or more processors of a computer device, real-time images obtained by a camera located in the vehicle; comparing, by one or more processors, at least one image from the real-time images with at least one image from the previously captured images to determine whether the first landmark is observable by the vehicle driver; and upon determining that the vehicle driver will not observe the first landmark, modifying, by the one or more processors, the initial directions for the driving route to indicate at least one other landmark different from the first landmark.
[0006] In one example, the comparison to determine whether the first landmark is observable by the driver of the vehicle includes determining whether the first landmark is partially or completely obscured from view by the driver of the vehicle. Here, determining whether the first landmark is partially or completely obscured may include determining that the visibility of the first landmark is adversely affected by weather conditions, low light conditions, another vehicle, a building, or vegetation. The camera may obtain real-time imagery during driving along the route. When at least one other landmark is visible to the driver, the modified direction may instruct the driver to take a specific driving action. Here, the specific driving action may be selected from the group consisting of slowing down, turning, stopping, or parking.
[0007] Modifying the initial directions may include at least one of updating a graphical display within the vehicle or playing audible instructions within the vehicle. A camera located in the vehicle has a field of view, and comparing at least one image from the real-time imagery with at least one image from previously captured images may include evaluating the field of view of the camera against an expected field of view of the driver. And modifying the initial directions may include augmenting or replacing the initial directions with at least one visual cue based on one or more objects along the driving route visible to the camera.
[0008] According to another aspect, a method includes: receiving, by one or more processors of a computer device, real-time imagery obtained by a camera located in a vehicle while a driver is driving the vehicle; analyzing, by the one or more processors, at least one image from the real-time imagery to determine current conditions in the external environment surrounding the vehicle; based on the analysis, determining, by the one or more processors, that the current conditions are expected to obscure selected landmarks in the external environment from being viewed by the driver; upon determining that the current conditions are expected to obscure selected landmarks in the external environment from being viewed by the driver, identifying, by the one or more processors, different landmarks observable by the camera located in the vehicle; and providing visual, audible, or tactile indications of the different landmarks to the driver.
[0009] The method may include, prior to providing, modifying an initial set of directions for driving to a destination using indications of different landmarks. The modified initial directions may instruct the driver to take a specific driving action when the different landmarks are visible to the driver. The specific driving action may be selected from the group consisting of slowing down, turning, stopping, or parking.
[0010] Determining that current conditions are expected to obscure a selected landmark in the external environment from view by the driver may include determining that visibility of the selected landmark is adversely affected by weather conditions, low light conditions, another vehicle, construction, or vegetation. And analyzing at least one image from the real-time imagery to determine current conditions in the external environment surrounding the vehicle may include applying a machine learning model to the real-time imagery to detect obstructions or other visibility deficiencies in the external environment.
[0011] According to another aspect, a system includes a memory configured to store at least one of instructions or data; and one or more processors operatively coupled to the memory. The one or more processors are configured to obtain an initial direction of a driving route for a vehicle driver, the direction used to plan a route for the vehicle from a starting point to a destination; obtain previously captured images of one or more locations along the route; identify a first landmark located in the previously captured image; receive real-time images obtained by a camera located in the vehicle; compare at least one image from the real-time image with at least one image from the previously captured image to determine whether the first landmark is observable by the vehicle driver; and when it is determined that the first landmark is not observable by the vehicle driver, modify the initial direction of the driving route to indicate at least one other landmark different from the first landmark.
[0012] In one example, the system further includes a camera. The camera may be part of a personal communication device.
[0013] According to another aspect, a system includes a memory configured to store at least one of instructions or data and one or more processors operatively coupled to the memory. The one or more processors are configured to receive real-time imagery obtained by a camera located in a vehicle while a driver is driving the vehicle; analyze at least one image from the real-time imagery to determine a current condition of an external environment surrounding the vehicle; based on the analysis, determine that the current condition is expected to obscure a selected landmark in the external environment from being viewed by the driver; when determining that the current condition is expected to obscure a selected landmark in the external environment from being viewed by the driver, identify different landmarks observable by the camera located in the vehicle; and provide visual, audible, or tactile indications of the different landmarks to the driver.
[0014] Analysis of at least one image from the real-time imaging to determine current conditions in the external environment surrounding the vehicle may be performed by one or more processors according to a machine learning process.
[0015] According to yet another aspect, a computing system includes one or more processors and a non-transitory computer-readable memory. The memory includes instructions that, when executed, cause at least one processor to perform the method described above.
[0016] According to another aspect, a non-transitory computer-readable storage medium stores instructions that, when executed, cause one or more processors to perform the method as described above. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1A An example passenger vehicle for use with the technical aspects is illustrated from an exterior perspective.
[0018] Figure 1B Diagram from the inside Figure 1A Example of a passenger vehicle.
[0019] Figure 2A is a functional diagram of an example passenger vehicle in accordance with aspects of the present disclosure.
[0020] Figure 2B is an example functional diagram of an imaging device according to aspects of the present disclosure.
[0021] Figure 3A -B illustrates an example device network according to aspects of the present disclosure.
[0022] Figure 4A -B illustrates various aspects of driving according to technology.
[0023] FIG. 5A to FIG. 5C Another driving scenario according to aspects of the present disclosure is illustrated.
[0024] Figure 6 Example methods according to aspects of the present disclosure are illustrated.
[0025] Figure 7 Another example method according to aspects of the present disclosure is illustrated. DETAILED DESCRIPTION
[0026] Environmental factors near the vehicle may affect the ease with which a driver is able to follow displayed or audible driving instructions. Accordingly, aspects of the technology relate to enhancing such instructions with different or additional (e.g., enhanced) visual or audible cues. This may be particularly important in adverse driving conditions such as where overall visibility is poor, or where obstructions may prevent the driver from seeing road signs, commercial signs, or other landmarks. The system uses an onboard camera or other imaging device to obtain imagery of the external environment and employs this information in evaluating whether and how to present enhanced visual and / or audible cues or directions to the driver. By improving the guidance provided to the user, the method assists the user in performing the task of controlling the vehicle. Concomitant benefits may include more efficient use of vehicle resources and improved road safety.
[0027] Example Vehicle Systems
[0028] Figure 1A The example passenger vehicle 100 is illustrated from an external perspective, and Figure 1B The vehicle 100 is illustrated from an interior perspective. As shown in both views, the imaging device 200 is positioned so that one or more image sensors face the environment outside the vehicle. For example, the imaging device 200 can be arranged inside the vehicle to have the same or similar vantage point as the driver without blocking the driver's line of sight. For example, the imaging device 200 can be placed along the inside of the dashboard or windshield on the driver's side of the vehicle. The vehicle's windshield wipers and / or defrosters are designed to keep the windshield clear. Therefore, positioning the imaging device 200 in this manner enables still images or video images to be captured through the glass.
[0029] Although certain aspects of the present disclosure may be particularly useful in conjunction with certain types of vehicles, the vehicle may be any type of vehicle including, but not limited to, automobiles, trucks, motorcycles, buses, recreational vehicles, and the like.
[0030] Figure 2A Illustrations such as Figure 1A -B is a block diagram of various systems of an example passenger vehicle 100. Figure 2AAs shown in the block diagram of , the vehicle includes a control system of one or more computing devices, such as a computing device 102 that includes one or more processors 104, a memory 106, and other components typically found in a general-purpose computing device. The memory 106 stores information accessible to the one or more processors 104, including instructions 108 and data 110 that can be executed or otherwise used by the processor 104. The memory 106 can be of any type capable of storing information accessible to the processor, including computing device readable media. The memory is a non-transitory medium, such as a hard drive, a memory card, an optical disk, a solid-state memory, a tape storage, etc. The system may include different combinations of the foregoing, whereby different portions of the instructions and data are stored on different types of media.
[0031] Instructions 108 may be any set of instructions (such as machine code) to be executed directly by a processor or any set of instructions (such as a script) to be executed indirectly. For example, instructions may be stored as computing device code on a computing device readable medium. In this regard, the terms "instructions" and "programs" may be used interchangeably herein. Instructions may be stored in an object code format for direct processing by a processor, or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or pre-compiled. Data 110 may be retrieved, stored, or modified by one or more processors 104 according to instructions 108. In one example, some or all of memory 106 may be an event data recorder or other secure data storage system configured to store vehicle diagnostics and / or detected sensor data, which may be onboard or remote, depending on the implementation.
[0032] The one or more processors 104 may be any conventional processor, such as a commercially available CPU. Alternatively, the one or more processors may be a dedicated device, such as an ASIC or other hardware-based processor. Figure 2A The processor, memory, and other elements of computing device 102 are illustrated as being functionally within the same block, but such a device may actually include multiple processors, computing devices, or memories that may or may not be stored within the same physical housing. Similarly, memory 106 may be a hard drive or other storage medium located in a different housing than processor 104. Thus, references to a processor or computing device will be understood to include references to a collection of processors or computers that may or may not operate in parallel.
[0033] In one example, computing device 202 can form a driving computing system incorporated into vehicle 100, for example, so that the vehicle can operate in a partially or fully autonomous driving mode. For example, a partially autonomous driving mode can be implemented as part of an advanced driver assistance system (ADAS) to provide adaptive cruise control or emergency braking. The driving computing system can be able to communicate with various components of the vehicle. For example, computing device 102 can communicate with various systems of the vehicle, including a drive system, which includes a deceleration system 112 (for controlling the braking of the vehicle), an acceleration system 114 (for controlling the acceleration of the vehicle), a steering system 116 (for controlling the orientation of the wheels and the direction of the vehicle), a signaling system 118 (for controlling the turn signal), a navigation system 120 (for navigating the vehicle to a location or around an object and / or providing navigation directions to the driver when operating in a manual or partially autonomous driving mode) and a positioning system 122 (for determining the positioning of the vehicle).
[0034] Computing device 102 is also operatively coupled to perception system 124 (for detecting objects in the vehicle environment), power system 126 (e.g., a battery and / or a gasoline or diesel powered engine), and drive system 128 to facilitate controlling the movement, speed, etc. of the vehicle in accordance with instructions 108 of memory 106 in driving modes that do not require or need continuous or periodic input from a vehicle occupant. Some or all of wheels / tires 130 are coupled to drive system 128, and computing device 102 may be able to receive information about tire pressure, balance, and other factors that may affect driving in any driving mode.
[0035] In certain driving modes, such as adaptive cruise control or emergency braking, the computing device 102 can control the direction and / or speed of the vehicle by controlling various components. The computing device 102 can use data from the map information and navigation system 120 to navigate the vehicle or assist the driver in navigating the vehicle to a destination location. The computing device 102 can use the positioning system 122 to determine the location of the vehicle and the perception system 124 to detect and respond to objects when it is necessary to safely reach the location. To do so, the computing device 102 can accelerate the vehicle (e.g., by increasing the fuel or other energy provided to the engine by the acceleration system 114), decelerate (e.g., by reducing the fuel supplied to the engine, changing gears, and / or applying brakes through the deceleration system 112), change direction (e.g., by turning the front wheels or other wheels of the vehicle 100 through the steering system 116) and signal such changes (e.g., by illuminating the turn signal of the signaling system 118). Therefore, the acceleration system 114 and the deceleration system 112 can be part of a powertrain or other transmission system 128, which includes various components between the engine of the vehicle and the wheels of the vehicle. Likewise, by controlling these systems, the computing device 102 may also control the vehicle's drivetrain 128 in order to maneuver the vehicle in a partially (or even fully) autonomous mode.
[0036] As an example, computing device 102 may interact with deceleration system 112 and acceleration system 114 to control the speed of the vehicle. Similarly, computing device 102 may use steering system 116 to control the direction of the vehicle. Signaling system 118 may be used by computing device 102 or the driver to signal the vehicle's intentions to other drivers or vehicles, such as by illuminating turn signals or brake lights when necessary.
[0037] The navigation system 120 may be used by the computing device 102 to determine and follow a route to a location. In this regard, the navigation system 120 and / or the memory 106 may store map information, for example, highly detailed maps that the computing device 102 may use to navigate or control a vehicle. As an example, these maps may identify the shape and elevation of roads, lane markings, intersections, crosswalks, speed limits, traffic lights, buildings, signs, real-time traffic information, vegetation, or other such objects and information. Lane markings may include features such as solid or dashed double or single lane lines, solid or dashed lane lines, reflectors, and the like. A given lane may be associated with left and right lane lines or other lane markings that define lane boundaries. Thus, most lanes may be bounded by the left edge of one lane line and the right edge of another lane line. Map information may be received from a remote computing system such as a map database, from other vehicles, and / or from previous trips made by the vehicle 100.
[0038] The perception system 124 also includes sensors for detecting objects outside the vehicle. The detected objects may be other vehicles, obstacles on the road, traffic signals, signs, trees, etc. For example, the perception system 124 may include one or more light detection and ranging (lidar) sensors, acoustic (sonar, internal and / or external microphones) devices, radar units, imaging devices such as cameras (e.g., optical and / or infrared), inertial sensors (e.g., gyroscopes or accelerometers), and / or any other detection devices that record data, which may be processed by the computing device 102. The sensors of the perception system 124 may detect objects and their characteristics, such as location, orientation, size, shape, type (e.g., vehicle, pedestrian, cyclist, etc.), heading, and speed of movement, etc. The raw data from the sensors and / or the aforementioned characteristics may be periodically and continuously sent to the computing device 102 for further processing when generated by the perception system 124. The computing device 102 may use the positioning system 122 to determine the location of the vehicle, and the perception system 124 may detect and respond to objects when needed to safely reach that location. Additionally, computing device 102 may perform calibration for a single sensor, all sensors in a particular sensor assembly, or between sensors in different sensor assemblies.
[0039] exist Figure 2A Also shown is a communication system 132. The communication system 132 may include one or more wireless network connections to facilitate communication with personal computing devices such as the driver and / or passengers within the vehicle and computing devices outside the vehicle such as in other nearby vehicles on the road or on a remote network. The network connections may include short-range communication protocols such as Bluetooth. TM ,Bluetooth TM Low Energy (LE), cellular connections, and various configurations and protocols including the Internet, the World Wide Web, Ethernet, VPNs, WANs, local networks, private networks using one or more company-proprietary communication protocols, Ethernet, WiFi and HTTP, and various combinations of the foregoing.
[0040] Figure 2A Also illustrated is a user interface subsystem 134. The user interface subsystem 134 may include one or more user inputs 136 (e.g., a mouse, keyboard, touch screen or other tactile input, and / or one or more microphones) and one or more display devices of a display subsystem 138 (e.g., a monitor having a screen or any other device operable to display information). In this regard, an interior electronic display 140 may be located within the cabin of the passenger vehicle (see Figure 1B), and can be used by computing device 102 to provide information to a driver or passenger within the vehicle. Output devices such as speakers 142 may also be located within the passenger vehicle. As discussed in more detail below, display 140 and / or speaker 142 are configured to provide enhanced directions or driving tips to the driver when the vehicle is operating in a manual or partially autonomous driving mode.
[0041] Figure 2B An example of an imaging device 200 is shown. The imaging device 200 may be part of the vehicle's onboard perception system 124. Or, alternatively, the imaging device 200 may be a self-contained or otherwise separate sensor unit. For example, the imaging device 200 may be a driver's (or passenger's) mobile phone, or may be a wearable device (e.g., a head mounted display system, a smart watch, etc.) or other device (e.g., a PDA, a tablet computer, etc.) that may be mounted within the interior cabin of the vehicle.
[0042] As shown, imaging device 200 includes one or more computing devices 202, such as a computing device that includes one or more processors 204, memory 206, and other components typically found in general-purpose computing devices. Memory 206 stores information accessible to one or more processors 204, including instructions 208 and data 210 that may be executed or otherwise used by processor 304. Figure 2A Description of processors, memory, instructions and data applies to Figure 2B of these elements.
[0043] In this example, the imaging device 200 also includes a communication system 212, which may have one or both of a wireless connection 214 and a wired connection 216. In this manner, the individual sensor units are configured to communicate directly or indirectly with the computing device 102 or other systems of the vehicle and / or with a remote system (see Figure 3A -B).
[0044] The imaging device 200 also includes an imaging system 218 having one or more image sensors 220. By way of example, the image sensor 220 may include one or more sensor devices or arrays of sensor devices configured for optical and / or infrared image capture. Image capture may be accomplished on a frame-by-frame basis as a still image and / or a series of video images. The one or more image sensors 220 are capable of capturing images of the environment outside the vehicle, such as a landscape image or a panoramic image.
[0045] The imaging device 200 may also include a user interface subsystem 222 having a user input 224. The user input 224 may include one or more mechanical actuators 226 (e.g., switches, buttons, and / or dials) and / or one or more soft actuators 228 (e.g., a capacitive or inductive touch screen). The user interface subsystem 222 may also include a graphical display 230, such as a preview screen for capturing images, a display for viewing captured images, or both.
[0046] In addition, the imaging device 200 may include other components, such as a position determination subsystem 236. The subsystem 236 may include sensors such as an accelerometer 238, a gyroscope 240, and / or other inertial sensors to help determine the pose or orientation of the imaging device 200. The position determination subsystem 236 may also include a GPS unit 242 or other positioning element to determine the latitude / longitude (and altitude) of the imaging device 200. The imaging device 200 may also include one or more internal clocks 244, such as to obtain timing information for the processor 204, the GPS unit 242, or to generate time stamps for images acquired by the imaging system 218. And a power source (or source) 246, such as one or more batteries internal to the imaging device 200 or a connection to an external source, may also be provided.
[0047] Example Collaboration System
[0048] As described above, the in-vehicle system can communicate with remote systems (e.g., remote assistance or fleet management), with other vehicles, and / or from various imaging devices within the vehicle. An example of this is Figure 3A and 3B In particular, Figure 3A and 3B 300, an imaging device 310, and a storage system 312 connected via a network 314. For example, computing device 304 may be a tablet, laptop, or netbook computer, and computing device 306 may be a PDA or mobile phone. Computing device 308 may be a wearable device, such as a smart watch or a head mounted display (HMD) device. Any of these may include a camera or other imager and may be used as a wearable device such as a smartwatch or head mounted display (HMD) device. Figure 2B The illustrated imaging device 200 operates.
[0049] System 300 is also illustrated by vehicles 316 and 318, which may be configured to communicate with Figure 1A and Figure 1B100. For example, vehicle 316 may be a passenger vehicle such as a sedan or minivan. Vehicle 318 may be a larger vehicle such as a truck, bus, etc. Vehicles 316 and / or 318 may be part of a fleet of vehicles. Although only a few vehicles, computing devices, and imaging devices are depicted for simplicity, such a system may include significantly more elements. Figure 3B As shown in FIG. 1 , each of computing devices 302, 304, 306, and 308 may include one or more processors, memories, data, and instructions. Such processors, memories, data, and instructions may be similar to those described above with respect to Figure 2A Those described are configured.
[0050] The various computing devices and vehicles may communicate via one or more networks, such as network 314. Network 314 and intermediate nodes may include various configurations and protocols, including short-range communication protocols such as Bluetooth. TM ,Bluetooth TM LE, the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using one or more company-proprietary communication protocols, Ethernet, WiFi, and HTTP, and various combinations of the foregoing. Such communications may be facilitated by any device such as a modem and a wireless interface that is capable of transmitting data to and from other computing devices.
[0051] In one example, computing device 302 may include one or more server computing devices with multiple computing devices, such as a load balancing server farm, which exchanges information with different nodes of the network for the purpose of receiving, processing and transmitting data to and from other computing devices. For example, computing device 302 may include one or more server computing devices capable of communicating with computing devices of vehicle 316 and / or 318 and computing devices 304, 306 and 308 via network 314. For example, vehicle 316 and / or 318 may be part of a fleet of vehicles that may be dispatched to various locations by a server computing device. In this regard, computing device 302 may be used as a dispatch server computing system that may be used to dispatch vehicles to different locations in order to pick up passengers or drop off passengers or pick up and deliver goods. In addition, server computing device 302 may use network 314 to transmit information to and present information to a user (e.g., driver) or a vehicle passenger of one of the other computing devices. In this regard, computing devices 304, 306 and 308 may be considered client computing devices.
[0052] like Figure 3BAs shown in , each client computing device 304, 306, and 308 can be a personal computing device intended for use by a corresponding user and has all components typically used with a personal computer device, including one or more processors (e.g., a central processing unit (CPU)), memory (e.g., RAM and internal hard drive) to store data and instructions, a display (e.g., a monitor with a screen, a touch screen, a projector, a television, or other device such as a smart watch display operable to display information), and a user input device (e.g., a mouse, keyboard, touch screen, or microphone). The client computing device may also include a camera for recording imagery that can be used to enhance driving tips as discussed herein, speakers, a network interface device, and all components for connecting these elements to each other. See also above for other features and components that may be part of the computing device. Figure 2B Discussion.
[0053] Storage system 313 may be a database or any type of computerized storage capable of storing information accessible to server computing device 302, such as a hard drive, memory card, ROM, RAM, DVD, CD-ROM, flash drive, and / or tape drive. In addition, storage system 313 may include a distributed storage system in which data is stored on multiple different storage devices that may be physically located in the same or different geographic locations. Storage system 314 may be accessed through, for example, Figure 3A The network 314 described in -B is connected to the computing devices and / or can be directly connected to or incorporated into any computing device.
[0054] Storage system 312 can store various types of information. For example, storage system 312 can store maps, landmarks, and / or images in one or more databases. This information can be retrieved or otherwise accessed by a server computing device, such as one or more server computing devices 302, in order to perform some or all of the features described herein.
[0055] For example, the storage system 312 may store map data. The map data may include, for example, sensor data generated by a perception system such as the perception system of the vehicle 100. The map data may also be map information obtained from various images. The landmark database may store information about various landmarks, which may or may not be associated with locations in the map database. For example, the map database may include street information about New York City, while the landmark database may include information about various places of interest or projects (e.g., the Empire State Building). The image database may be part of or separate from these databases and may include previously obtained images of locations in the map and / or landmark databases. The image database may include street level or other types of images about various locations. One or more of the map, landmark, and / or image databases may include supplemental information, such as road signs, commercial signs, sculptures or other artwork, etc., which is visible to drivers along the route. The server system 302, computing devices 304, 306, 308, and / or imaging device 310 may access such information directly or indirectly.
[0056] Implementation Method
[0057] In view of the structures and configurations described above and illustrated in the accompanying drawings, various embodiments will now be described.
[0058] An in-vehicle (or otherwise onboard) camera, such as imaging device 200, is configured to detect objects and conditions in an environment adjacent to or otherwise in the vicinity of the vehicle, such as within the line of sight of the front of the vehicle. As described above, the camera may be positioned within the vehicle, such as along the dashboard or inside the windshield, such as adjacent to a rearview mirror or sun visor, for example. In one example, the camera is positioned to provide a field of view that is similar to or aligned with the field of view that the driver will observe in front of the vehicle while driving. In this manner, an image sensor may be arranged on a camera or positioned within the vehicle cabin to view the road in front of the vehicle and in the vicinity, such as being able to detect road signs, commercial signs, storefronts, landmarks, and other points of interest.
[0059] The field of view (FOV) of the camera may be constrained based on the type of imaging device used, the placement of the camera within the vehicle, and other factors. Still images or video images may be acquired. The camera may include multiple imaging devices, or multiple cameras may be employed, for example, to provide stereoscopic imaging. For example, while in Figure 1A -B shows a single imaging device 200, but two or more imaging devices 200 may be arranged in the vehicle cabin to provide different fields of view around the vehicle. These imaging devices 200 may be directly or indirectly (e.g., via Bluetooth TMconnected) to communicate with each other or with vehicle systems, to communicate with remote systems or services, or to communicate with any or all such devices.
[0060] Despite Figure 2B 2, but the imaging device 200 or other camera may include one or more microphones, or may be operatively coupled to one or more microphones positioned around the vehicle to capture sound. Such microphones may be used, for example, to detect whether the vehicle is driving in snow or rain based on the sound of the tires as the vehicle moves along a road.
[0061] Example scenario
[0062] For situations where the driver is driving the vehicle manually or in a partially autonomous mode, he or she may drive to a destination according to a set of directions along a pre-planned route. The directions may be generated by or downloaded to a personal device such as a mobile phone, PDA, tablet computer, etc., which may or may not include the imaging device 200. Alternatively, the directions may be generated or maintained by the navigation system 120 of the vehicle itself. The directions may instruct the driver to turn onto a particular street, to maneuver a certain number of feet or meters, to proceed toward (or away from) certain objects or points of interest, etc.
[0063] Unfortunately, real-world conditions and other factors may adversely affect the driver's ability to follow driving directions. For example, snow, fog, or high winds may cause swaying of vegetation to obscure road signs. Or trucks or construction equipment may prevent the driver from seeing landmarks. Further, the driver may encounter problems when he / she needs to turn in heavy snow onto a side street where almost no signage can be seen under the current conditions. Therefore, even if there may be a stop sign, stop light, or other signage nearby, the driver may not be able to see the stop sign, stop light, or other signage due to obstructions (other vehicles, buildings), bad weather (e.g., snow, fog, heavy rain), or otherwise poor driving conditions (e.g., at dusk or other low light conditions). The received or previously generated directions may not be sufficient to cope with such factors.
[0064] Using the image information received from the camera and the onboard location information, the system is able to determine in real time current weather conditions (e.g., rain, fog, snow, dusk or nighttime, etc.) and / or road conditions (e.g., changes in surface type, road damage such as potholes, wet or slippery roads, debris on the road, closures, etc.). Changes in signs or obstructions of signs can also be determined based on the imagery captured by the camera. For example, this can be done by comparing the captured imagery to previously acquired images of the current location. For example, the onboard system can obtain previously acquired images from a remote server, from other vehicles, and / or from previous trips made by the same vehicle or even the imaging device itself.
[0065] The imaging device 200 is configured to provide a "camera feedback loop" of real-time images to the onboard system. During the trip, the system can compare previously acquired images with the real-time images from the camera in the vehicle. Such image processing can be performed by the camera or other device that captures the real-time image (e.g., Figure 3A -B's 304, 306, 308, or 310), the vehicle's onboard computer system (e.g., Figure 2A 102) and / or by a remote server (e.g., Figure 3A -B server 302). Image comparison or difference determination can be shared among different processing devices of the system, for example, based on processing resource availability, communication bandwidth and other factors. Based on the analysis, information about the differences in the obtained and acquired images may be generated. Depending on what the difference is (e.g., a road sign is not visible or partially obscured), the system may identify other visual cues for the driver.
[0066] In another example, an onboard system, a remote system (e.g., server 302), or both can employ a machine learning model on real-time acquired imagery to detect the presence and amount of snow, rain, construction equipment, or other types of obstructions and visibility defects. In this way, current driving conditions and environment are determined in light of the real-time imagery.
[0067] As discussed below, this feedback loop allows the system to warn the driver based on the real-time image data acquired. For example, this can be done by using more detailed instructions to update or change the initial set of driving directions based on things that are easily observed compared to the conventional expected street-by-street navigation mode. The system uses more lengthy instructions to guide the driver to the destination. This can include using visual cues to augment the instructions, which are expected or known to provide good orientation on their route. In particular, the system can determine whether and how to enhance the current driving instructions (such as making the instructions more lengthy), and make changes in real time. That is, in turn, this can make it easier and more likely for the driver to correctly follow the planned route. In addition, the system can change the route to avoid certain suboptimal conditions.
[0068] For example, larger, more easily spotted visual guides, such as landmarks, buildings, or various other features of the urban landscape (parks, billboards, lamp posts, etc.), can be used to provide better orientation to the driver and tell him / her when to turn. The result is a more lengthy set of instructions, such as "Turn left on 15th St., after a large billboard on your right" or "Begin slowing down near the tall brown building on your right, and prepare to turn right before the next such tall building", assuming, of course, that in the latter example, these two buildings are more prominent than the rest. These instructions can be presented visually, such as on a display 140 of the vehicle, a graphical display 230 of an imaging device 200, etc. Here, visual cues, such as larger fonts, different colors, and / or highlighting, can be used.
[0069] The instructions may also be presented audibly, such as through the speaker 142 of the vehicle user interface 134 or the speaker 232 of the imaging device 200. Additionally or alternatively, tactile feedback (e.g., through a steering wheel) may also help provide enhanced updated guidance to the driver, such as by vibrating the steering wheel in a manner indicating a right or left turn.
[0070] Another aspect of the technology relates to the likelihood that the driver will correctly follow the provided navigation instructions in light of the current environmental conditions. This likelihood can be based on previous navigation situations involving the same driver, the same or similar location, and / or other drivers under the same or similar conditions, aggregated (and anonymous) statistics, the type of landmark information available at the current location, and other factors. Such information can be used to calibrate the system and / or select additional (lengthy) feature cues for the driver.
[0071] For example, the system can analyze individual user deviations (e.g., a driver's failure to correctly follow an initial set of instructions) and how these deviations occur in different situations, as well as the individual's driving habits. Any person-specific information can be stored on the user's device and used only when determining whether to obtain a regular or "enhanced" series of instructions.
[0072] As part of this approach, the system may analyze not only individual failures to follow the route, but also aggregated failures. Thus, if a driver is missing multiple turns while driving, or if the system is made aware that several drivers along the route are missing turns, more lengthy visual and / or audible (and tactile) prompts may be provided to the driver. In another example, if the system detects that the driver is slowing down in an area where it was not expected, e.g., in the middle of a straight segment of the route, this may also indicate that the driver is having difficulty following the original directions. For example, this may be because road signs are difficult to read at night or when it is snowing, or the roads are icy or snowy, etc.
[0073] Depending on the scenario, the approach may involve accumulating information over time. For example, an explicit condition-dependent visibility model may be developed based on evidence from cameras and / or other devices. Here, if the system detects that it has been snowing for a certain amount of time, e.g., 15 minutes, an hour, etc., the system may select different or additional prompts to assist the driver.
[0074] Furthermore, even if signs or other landmarks are not obscured due to weather or poor lighting, they may be difficult to see, such as due to bends in the road or temporary obstacles such as trucks and other large vehicles. In such cases, an audible prompt may be presented to assist the driver when the camera feedback loop analysis suggests that the driver may be having trouble with directions.
[0075] One way to use lengthy clues to assist drivers involves first understanding what types of landmarks are present in the environment. For example, this may involve analyzing publicly available street-level imagery (e.g., Google Street View panoramas and other publicly available imagery). Image content analysis systems can detect a wide variety of urban features and various attributes of these features (shape, color, size), as well as combinations of semantic features (e.g., "a mural on the north-facing wall of a tall tower").
[0076] For example, the system can obtain image information of specific locations along the planned route from a remote server or other vehicles. The amount of information obtained may depend on the robustness of the wireless communication link. Or, alternatively, the driver's (or passenger's) personal device can download the imagery before starting the journey. Content updates can be provided as needed or available during the journey. Other information such as weather, traffic, and emergency notifications can also be obtained from external sources, including from other vehicles.
[0077] All of this information may be analyzed and / or combined as factors for use by the system in deciding whether or how to provide lengthy, enhanced directional information.
[0078] Figure 4A-B illustrates a scenario 400 where typical directional information may be obscured or otherwise difficult to detect. Figure 4A , the driver may be driving the vehicle 100 along the road and following pre-generated directions. However, the Main Street sign 402 may be obscured by the building 404. The business sign "Joe's Barber Shop" 406 is difficult to see due to snow, rain or low light conditions. However, there is another landmark associated with the barber shop, or otherwise located nearby, which is a barber pole 408 on the building 404.
[0079] like Figure 4B As shown in the top view of the vehicle, the in-vehicle imaging device 200 can have a field of view 410. The field of view 410 can be similar to the driver's viewpoint of the road. Here, through image comparison, machine learning, and / or any other method discussed herein, the system can determine from the real-time image acquired by the imaging device that road signs are obscured and conditions may be too deteriorated for the driver to see commercial signs. However, the image captured by the imaging device 200 can easily identify a barber pole 408, which can be on or off. This additional landmark can be determined by the system to provide a more visible point of interest. As a result, driving directions can be modified, or additional prompts can be presented to the driver visually or audibly, such as, parking near a barber pole (see Figure 1B ). Such lengthy reinforcement may be performed by the imaging device 200, the vehicle's computing device 102 and / or navigation system 120, or by the remote server 302 (or a combination thereof).
[0080] Figure 5A -C illustrates another scenario where the driver's field of view may be restricted or obscured. For example, Figure 5A The field of view range 500 is illustrated. This figure shows that under good conditions, the driver may have an unobstructed field of view 502 of 200 feet or meters or more toward the front of the vehicle. Figure 5B As shown in driving scene 510 of FIG. 5 , while driving along a road, there may be objects such as a stop sign 512 located adjacent to an intersection. There may also be puddles or ice 514 along a portion of the road ahead of the vehicle. However, there may also be a large truck 516 in the next lane. And as shown by Figure 5CAs shown in the overhead view 520 of the vehicle, at least a portion of the sign and the puddle or ice may be obscured from the driver's field of view by the truck. The puddle or ice may also be obscured by heavy rain, snow, etc. Nevertheless, using the techniques discussed herein, images obtained from an onboard imaging device can still be used to identify whether visibility has been reduced or whether an expected object (e.g., a stop sign) is not within the driver's field of view. Based on this, the system can provide enhanced or alternative directions or other instructions, such as performing a specific driving action. This may involve changing lanes or otherwise changing the route. Examples of specific driving actions include slowing down, turning, stopping, or parking. It may also involve visually, audibly, or tactilely providing information about other landmarks to help the driver get to the desired destination.
[0081] Example Method
[0082] Figure 6 An example 600 of a method for modifying driving directions is illustrated. For example, as shown in frame 602, an initial direction of a vehicle driver's driving route is obtained. These directions are used to plan a route from a starting point to a destination for the vehicle. At frame 604, previously captured images of one or more locations along the route are obtained. This can come from, for example, an image database of a remote system, another vehicle, or an image previously obtained by the current vehicle or by an imaging device used with the vehicle. At frame 606, the process identifies a first landmark located in the previously captured image. The processes of frames 602, 604, and 606 can be performed in different orders or in parallel. Any one or all of them can be performed before the driver drives the vehicle or while the vehicle is on the way to the destination.
[0083] At box 608, a real-time image obtained by a camera located in the vehicle is received. It can be received by one or more processors of a camera, a vehicle, and / or a remote system such as server system 302. At box 610, at least one image from the real-time image is compared with at least one image from a previously captured image to determine whether the first landmark will be observable by the vehicle driver. For example, the images can be compared to evaluate relative attributes associated with the observability of the first landmark, such as evaluating the relative brightness and / or other image characteristics of the image. The image characteristics may include semantic content, such as objects identified from the image or current conditions in the external environment. This is desirably done in real time while the vehicle is driving. Then, at box 612, when it is determined that the first landmark will not be observable by the driver of the vehicle, the initial direction of the driving route is modified to indicate at least one other landmark different from the first landmark. In this way, robust information can be provided to the driver to effectively assist the driver in maneuvering the vehicle to the destination.
[0084] Figure 7An example 700 of a method for providing updated or enhanced directional information to a driver of a vehicle is illustrated. In box 702, the process receives real-time imagery obtained by a camera located in a vehicle. The real-time imagery is obtained while the driver is driving the vehicle. At box 704, at least one image from the real-time imagery is analyzed to determine current conditions in the external environment surrounding the vehicle. This analysis can be done, for example, by one or more processors of a camera, a vehicle, and / or a remote server system such as server system 302. The current conditions may include, but are not limited to, weather conditions and / or time of day. At box 706, the system determines based on the analysis that the current conditions are expected to obscure selected landmarks in the external environment from being viewed by the driver. For example, each landmark can be associated with a condition under which the landmark is considered to be obscured (e.g., an unlit road sign can be considered to be obscured at night). To avoid this problematic situation, at box 708, it is explained that once it is determined that the current conditions are expected to obscure selected landmarks in the external environment from being viewed by the driver, the system identifies different landmarks observable by the camera located in the vehicle. The different landmarks may be currently observed in real time, or may be observed in the near future, e.g., within the next 5-10 seconds or no more than a minute, within 1-3 blocks, etc. Based on this information, at box 710, visual, audible, and / or tactile indications of the different landmarks are provided to the driver.
[0085] Unless otherwise stated, the aforementioned alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the aforementioned description of the embodiments should be made in an illustrative manner rather than in a manner that limits the subject matter defined by the claims. In addition, the provision of the examples described herein and the phrases expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; on the contrary, these examples are intended to illustrate only one of many possible embodiments. In addition, the same reference numerals in different figures can identify the same or similar elements. Unless otherwise expressly stated herein, these processes or other operations may be performed in different orders or simultaneously.
[0086] As mentioned above, one area where this technology can be useful is in inclement weather. For example, the system can provide target lane guidance, such as when snow covers or closes a lane of traffic. Here, directions can be changed to identify different exits or turning points. The system also works well in other non-optimal conditions. For example, a particular road sign may not be visible at night on an unlit residential street.
[0087] Other benefits may relate to the ability to provide lengthy modified instructions for factors other than those related to external environmental conditions. For example, as described above, the system can evaluate how closely the driver can follow the driving instructions. In view of this, alternative factors may include the complexity of the directions, many turns close together in time or distance, etc. Any or all of this information can be used to provide updated or enhanced instructions or prompts to the driver in real time, so as to improve the possibility that he or she will be able to effectively follow the driving directions to the desired destination.
[0088] The technology is also vehicle form factor agnostic. A similar approach could be applied to passenger vehicles, buses, motorcycles, light and heavy trucks, construction equipment, and more.
Claims
1. A method, include: obtaining initial directions for a driving route for a driver of a vehicle, the initial directions used to plan a route for the vehicle from a starting point to a destination, the initial directions including an indication of a first landmark; obtaining previously captured imagery of one or more locations along the driving route; identifying the first landmark located in the previously captured image; Receiving, by one or more processors of a computer device, real-time images acquired by a camera located in the vehicle; receiving, by the one or more processors, real-time tire sounds obtained by a microphone located in the vehicle; comparing, by the one or more processors, in real time upon receiving the real-time imagery, at least one image from the real-time imagery with at least one image from the previously captured imagery to determine whether the first landmark will be observable by a driver of the vehicle; determining, by the one or more processors in real time based on the comparing and the received real-time tire sounds, that the first landmark will be obscured from a viewpoint of the driver of the vehicle; identifying, by the one or more processors, in response to determining that the first landmark will be obscured, at least one second landmark from the real-time imagery, the at least one second landmark being different from the first landmark and observable from the viewpoint of the driver of the vehicle; as well as Upon determining that the first landmark will be obscured, the initial direction of the driving route is modified, by the one or more processors in real time, to indicate the at least one second landmark.
2. The method according to claim 1, in, Determining whether the first landmark is obscured includes determining that visibility of the first landmark is adversely affected by weather conditions, low light conditions, another vehicle, a building, or vegetation.
3. The method according to claim 1, in, The modified directions instruct the driver to take a specific driving action when the at least one second landmark is visible to the driver.
4. The method according to claim 3, in, The specific driving action is selected from the group consisting of slowing down, turning, stopping and parking.
5. The method according to claim 1, in, Modifying the initial directions includes at least one of updating a graphical display within the vehicle and playing audible instructions within the vehicle.
6. The method according to claim 1, in, The camera located in the vehicle has a field of view, and comparing the at least one image from the real-time imagery with the at least one image from the previously captured images includes evaluating the camera's field of view compared to the driver's expected field of view.
7. The method according to any one of claims 1 to 6, in, Modifying the initial directions further includes augmenting or replacing the initial directions with at least one visual cue based on one or more objects along the driving route visible to the camera.
8. A method, include: Receiving, by one or more processors of a computer device, real-time images acquired by a camera located in a vehicle while a driver is driving the vehicle; receiving, by the one or more processors, real-time tire sounds acquired by a microphone located in the vehicle during driving; analyzing, by the one or more processors, in real time at least one image from the real-time imagery and the real-time tire sounds to determine current conditions in an external environment surrounding the vehicle; determining, by the one or more processors, in real time after the analyzing and based on the real-time imagery and the real-time tire sounds, that the current condition is expected to at least partially obscure a selected landmark in the external environment from view by the driver; Upon determining that the current condition is expected to at least partially obscure the selected landmark in the external environment from view by the driver, identifying, by the one or more processors in real time based on the real-time image, different landmarks currently observable by the camera located in the vehicle, the identifying comprising selecting the different landmarks from the real-time image; as well as A visual, audible or tactile indication of the different landmarks is provided to the driver.
9. The method according to claim 8, further comprising: include: Prior to said providing, an initial set of directions for navigating to a destination is modified using said indications of said different landmarks.
10. The method according to claim 9, in, The modified set of directions instructs the driver to take a specific driving action when the different landmarks are visible to the driver.
11. The method according to claim 10, in, The specific driving action is selected from the group consisting of slowing down, turning, stopping and parking.
12. The method according to claim 8, in, Determining that the current conditions are expected to at least partially obscure the selected landmark in the external environment from view by the driver includes determining that visibility of the selected landmark is adversely affected by weather conditions, low light conditions, another vehicle, a building, or vegetation.
13. The method according to any one of claims 8 to 12, in, Analyzing the at least one image from the real-time imagery to determine the current conditions in the external environment surrounding the vehicle includes applying a machine learning model to the real-time imagery to detect obstructions or other visibility defects in the external environment.
14. A system, include: a memory configured to store at least one of instructions or data; as well as one or more processors, the one or more processors being operatively coupled to the memory, the one or more processors being configured to: obtaining initial directions for a driving route for a driver of a vehicle, the initial directions used to plan a route for the vehicle from a starting point to a destination, the initial directions including an indication of a first landmark; obtaining previously captured imagery of one or more locations along the driving route; identifying the first landmark located in the previously captured image; receiving a real-time image obtained by a camera located in the vehicle; receiving real-time tire sounds obtained by a microphone located in the vehicle; comparing, in real time as the real-time imagery is received, at least one image from the real-time imagery with at least one image from the previously captured imagery to determine whether the first landmark will be observable by a driver of the vehicle; determining in real time based on the comparing and the received real-time tire sounds that the first landmark will be obscured from a viewpoint of the driver of the vehicle; identifying at least one second landmark from the real-time imagery in response to determining that the first landmark will be obscured, the at least one second landmark being different from the first landmark and observable from the viewpoint of the driver of the vehicle; as well as Upon determining that the first landmark will be obscured, the initial direction of the driving route is modified in real time to indicate the at least one second landmark.
15. The system of claim 14, further comprising the camera.
16. A system according to claim 14 or claim 15, in, The camera is part of a personal communication device.
17. A system, include: a memory configured to store at least one of instructions or data; and one or more processors operatively coupled to the memory, the one or more processors configured to: receiving a real-time image obtained by a camera located in the vehicle while the driver is driving the vehicle; receiving real-time tire sounds obtained by a microphone located in the vehicle during driving; Analyzing in real time at least one image from the real-time image and the real-time tire sound to determine current conditions in the external environment around the vehicle; determining in real time, subsequent to the analyzing and based on the real-time imagery and the real-time tire sounds, that the current condition is expected to at least partially obscure a selected landmark in the external environment from view by the driver; Upon determining that the current condition is expected to at least partially obscure the selected landmark in the external environment from view by the driver, identifying in real time based on the real-time imagery different landmarks currently observable by the camera located in the vehicle, the identifying comprising selecting the different landmarks from the real-time imagery; as well as A visual, audible or tactile indication of the different landmarks is provided to the driver.
18. The system according to claim 17, in, Analyzing the at least one image from the real-time imagery to determine the current conditions in the external environment surrounding the vehicle is performed by the one or more processors according to a machine learning process.
19. A computing system, include: one or more processors; and A non-transitory computer readable memory comprising instructions which, when executed, cause the one or more processors to perform the method according to any one of claims 1 to 13.
20. A non-transitory computer-readable storage medium storing instructions which, when executed, cause one or more processors to perform the method according to any one of claims 1 to 13.
Citation Information
Patent Citations
Zone driving
CN103339010A
Vision augmented navigation
US20140372020A1
Systems and Methods for Using Real-Time Imagery in Navigation
US20170314954A1
Driving assistant system
US20180136001A1
System and method for determining pose data for a vehicle
US9915947B1
