Navigation system with traffic sign tracking and positioning mechanism based on monocular camera and operation method thereof

Abstract

A navigation system with a monocular camera-based traffic sign tracking and positioning mechanism and an operation method thereof are disclosed. A method of operating a navigation system includes: receiving a plurality of frames of images, including a first frame and a second frame; detecting a traffic sign from an image between the first frame and the second frame based on a sign recognition model; extracting a first image from the first frame and extracting a second image from the second frame; matching that the traffic signs in the first image and the second image are the same based on a similarity model; generating a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; and generating global coordinates for the sign position for display on a navigation map.

Description

Technical Field

[0001] Embodiments of the present invention relate generally to navigation systems, and more particularly to systems for mono-camera based traffic sign tracking and localization mechanisms. Background Art

[0002] As users become more and more able to obtain navigation devices and vehicle-based navigation services grow, new and old paradigms begin to take advantage of this new device space. Many technical solutions exist to take advantage of this new device capability to enhance or expand navigation and improve vehicle safety. However, users are not typically equipped with the ability to determine the location of traffic signs during navigation or route guidance. Research and development in the prior art can take countless different directions.

[0003] Therefore, there remains a need for a navigation system with a monocular camera-based traffic sign tracking and location mechanism for informing the operator when using the navigation system and route guidance. Finding answers to these questions is increasingly critical in light of increasing commercial competitive pressures, growing consumer expectations, and diminishing opportunities for meaningful product differentiation in the marketplace. Additionally, the need to reduce costs, improve efficiency and performance, and meet competitive pressures adds even greater urgency to the critical necessity of finding answers to these questions.

[0004] Solutions to these problems have been long sought but prior developments have not taught or suggested any solutions and, thus, solutions to these problems have long eluded those skilled in the art. Summary of the invention

[0005] An embodiment of the present invention provides an operating method of a navigation system, comprising: receiving multiple frames of images, including a first frame and a second frame; detecting traffic signs from images between the first frame and the second frame based on a sign recognition model; extracting a first image from the first frame, and extracting a second image of the second frame; matching the traffic signs in the first image and the second image to be the same based on a similarity model; generating a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; and generating global coordinates of the sign position for display on a navigation map.

[0006] An embodiment of the present invention provides a navigation system, comprising: a communication unit configured to receive multiple frames of images, including a first frame and a second frame; and a control unit coupled to the communication unit, the control unit being configured to: detect traffic signs from images between the first frame and the second frame based on a sign recognition model; extract a first image from the first frame, and extract a second image of the second frame; match the traffic signs in the first image and the second image to be the same based on a similarity model; generate a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; and generate global coordinates for the sign position for display on a navigation map.

[0007] An embodiment of the present invention provides a non-temporary computer-readable medium, comprising instructions that can be executed by a control circuit for a navigation system, the instructions including: receiving multiple frames of images, including a first frame and a second frame; detecting traffic signs from images between the first frame and the second frame based on a sign recognition model; extracting a first image from the first frame and extracting a second image of the second frame; matching the traffic signs in the first image and the second image to be the same based on a similarity model; generating a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; and generating global coordinates of the sign position for display on a navigation map.

[0008] Certain embodiments of the present invention have other steps or elements in addition to the above steps or elements, or have other steps or elements to replace the above steps or elements. These steps or elements will become clear to those skilled in the art by reading the following detailed description and referring to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 The invention relates to a navigation system having a traffic sign tracking and positioning mechanism based on a monocular camera in an embodiment of the present invention.

[0010] Figure 2 is an example of a top plan view illustration of a vehicle for use with a navigation system.

[0011] Figure 3 is an exemplary block diagram of a navigation system.

[0012] Figure 4 is an example of a display interface for a navigation system.

[0013] Figure 5 is an example of a first frame having a first image.

[0014] Figure 6 is an example of a second frame having a second image.

[0015] Figure 7 is an exemplary control flow of a navigation system.

[0016] Figure 8 is an exemplary flow chart of an operating method of a navigation system in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following embodiments are described in sufficient detail to enable those skilled in the art to make and use the invention. It is understood that other embodiments will be apparent based on the present disclosure, and that system, process or mechanical changes may be made without departing from the scope of the embodiments of the invention.

[0018] In the following description, many specific details are given to provide a thorough understanding of the present invention. However, it is apparent that the present invention can be implemented without these specific details. In order to avoid obscuring the embodiments of the present invention, some well-known circuits, system configurations and processing steps are not disclosed in detail.

[0019] The drawings showing embodiments of the system are semi-diagrammatic and not to scale, and in particular, some dimensions are for clarity of presentation and are shown exaggerated in the drawings. Similarly, although the views in the drawings generally show similar orientations for ease of description, such depictions in the drawings are arbitrary for the most part. In general, the present invention can be operated in any orientation.

[0020] In embodiments of the present invention according to the context in which the term "module" is used, the term "module" referred to herein may include software, hardware, or a combination thereof. For example, software may be machine code, firmware, embedded code, and application software. For another example, hardware may be a circuit, a processor, a computer, an integrated circuit core, a pressure sensor, an inertial sensor, a micro-electromechanical system (MEMS), a passive device, or a combination thereof. In addition, if modules are written in the device claim section below, these modules are considered to include hardware circuits for the purpose and scope of the device claim.

[0021] The modules in the following description of the embodiments may be coupled to each other as described or shown. The coupling may be direct or indirect, with no or intermediate items between the coupled items, respectively. The coupling may be physical contact or through communication between items.

[0022] Reference now Figure 1 , which shows a navigation system 100 with a monocular camera-based traffic sign tracking and positioning mechanism in an embodiment of the present invention. The navigation system 100 includes a first device 102, such as a client or a server, which is connected to a second device 106, such as a client or a server. The first device 102 can communicate with the second device 106 using a communication path 104 (such as a wireless or wired network).

[0023] For example, the first device 102 can be any of a variety of computing devices, such as a cellular phone, a tablet computer, a smart phone, a notebook computer, a vehicle embedded navigation system, or a computing device. The first device 102 can be directly or indirectly coupled to the communication path 104 to communicate with the second device 106, or can be an independent device.

[0024] The second device 106 may be any of various centralized or decentralized computing devices, sensor devices for measuring or recording environmental information, such as sensor instruments, sensor devices, or sensor arrays. For example, the second device 106 may be a multimedia computer, a laptop computer, a desktop computer, a grid computing resource, a virtualized computer resource, a cloud computing resource, a router, a switch, a peer distributed computing device, or a combination thereof.

[0025] The second device 106 can be mounted externally or internally on a vehicle, centralized in a single room or within a vehicle, distributed across different rooms, distributed across different geographic locations, embedded within a telecommunications network. The second device 106 can be coupled to the communication path 104 to communicate with the first device 102 .

[0026] For purposes of illustration, the navigation system 100 is described with the second device 106 as a computing device, but it is understood that the second device 106 can be a different type of device, such as an independent sensor or measurement device. Also for purposes of illustration, the navigation system 100 is shown with the second device 106 and the first device 102 as endpoints of the communication path 104, but it is understood that the navigation system 100 can have a different partitioning between the first device 102, the second device 106, and the communication path 104. For example, the first device 102, the second device 106, or a combination thereof can also function as part of the communication path 104.

[0027] The communication path 104 can span and represent various networks and network topologies. For example, the communication path 104 can include wireless communication, wired communication, optical, ultrasonic, or a combination thereof. Satellite communication, cellular communication, Bluetooth, infrared data association standard (IRDA), wireless fidelity (WiFi), and world microwave access interoperability (WiMAX) are examples of wireless communication that can be included in the communication path 104. Ethernet, digital subscriber line (DSL), fiber to the home (FTTH), and plain old telephone service (POTS) are examples of wired communication that can be included in the communication path 104. In addition, the communication path 104 can traverse several network topologies and distances. For example, the communication path 104 can include a direct connection, a personal area network (PAN), a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), or a combination thereof.

[0028] Reference now Figure 2 , which shows the Figure 1An example of a top plan view of a vehicle 202 of the navigation system 100. As an example, the navigation system 100 may include Figure 1 The first device 102 of the embodiment of the present invention serves as a vehicle 202, or interacts with it. The vehicle 202 may also include one or more environmental sensors 210. The vehicle 202 is an object or machine used to transport people or goods. The vehicle 202 may also provide assistance in manipulating or operating an object or machine.

[0029] Vehicle 202 may include or represent different types of vehicles. For example, vehicle 202 may be an electric vehicle, a fuel vehicle, or a hybrid vehicle. For another example, vehicle 202 may be an autonomous vehicle or a non-autonomous vehicle. As a specific example, vehicle 202 may include a car, a truck, a cart, or a combination thereof.

[0030] The vehicle 202 may include devices, circuits, one or more specific sensors, or a combination thereof, for providing assistance or additional information to control, manipulate, or operate the vehicle 202. The vehicle 202 may include vehicle communication circuitry 204, vehicle control circuitry 206, vehicle storage circuitry 208, other interfaces, or a combination thereof.

[0031] The vehicle storage circuit 208 may include a functional unit or circuit integrated into the vehicle 202 and configured to store and call up information. The vehicle storage circuit 208 may be a volatile memory, a non-volatile memory, an internal memory, an external memory, or a combination thereof. For example, the vehicle storage circuit 208 may be a non-volatile memory, such as a non-volatile random access memory (NVRAM), flash memory, disk storage, or a volatile memory, such as a static random access memory (SRAM). The vehicle storage circuit 208 may store vehicle software, other relevant data, such as input information, information from sensors, processing results, information predetermined or preloaded by the computer system 100 or the vehicle manufacturer, or a combination thereof.

[0032] The vehicle control circuit 206 may include a functional unit or circuit integrated into the vehicle 202 and configured to execute or implement instructions. The vehicle control circuit 206 may execute or implement vehicle software to provide intelligence for the vehicle 202, the navigation system 100, or a combination thereof. The request may come from other parts of the vehicle 202, the navigation system 100, or a combination thereof, or from outside the navigation system 100.

[0033] The vehicle control circuit 206 can be implemented in a variety of different ways. For example, the vehicle control circuit 206 can be a processor, an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. As a more specific example, the vehicle control circuit 206 can include an engine control unit, one or more central processing units, or a combination thereof.

[0034] The vehicle communication circuit 204 may include functional units or circuits integrated into the vehicle 202 and configured to enable external communications to and from the vehicle 202. For example, the vehicle communication circuit 204 may allow the vehicle 202 to communicate with Figure 1 The first device 102, the second device 106, Figure 1 The vehicle communication circuit 204 can be provided to other parts of the vehicle 202, the navigation system 100, or a combination thereof, or external to the navigation system 100.

[0035] The vehicle communication circuit 204 may also function as a communication hub that allows the vehicle 202 to function as part of the communication path 104 and is not limited to being an endpoint or terminal circuit of the communication path 104. The vehicle communication circuit 204 may include active and passive components, such as microelectronics or an antenna, for interacting with the communication path 104. For example, the vehicle communication circuit 204 may include a modem, a transmitter, a receiver, a port, a connector, or a combination thereof, for wired communication, wireless communication, or a combination thereof.

[0036] The vehicle communication circuit 204 can be coupled to the communication path 104 to directly send or receive information between the vehicle communication circuit 204 and the first device 102, the second device 106, or a combination thereof as endpoints of the communication, such as for direct line-of-sight communication or peer-to-peer communication. The vehicle communication circuit 204 can also be coupled to the communication path 104 to send or receive information through a server or another intermediate device between the communication endpoints.

[0037] The vehicle 202 may also include various interfaces. The vehicle 202 may include one or more interfaces for interaction or internal communication between functional units or circuits of the vehicle 202. For example, the vehicle 202 may include one or more interfaces for the vehicle storage circuit 208, the vehicle control circuit 206, or a combination thereof, such as a driver, firmware, wire connection or bus, protocol, or a combination thereof.

[0038] The vehicle 202 may also include one or more interfaces for interacting with an occupant, operator or driver, passenger, or combination thereof relative to the vehicle 202. For example, the vehicle 202 may include a user interface including input or output devices or circuits, such as a screen or touch screen, a speaker, a microphone, a keyboard or other input device, a dashboard, or a combination thereof.

[0039] The vehicle 202 may also include one or more interfaces and switches or actuators for physically controlling movable components of the vehicle 202. For example, the vehicle 202 may include one or more interfaces and control mechanisms to physically perform and control manipulation of the vehicle 202, such as for automated driving or maneuvering features.

[0040] Functional units or circuits in the vehicle 202 may operate individually and independently of other functional units or circuits. The vehicle 202 may operate individually and independently of the first device 102, the communication path 104, the second device 106, other devices or vehicles, or a combination thereof.

[0041] The above functional units or circuits may be implemented in hardware. For example, one or more of the functional units or circuits may be implemented using gates, circuits, processors, computers, integrated circuits, integrated circuit cores, pressure sensors, inertial sensors, micro-electromechanical systems (MEMS), passive devices, physical non-transitory storage media containing instructions for executing software functions, a portion thereof, or a combination thereof.

[0042] The environmental sensors 210 are each a device for detecting or identifying the environment of the vehicle 202. The environmental sensors 210 may detect, identify, determine the vehicle 202 itself to obtain the state or movement of the vehicle 202, or a combination thereof. The environmental sensors 210 may detect, identify, determine the environment inside the cabin of the vehicle 202, the environment outside and around the vehicle 202, or a combination thereof.

[0043] For example, the environmental sensor 210 may include a position-mobile sensor 212, a visual sensor 214, a radar sensor 216, an accessory sensor 218, a volume sensor 220, or a combination thereof. The position-mobile sensor 212 may identify or calculate the geographic location of the vehicle 202, determine the movement of the vehicle 202, or a combination thereof. Examples of the position-mobile sensor 212 may include an accelerometer, a speedometer, a GPS receiver or device, a gyroscope or compass, or a combination thereof. In addition to the position-mobile sensor 212, the vehicle 202 may include an environmental sensor 210, such as a thermal sensor. The thermal sensor may capture and provide a temperature reading of a portion of the vehicle 202. The thermal sensor may also capture and provide a temperature reading of the exterior of the vehicle 202.

[0044] The visual sensor 214 may include a sensor for detecting or determining visual information representing the environment outside and around the vehicle 202. The visual sensor 214 may include a camera attached to or integrated with the vehicle 202. For example, the visual sensor 214 may include a camera such as a forward camera, a rear view or reverse camera, a side view or blind spot camera, or a combination thereof. For another example, the visual sensor 214 may include an infrared sensor or a night vision sensor. As a specific example, the visual sensor 214 may include a monocular camera 215. The monocular camera 215 detects or determines visual information using a single visual system. The monocular camera 215 does not require a dual view or stereo vision system.

[0045] The visual sensor 214 may further include a camera on the first device 102 that is connected to and interacts with the vehicle 202. The visual sensor 214 may also include a cabin camera for detecting or determining visual information of the vehicle or the interior of the cabin of the vehicle.

[0046] Radar sensor 216 may include an object detection system, device, or circuit. Radar sensor 216 may determine or identify the presence of an object or target (e.g., an obstacle or another vehicle) outside of vehicle 202, the relative position or distance between the object or target and vehicle 202, or a combination thereof.

[0047] Radar sensor 216 may utilize radio waves to determine or identify the presence of an object or target, a relative position or distance from vehicle 202, or a combination thereof. For example, radar sensor 216 may include a proximity sensor or warning system, such as for an area geographically or physically relative to the front, rear, adjacent to, or to the side of vehicle 202, or a combination thereof.

[0048] The accessory sensors 218 may include sensors for determining or detecting the status of subsystems or features of the vehicle 202. The accessory sensors 218 may determine or detect the status or setting of windshield wipers, turn signals, gear settings, headlights, or a combination thereof.

[0049] Volume sensor 220 may include a sensor for detecting or determining sounds of vehicle 202. Volume sensor 220 may include a microphone for detecting or determining sounds within a cabin of vehicle 202. Volume sensor 220 may also include circuitry for detecting or determining a volume level or output level of a speaker within vehicle 202. Vehicle 202 may use one or more of environmental sensors 210 that describe or represent information about the environment within or around vehicle 202.

[0050] The vehicle 202 may also include on-board diagnostics 222 (OBD) that may be accessed by the vehicle control circuitry 206. As an example, the vehicle control circuitry 206 may access the on-board diagnostics 222 using the vehicle communication circuitry 204. The vehicle 202 may store and retrieve the on-board diagnostics 222 to and from the vehicle storage circuitry 208.

[0051] The on-board diagnostics 222 represent information about the vehicle 202. For example, the on-board diagnostics 222 can provide a status or condition of the vehicle 202 or a portion thereof. As a specific example, the on-board diagnostics 222 can represent the ignition status 224.

[0052] The ignition state 224 indicates the current state of the ignition. The ignition state 224 may indicate whether the engine of the vehicle 202 is on or off. The term “on” refers to a state when the engine of the vehicle 202 is running. The term “off” refers to a state when the engine of the vehicle 202 is not running. The on-board diagnostics 222 may be carried on the vehicle bus 226 to communicate the ignition state 224.

[0053] As an example, the vehicle bus 226 may be a controller area network (CAN) of the vehicle 202 that allows interaction between the vehicle communication circuitry 204 , the vehicle control circuitry 206 , the vehicle storage circuitry 208 , the on-board diagnostics 222 , other interfaces, or combinations thereof.

[0054] The vehicle 202 may further include Figure 1 . For example, the vehicle 202 may include the first device 102. As a more specific example, the vehicle communication circuit 204, the vehicle control circuit 206, the vehicle storage circuit 208, the environmental sensor 210, one or more interfaces or a combination thereof may be included in the first device 102 or constitute the first device, and the first device is included in the vehicle 202 or integrated with the vehicle. As another more specific example, the vehicle 202 may include or be integrated with the first device 102, and the first device includes an embedded computer system, an infotainment system, an intelligent driving or driver assistance system, a self-driving or manipulation system for a vehicle, or a combination thereof.

[0055] Reference now Figure 3 , which shows an example block diagram of a navigation system 100. The navigation system 100 may include a first device 102, a communication path 104, and a second device 106. The first device 102 may send information to the second device 106 via the communication path 104 in a first device transmission 308. The second device 106 may send information to the first device 102 via the communication path 104 in a second device transmission 310.

[0056] For illustrative purposes, the navigation system 100 is shown with the first device 102 as a client device, but it is understood that the navigation system 100 can enable the first device 102 to be a different type of device. For example, the first device 102 can be a server with a display interface.

[0057] Also for illustrative purposes, the navigation system 100 is shown with the second device 106 acting as a server, but it is understood that the navigation system 100 can act as a different type of device with the second device 106. For example, the second device 106 can be a client device.

[0058] For simplicity of description in this embodiment of the invention, the first device 102 will be described as a client device and the second device 106 will be described as a server device. Embodiments of the invention are not limited to this selection of device types. This selection is an example of an embodiment of the invention.

[0059] The first device 102 can include a first control unit 312, a first storage unit 314, a first communication unit 316, a first user interface 318, and a first location unit 320. The first control unit 312 can include a first control interface 322. The first control unit 312 can execute first software 326 to provide intelligence of the navigation system 100.

[0060] The first control unit 312 can be implemented in a variety of different ways. For example, the first control unit 312 can be a processor, an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. As a specific example, the first control unit 312 can represent Figure 2 Vehicle 202 Figure 2 The first control interface 322 can be used for communication between the first control unit 312 and other functional units in the first device 102. The first control interface 322 can also be used for communication outside the first device 102.

[0061] The first control interface 322 can receive information from the other functional units or from external sources, or can transmit information to the other functional units or to external destinations. The external sources and the external destinations refer to sources and destinations outside the first device 102.

[0062] The first control interface 322 can be implemented in different ways and can include different implementations, depending on which functional units or external units are being interfaced with the first control interface 322. For example, the first control interface 322 can be implemented with a pressure sensor, an inertial sensor, a micro-electromechanical system (MEMS), an optical circuit, a waveguide, a wireless circuit, a wired circuit, or a combination thereof.

[0063] The first location unit 320 can generate, for example, location information, current heading, and current speed of the first device 102. The first location unit 320 can be implemented in many ways. For example, the first location unit 320 can be used as at least a portion of a global positioning system (GPS) (e.g., a GPS receiver), an inertial navigation system, a cellular tower positioning system, a pressure positioning system, or any combination thereof.

[0064] The first location unit 320 may include a first location interface 332. The first location interface 332 may be used for communication between the first location unit 320 and other functional units in the first device 102. The first location interface 332 may also be used for communication outside the first device 102.

[0065] The first location interface 332 can receive information from other functional units or from external sources, or can transmit information to other functional units or external destinations. The external sources and the external destinations refer to sources and destinations that are physically separated from the first device 102.

[0066] The first location interface 332 can include different implementations, depending on which functional units or external units are interfaced with the first location unit 320. The first location interface 332 can be implemented with technologies and methods similar to the implementations of the first control interface 322.

[0067] The first storage unit 314 can store the first software 326. The first storage unit 314 can also store related information. For example, the first storage unit 314 can store information such as a navigation map.

[0068] The first storage unit 314 may be a volatile memory, a non-volatile memory, an internal memory, an external memory, or a combination thereof. For example, the first storage unit 314 may be a non-volatile memory, such as a non-volatile random access memory (NVRAM), a flash memory, a disk memory, or a volatile memory, such as a static random access memory (SRAM). As a specific example, the first storage unit 314 may represent a memory of the vehicle 202. Figure 2 The vehicle storage circuit 208.

[0069] The first storage unit 314 can include a first storage interface 324. The first storage interface 324 can be used for communication between other functional units in the first device 102. The first storage interface 324 can also be used for communication outside the first device 102.

[0070] The first storage interface 324 can receive information from the other functional units or from external sources, or can transmit information to the other functional units or to external destinations. The external sources and the external destinations refer to sources and destinations external to the first device 102.

[0071] The first storage interface 324 can include different implementations, depending on which functional units or external units are interfaced with the first storage unit 314. The first storage interface 324 can be implemented with technologies and techniques similar to the implementation of the first control interface 322.

[0072] The first communication unit 316 can enable external communications to and from the first device 102. For example, the first communication unit 316 can allow the first device 102 to communicate with the second device 106, accessories (eg, peripheral devices or a computer desktop), and the communication path 104.

[0073] The first communication unit 316 may also function as a communication hub, which allows the first device 102 to function as a part of the communication path 104 and is not limited to being an end point or terminal unit of the communication path 104. The first communication unit 316 may include active and passive components, such as microelectronics or antennas, to interact with the communication path 104. As a specific example, the first communication unit 316 may represent a Figure 2 The vehicle communication circuit 204.

[0074] The first communication unit 316 may include a first communication interface 328. The first communication interface 328 may be used for communication between the first communication unit 316 and other functional units in the first device 102. The first communication interface 328 may receive information from the other functional units or may send information to the other functional units.

[0075] The first communication interface 328 can include different implementations, depending on which functional units are interfaced with the first communication unit 316. The first communication interface 328 can be implemented with technologies and methods similar to the implementations of the first control interface 322.

[0076] The first user interface 318 allows a user (not shown) to dock and interact with the first device 102. The first user interface 318 may include an input device and an output device. Examples of the input device of the first user interface 318 may include a keypad, a touch pad, soft keys, a keyboard, a microphone, an infrared sensor for receiving a remote signal, or any combination thereof, to provide data and communication input. Examples of output devices may include a first display interface 330 and a first audio interface 352.

[0077] The first display interface 330 may include a display, a projector, a video screen, a speaker, or any combination thereof. The first audio interface 352 may include a sensor, a speaker, a microphone, a headset, a subwoofer, a surround sound component, a transducer, or any combination thereof. The first display interface 330 and the first audio interface 352 allow a user of the navigation system 100 to interact with the navigation system 100.

[0078] The first control unit 312 can operate the first user interface 318 to display information generated by the navigation system 100. The first control unit 312 can also execute the first software 326 for other functions of the navigation system 100. The first control unit 312 can further execute the first software 326 to interact with the communication path 104 via the first communication unit 316.

[0079] The second device 106 can be optimized to implement embodiments of the present invention in a multi-device embodiment with the first device 102. The second device 106 can provide additional or higher performance processing capabilities compared to the first device 102. The second device 106 can include a second control unit 334, a second communication unit 336, and a second user interface 338.

[0080] The second user interface 338 allows a user (not shown) to dock and interact with the second device 106. The second user interface 338 may include an input device and an output device. The example of the input device of the second user interface 338 may include a keypad, a touch pad, soft keys, a keyboard, a microphone or any combination thereof to provide data and communication input. The example of the output device of the second user interface 338 may include a second display interface 340 and a second audio interface 354.

[0081] The second display interface 340 may include a display, a projector, a video screen, a speaker, or any combination thereof. The second audio interface 354 may include a sensor, a speaker, a microphone, a headset, a subwoofer, a surround sound component, a transducer, or any combination thereof. The second display interface 340 and the second audio interface 354 allow a user of the navigation system 100 to interact with the navigation system 100. The second control unit 334 may execute the second software 342 to provide the intelligence of the second device 106 of the navigation system 100. The second software 342 may operate in conjunction with the first software 326. Compared with the first control unit 312, the second control unit 334 may provide additional performance.

[0082] The second control unit 334 can operate the second user interface 338 to display information. The second control unit 334 can also execute the second software 342 for other functions of the navigation system 100 , including operating the second communication unit 336 to communicate with the first device 102 via the communication path 104 .

[0083] The second control unit 334 can be implemented in many different ways. For example, the second control unit 334 can be a processor, an embedded processor, a microprocessor, a hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof.

[0084] The second control unit 334 may include a second controller interface 344. The second controller interface 344 may be used for communication between the second control unit 334 and other functional units in the second device 106. The second controller interface 344 may also be used for communication outside the second device 106.

[0085] The second controller interface 344 can receive information from the other functional units or from external sources, or can transmit information to the other functional units or to external destinations. The external sources and the external destinations refer to sources and destinations external to the second device 106 .

[0086] The second controller interface 344 can be implemented in different ways and can include different implementations, depending on which functional units or external units are interfaced with the second controller interface 344. For example, the second controller interface 344 can be implemented with a pressure sensor, an inertial sensor, a micro-electromechanical system (MEMS), an optical circuit, a waveguide, a wireless circuit, a wired circuit, or a combination thereof.

[0087] The second storage unit 346 can store the second software 342. The second storage unit 346 can also store information used to generate a map. The second storage unit 346 can be sized to provide additional storage capacity to supplement the first storage unit 314.

[0088] For purposes of illustration, the second storage unit 346 is shown as a single element, but it is understood that the second storage unit 346 can be a distribution of storage elements. Also for purposes of illustration, the navigation system 100 is shown such that the second storage unit 346 is a single hierarchical storage system, but it is understood that the navigation system 100 can have a differently configured second storage unit 346. For example, the second storage unit 346 can be formed with different storage technologies forming a memory hierarchy system including different levels of cache, main memory, rotating media, or offline storage.

[0089] The second storage unit 346 may be a volatile memory, a non-volatile memory, an internal memory, an external memory, or a combination thereof. For example, the second storage unit 346 may be a non-volatile memory such as a non-volatile random access memory (NVRAM), a flash memory, a disk memory, or a volatile memory such as a static random access memory (SRAM).

[0090] The second storage unit 346 can include a second storage interface 348. The second storage interface 348 can be used for communication between other functional units in the second device 106. The second storage interface 348 can also be used for communication outside the second device 106.

[0091] The second storage interface 348 can receive information from the other functional units or from external sources, or can send information to the other functional units or to external destinations. The external sources and the external destinations refer to sources and destinations external to the second device 106.

[0092] The second storage interface 348 can include different implementations depending on which functional units or external units are interfaced with the second storage unit 346. The second storage interface 348 can be implemented with technologies and methods similar to the implementations of the second controller interface 344.

[0093] The second communication unit 336 can enable external communications to and from the second device 106. For example, the second communication unit 336 can allow the second device 106 to communicate with the first device 102 via the communication path 104.

[0094] The second communication unit 336 may also function as a communication hub, which allows the second device 106 to function as part of the communication path 104 and is not limited to being an endpoint or terminal unit of the communication path 104. The second communication unit 336 may include active and passive components, such as microelectronics or an antenna, to interact with the communication path 104.

[0095] The second communication unit 336 may include a second communication interface 350. The second communication interface 350 may be used for communication between the second communication unit 336 and other functional units in the second device 106. The second communication interface 350 may receive information from other functional units or may send information to other functional units.

[0096] The second communication interface 350 can include different implementations, depending on which functional units are interfaced with the second communication unit 336. The second communication interface 350 can be implemented with similar techniques and methods as the implementation of the second controller interface 344.

[0097] The first communication unit 316 can be coupled to the communication path 104 to send information in the first device transmission 308 to the second device 106. The second device 106 can receive the information in the second communication unit 336 from the first device transmission 308 of the communication path 104.

[0098] The second communication unit 336 can be coupled to the communication path 104 to send information to the first device 102 in the second device transmission 310. The first device 102 can receive information from the second device transmission 310 of the communication path 104 in the first communication unit 316. The navigation system 100 can be executed by the first control unit 312, the second control unit 334, or a combination thereof. For illustrative purposes, the second device 106 is shown as having a partition with a second user interface 338, a second storage unit 346, a second control unit 334, and a second communication unit 336, but it should be understood that the second device 106 can have different partitions. For example, the second software 342 can be divided differently so that some or all of its functions can be in the second control unit 334 and the second communication unit 336. In addition, the second device 106 may include other functional units, which are not shown for clarity. Figure 3 middle.

[0099] The functional units in the first device 102 can work individually and independently of other functional units. The first device 102 can work individually and independently of the second device 106 and the communication path 104.

[0100] The functional units in the second device 106 can work individually and independently of other functional units. The second device 106 can work individually and independently of the first device 102 and the communication path 104.

[0101] For illustrative purposes, the navigation system 100 is described through the operation of the first device 102 and the second device 106. It should be understood that the first device 102 and the second device 106 can operate any modules and functions of the navigation system 100.

[0102] Now refer to Figure 4 , which shows Figure 1 4. The display interface may display a navigation map 402. The navigation map 402 is a visual representation of a geographic area. For example, the navigation map 402 may be a representation of a state, a city, a town, a neighborhood, or any portion thereof. As a further example, the navigation map 402 may include roads in the geographic area. The navigation map 402 may be displayed by Figure 3 a first display interface 330, Figure 3 The navigation map 402 may be displayed relative to the current location 404 .

[0103] The current location 404 may represent the physical location of the device or vehicle. As an example, the current location 404 may be represented by Figure 2 Position-motion sensor 212, Figure 3 The first position unit 320 or a combination thereof is generated. Also as an example, the current position 404 may provide Figure 1 The first device 102, Figure 2 The geographic location of vehicle 202, or a combination thereof.

[0104] The current location 404 can be generated in a variety of ways. For example, the current location 404 can be determined by a global positioning system (GPS), cellular triangulation, wireless fidelity (WiFi) triangulation, dead reckoning, or a combination thereof. Figure 1 The first device 102, Figure 1 The second device 106, the vehicle 202, or a combination thereof may track or provide the current location 404 dynamically and in real time.

[0105] The navigation system 100 tracks, provides, or receives the current location 404 "dynamically" to refer to receiving, monitoring (or a combination thereof) the current location 404, which is non-static or through potentially different mechanisms. The navigation system 100 tracks the current location 404 "in real time" to refer to receiving, monitoring, or a combination thereof as the current location 404 is read, regardless of the mechanism at a certain time period from the reading source of the current location 404 to the destination, thereby receiving the current location 404 without buffering delays for storage (rather than transmission). The navigation system 100 can display the current location 404 on the first display interface 206, the second display interface 234, or a combination thereof.

[0106] The navigation system 100 can display the current location 404 with a directionality 406. The directionality 406 is the direction of movement. The directionality 406 can be determined by calculating the change in the current location 404 or based on dead reckoning (e.g., using an accelerometer or gyroscope in the position-motion sensor 212, the first location unit 320, or a combination thereof). For example, the directionality 406 can be determined based on a change in degrees of latitude, longitude, altitude, or a combination thereof of the current location 404.

[0107] The navigation system 100 may also display a current road 408. The current road 408 is a road, path, traverse route, or portion thereof on which the vehicle 202, the first device 102, the second device 106, or a combination thereof is currently traveling. For example, the current road 408 may be a street, alley, freeway, parkway, highway, toll road, residential road, or unpaved path. The current road 408 may be determined based on the current location 404.

[0108] The navigation system 100 can display a travel route 410 of the vehicle 202, the first device 102, the second device 106, or a combination thereof for navigation. The travel route 410 is a path to navigate from the current location 404 to the route destination 412. For example, the travel route 410 can be determined based on the shortest distance to the route destination 412, the fastest time to reach the route destination 412, avoidance of tolls to the route destination 412, or other constraints.

[0109] Route destination 412 refers to a geographic location or point of interest. Route destination 412 may include an end point or termination point for a route or guidance. Route destination 412 may also include waypoints or intermediate stops. For example, route destination 412 may be a store, a landmark, an office building or site, a park, an address, a general geographic area, a street, a city or municipality, or a combination thereof. For another example, when guidance is terminated at a particular instance of a waypoint for a route, that waypoint may represent route destination 412.

[0110] For the sake of brevity, in this embodiment, Figure 3The first display interface 330 and the first user interface 318 are taken as an example, however, the description of the first display interface 330 and the first user interface 318 can be similarly applied to Figure 3 The second display interface 340 and the second user interface 338 are configured as follows.

[0111] In an embodiment, the first display interface 330 in conjunction with the first user interface 318 can enable a user of the navigation system 100 to input a route destination 412 to enable the navigation system 100 to generate the travel route 410. The user can input the route destination 412 by using alphanumeric characters, symbols, voice commands, gestures, or a combination thereof. For example, the user can input the route destination 412 by interfacing with the first user interface 318, the second user interface 338, or a combination thereof. As a further example, the user can input the route destination 412 by interfacing with the first user interface 318, the second user interface 338, or a combination thereof. Figure 3 The first audio interface 352, Figure 3 The route destination 412 may be inputted by interfacing with the second audio interface 354 or a combination thereof.

[0112] The navigation system 100 may process or utilize the current location 404 to obtain map information 414. The map information 414 refers to a graph or collection of data representing the arrangement or distribution of geographic features, physical features, non-physical features, or combinations thereof of a geographic location on the navigation map 402. For example, the map information 414 may include physical features such as paths, street names, infrastructure, geographic features, natural topology, points of interest, buildings, bodies of water, or combinations thereof. As another example, the map information 414 may also include non-physical features, such as speed limits, one-way designations, addresses, points of interest (POIs), or combinations thereof. As a specific example, the map information 414 may include traffic signs 416.

[0113] Traffic signs 416 provide instructions or information along the road. For example, traffic signs 416 may include various types of signs that control traffic flow, such as stop signs, yield signs, speed signs, one-way signs, no U-turn signs, or combinations thereof. As a further example, traffic signs 416 may include various types of signs that provide information associated with the road, such as school zone signs, warning signs, pedestrian signs, or combinations thereof.

[0114] The navigation map 402 may display a sign location 418 of the traffic sign 416. The sign location 418 is the geographic location of the traffic sign 416 relative to the current road 408. The sign location 418 may be displayed on the first display interface 330, the second display interface 340, or a combination thereof. For example, the sign location 418 may be on the left hand side of the current road 408, on the right hand side of the current road 408, above the current road 408, or a combination thereof. For another example, the sign location 418 may be before or after the current location 404 on the current road 408.

[0115] Reference now Figure 5 , which shows an example of a first frame 502 having a first image 504. The first frame 502 is composed of Figure 2 Monocular camera 215, Figure 1 The first device 102, Figure 1 The first frame 502 is an example of a picture or segment of a video 506 captured by the second device 106, or a combination thereof. The first frame 502 provides a visual representation of a view captured by the monocular camera 215, the first device 102, the second device 106, or a combination thereof at a particular time.

[0116] The video 506 includes a plurality of frames captured over a period of time. As an example, the video 506 may include the first frame 502. The video 506 provides a moving visual representation of the view of the monocular camera 215, the first device 102, the second device 106, or a combination thereof. For example, the video 506 may consist of instances of 30 frames per second. As another example, the video 506 may consist of instances of 60 frames per second. The video 506 may utilize Figure 3 a first display interface 330, Figure 3 The second display interface 340 or a combination thereof is used for display.

[0117] Returning to the example, first image 504 is an example of an image generated for an object or area in first frame 502. First image 504 is a visual representation of a view captured in first frame 502. For example, first image 504 may be an image of a pedestrian, a cyclist, a passenger car, a truck, a motorcycle, a bicycle, a road, a building, or a combination thereof in first frame 502. As a specific example, first image 504 may be traffic sign 416 in first frame 502. First image 504 may be a Figure 2 The vehicle control circuit 206, Figure 3 The first control unit 312, Figure 3 The second control unit 334 or a combination thereof generates or extracts.

[0118] Reference now Figure 6 , which shows an example of a second frame 602 having a second image 604. The second frame 602 is composed of Figure 2 Monocular camera 215, Figure 3 The first device 102, Figure 4 The second frame 602 provides a visual representation of a view captured by the monocular camera 215, the first device 102, the second device 106, or a combination thereof at a particular time. The second frame 602 may be an example of a frame in the video 506 captured by the second device 106, or a combination thereof. Figure 5506. The second frame 602 may be an example of a frame captured at a time after or after the first frame 502 of the video 506. For example, the second frame 602 may be a frame a few milliseconds after the first frame 502 in the video 506. As another example, the second frame 602 may be a frame a few seconds after the first frame 502 in the video 506.

[0119] The second frame 602 may include a second image 604. The second image 604 is an example of an image generated for an object or area within the second frame 602. For example, the second image 604 may be a pedestrian, a cyclist, a passenger car, a truck, a motorcycle, a bicycle, or a combination thereof in the second frame 602. As a specific example, the second image 604 may be a traffic sign 416 in the second frame 602. The second image 604 may be generated by Figure 2 The vehicle control circuit 206, Figure 3 The first control unit 312, Figure 3 The second control unit 334 or a combination thereof generates or extracts.

[0120] Now refer to Figure 7 , which shows an example control flow 700 of the navigation system 100. The control flow 700 may be used to track and locate a navigation system 100. Figure 4 The navigation system 100 may include a video capture module 702, a sign recognition module 704, a sign tracking module 706, a sign extraction module 708, a vehicle location module 710, a global positioning module 712, or a combination thereof.

[0121] Video capture module 702 may be coupled to landmark recognition module 704. Landmark recognition module 704 may be coupled to landmark tracking module 706. Landmark tracking module 706 may be coupled to landmark extraction module 708. Landmark extraction module 708 may be coupled to vehicle location module 710. Vehicle location module 710 may be coupled to global positioning module 712.

[0122] The modules may be coupled using wired or wireless connections, by including the output of one module as the input of another module, by including the operation of one module affecting the operation of another module, or a combination thereof. The modules may be coupled directly, with no intervening structures or objects other than connectors therebetween, or may be coupled indirectly.

[0123] The video capture module 702 is configured to Figure 2 Monocular camera 215, Figure 1 The first device 102, Figure 1 The second device 106 or a combination thereof captures Figure 5 The video capture module 702 can be based on the vehicle 202 Figure 2The video 506 can be captured by the video capture module 702 when the ignition state 224 is in the on position. When the ignition state 224 is in the on position, the video capture module 702 can automatically capture the video 506. The term "automatically" refers to the monocular camera 215, the first device 102, the second device 106, or a combination thereof capturing the video 215 without manual action for the specific purpose of capturing the video 215.

[0124] Video 506 may include frame 714 having image 716. Frame 714 may include Figure 5 The first frame 502, Figure 6 The first frame 502 and the second frame 602 may be captured in the order in which one frame is captured relative to the other frame. In other words, after the second frame 602 is captured, a subsequent frame may be captured and referred to as the first frame 502.

[0125] Image 716 may include Figure 5 The first image 504, Figure 6 The first image 504 and the second image 604, or a combination thereof. As with the first frame 502 and the second frame 602, the terms "first" and "second" are used for convenience and not to limit order or importance. As an example, the first image 504 and the second image 604 can be swapped in the order in which the first frame 502 and the second frame 602 are captured one frame relative to the other frame. In other words, after the second image 604 is captured in the second frame 602, a subsequent image can be captured and when the subsequent frame is referred to as the first frame 502, the subsequent image can be referred to as the first image 504.

[0126] Video capture module 702 available Figure 2 The vehicle control circuit 206, Figure 3 The first control unit 312, Figure 3 The video capture module 702 may store the video 506, the frame 714, the first frame 502, the second frame 602, the image 716, the first image 504, the second image 604, or a combination thereof in the second control unit 334 or a combination thereof. The video capture module 702 may store the video 506, the frame 714, the first frame 502, the second frame 602, the image 716, the first image 504, the second image 604, or a combination thereof in the second control unit 334 or a combination thereof. Figure 2 The vehicle storage circuit 208, Figure 3 The first storage unit 314, Figure 3 The video capture module 702 can use Figure 2 The vehicle communication circuit 204, Figure 3 The first communication unit 316, Figure 3The second communication unit 336 or a combination thereof is used to transmit the video 506, the frame 714, the first frame 502, the second frame 602, the image 716, the first image 504, the second image 604, or a combination thereof.

[0127] The control flow may be passed from the video capture module 702 to the landmark recognition module 704. For example, the control flow may pass the processing result as an output from the video capture module 702 to an input of the landmark recognition module 704.

[0128] The sign recognition module 704 is configured to detect a sign 718 in the video 506, the frame 714, the first frame 502, the second frame 602, or a combination thereof. The sign 718 can be any physical display (including Figure 4 The sign recognition module 704 can use the sign detection model 720 to detect the sign 718, the traffic sign 416, or a combination thereof in the first frame 502, the second frame 602, or a combination thereof.

[0129] Sign detection model 720 may be an artificial intelligence or machine learning implementation that may be trained to learn, determine, or detect signs 718 from video 506. For example, sign detection model 720 may be trained to learn to detect various instances of signs 718, including types of traffic signs 416, from video 506, frames 714, images 716 in frames 714, or a combination thereof.

[0130] Sign detection model 720 may be trained with information from a database or other input with pictures of various instances of sign 718. For example, sign detection model 720 may also detect traffic sign 414 as one of signs 718. Also as an example, the information or pictures of sign 718 may represent traffic sign 414 at many angles, partial views, lighting, colors, sign types, clarity, various weather conditions, or combinations thereof. Sign detection model 720 may be trained and learn to detect various instances of sign 718 even though traffic sign 414 is not an exact example of the information or pictures used for training.

[0131] The landmark detection model 720 can be implemented in a variety of ways. For example, the landmark detection model 720 can be implemented with a neural network, such as a convolutional neural network, a fully connected network, or a combination thereof. For another example, the landmark detection model 720 can include unsupervised learning and other forms of supervised learning. The landmark detection model 720 can work with artificial intelligence or machine learning that provides a global minimum, one or more local minima, or a combination thereof.

[0132] Continuing with this example, the sign detection model 720 can detect various instances of the sign 718 from the video 506, the frame 714, the image 716 in the frame 714, or a combination thereof. For example, the sign detection model 720 can also detect the traffic sign 414 as one of the signs 718. As a specific example, the sign detection model 720 can detect the sign 718 in the first frame 502, the second frame 602, or a combination thereof. Also as a specific example, the sign detection model 720 can detect a stop sign as the traffic sign 414 in the first frame 502, the second frame 602, or a combination thereof. As another example, the sign detection model 720 can detect a speed sign as the traffic sign 414 in the first frame 502, the second frame 602, or a combination thereof.

[0133] Continuing with this example, sign detection model 720 can detect plurality of signs 718. As a specific example, one of signs 718 can include traffic sign 414. Sign detection model 720 can determine an instance of traffic sign 414 from among plurality of signs 718.

[0134] The sign recognition module 704, the sign detection model 720, or a combination thereof may detect the sign 718, the traffic sign 414, or a combination thereof using the vehicle control circuit 206, the first control unit 312, the second control unit 334, or a combination thereof. The sign recognition module 704 may communicate the sign 718, the traffic sign 414, or a combination thereof using the vehicle communication circuit 204, the first communication unit 316, the second communication unit 336, or a combination thereof. The flow may proceed to the sign tracking module 706 to process the image 716 containing the sign 718, the traffic sign 414, or a combination thereof detected by the sign detection model 720.

[0135] The sign tracking module 706 can utilize the traffic signs 414 detected by the sign recognition module 704 to track each instance of the traffic sign 414 in the video 506, the frames 714, the images 716 in the frames 714, or a combination thereof. Each instance of the traffic sign 414 can be included in a portion of the video 506, the frames 714, the images 716 in the frames 714, or a combination thereof, and need not be present or detected in the entire video 506 or all frames 714.

[0136] For example, the sign tracking module 706 can analyze the first frame 502, the second frame 602, or a combination thereof, including each of the instances of the traffic sign 414 from the video 506. The sign tracking module 706 can also analyze multiple instances of the first frame 502, the second frame 602, or a combination thereof. The sign tracking module 706 can capture an image 716 containing an instance of the traffic sign 414 from the first frame 502, the second frame 602, or a combination thereof.

[0137] The first frame 502 and the second frame 602 may be adjacent to each other and in a consecutive order in the frames 714 in the video 506. The first frame 502 and the second frame 602 may also not be adjacent to each other and not in a consecutive order in the frames 714 in the video 506. The position of the first frame 502 and the second frame 602 relative to each other in the frames 714 or within the video 506 is not limited by how far apart one frame is relative to the other.

[0138] As an example, the sign tracking module 706 detects each instance of the traffic sign 414 throughout the video 506 and across the frames 714, regardless of the location of the frames 714 within the video 506. Also as an example, the sign tracking module 706 monitors and tracks each instance of the traffic sign 414 across the video 506, the frames 714, the images 716 of the frames 714, or a combination thereof. The sign tracking module 706 can detect each instance of the traffic sign 414 on the video 506 individually, collectively, as a subset, or a combination thereof.

[0139] Continuing with the example, the sign tracking module 706 can extract an image 716 of each instance of the traffic sign 414 from the frame 714. For example, the sign tracking module 706 can extract the first image 504 of the traffic sign 414 from the first frame 502 and the second image 604 of the traffic sign 414 from the second frame 602. The sign tracking module 706 can track or match each instance of the traffic sign 414 in the first image 504, the second image 604, or a combination thereof using the similarity model 722.

[0140] The similarity model 722 measures the similarity between each instance of the traffic sign 414 between the first image 504, the second image 604, or a combination thereof. The similarity model 722 can determine whether the instances of the traffic sign 414 in the first image 504, the second image 604, or a combination thereof represent the same instance of the traffic sign 414. For example, the similarity model 722 can utilize a deep learning model to determine the degree of similarity.

[0141] The similarity model 722 can be implemented in a variety of ways. For example, the similarity model 722 can be implemented using a neural network, such as a convolutional neural network, a fully connected network, or a combination thereof. For another example, the similarity model 722 can include unsupervised learning and other forms of supervised learning. The similarity model 722 can work with artificial intelligence or machine learning that provides a global minimum, one or more local minima, or a combination thereof.

[0142] Similar to the sign detection model 720, the similarity model 722 can be an artificial intelligence or machine learning implementation that can be trained to learn, determine, or detect the type of traffic sign 414 from the video 506. For example, the similarity model 722 can be trained to learn to detect various instances of the traffic sign 414 from the video 506, the frame 714, the image 716 in the frame 714, or a combination thereof. The similarity model 722 can be trained with information from a database or other input with pictures of various instances of the traffic sign 414. Also as an example, the information or pictures of the traffic sign 414 can represent the traffic sign 414 at many angles, partial views, lighting, color, sign type, clarity, various weather conditions, or a combination thereof. The similarity model 722 can be trained and learn to detect the traffic sign 414 even if the traffic sign 414 is not an exact example of the information or pictures used for training.

[0143] Additionally, the similarity model 722 can be trained to learn to detect specific instances of the traffic sign 414 across the video 506, the frames 714, the images 716 of the frames 714, or a combination thereof. In other words, the similarity model 722 can be used to match each specific instance of the traffic sign 414 in each frame 714 as needed, and beyond the general detection of any instance or type of traffic sign 414.

[0144] The sign tracking module 706 can detect the traffic sign 414 in the first frame 502, the second frame 602, the first image 504, the second image 604, or a combination thereof using the vehicle control circuit 206, the first control unit 312, the second control unit 334, or a combination thereof. The sign tracking module 706 can obtain the image 716, the first image 504, the second image 604, or a combination thereof of the traffic sign 414 using the vehicle communication circuit 204, the first communication unit 316, the second communication unit 336, or a combination thereof. The sign tracking module 706 can determine that the traffic sign 414 in the first frame 502 and the second frame 602 is the same instance of the traffic sign 414 using the vehicle control circuit 206, the first control circuit 312, the second control circuit 334, or a combination thereof.

[0145] After tracking the traffic sign 414 in the video 506, the frame 714, the image 716 in the frame 714, or a combination thereof, control can pass to the sign extraction module 708. For example, control can pass the processed result as an output from the sign tracking module 706 to an input of the sign extraction module 708.

[0146] The sign extraction module 708 is configured to process each of the images 716 in the frame 714, including the first image 504, the second image 604, or a combination thereof. The sign extraction module 708 processes the first image 504, the second image 604, or a combination thereof by extracting features 724 representing, related to, or associated with the sign 718, the traffic sign 414, or a combination thereof in the frame 714, the first frame 502, the second frame 602, or a combination thereof. For example, the sign extraction module 708 can utilize KAZE features to process the frame 714, the first image 504, the second image 604, or a combination thereof. As another example, the sign extraction module 708 can utilize ORB features to process the frame 714, the first image 504, the second image 604, or a combination thereof.

[0147] Feature 724 is an attribute or characteristic of, or is associated with, sign 718 or traffic sign 414 in first image 504, second image 604, or a combination thereof. For example, feature 724 may include the shape of sign 718, traffic sign 414, or a combination thereof. As a further example, feature 724 may include text, numbers, or a combination thereof of or associated with sign 718, traffic sign 414, or a combination thereof. For further example, feature 724 may also include symbols, such as arrows, lines, outlines of animals, or a combination thereof.

[0148] The sign extraction module 708 can determine features 724 associated with the sign 718, the traffic sign 414, or a combination thereof from the frame 714 and the image 716 in the frame 714. The sign extraction module 708 can determine features 724 associated with the sign 718, the traffic sign 414, or a combination thereof based on information not directly from the frame 714 or the image 714, but from other information associated with the image 716.

[0149] As an example, the sign extraction module 708 can pull map data using the geographic location tagged with the frame 714 to confirm the presence or location of the sign 718, the traffic sign 414, or a combination thereof. As a further example, the sign extraction module 708 can determine the features 724 associated with the presence or location of the sign 718, the traffic sign 414, or a combination thereof, such as an intersection, a railroad track, a construction project along a road, or a combination thereof. The sign extraction module 708 can determine the presence or location of the sign 718, the traffic sign 414, or a combination thereof, such as an intersection or a railroad crossing, from an external source based on the map data that is typically associated with the sign 718.

[0150] The sign extraction module 708 can determine the features 724 associated with the sign 718, the traffic sign 414, or a combination thereof between frames 714 and between images 716 in the frames 714. For example, the sign extraction module 708 can determine the same instance of the features 724 of the traffic sign 414 in the first image 504 of the first frame 502, the second image 604 of the second frame 602, or a combination thereof. As a further example, the sign extraction module 706 can determine the same instance of the features 724 of the sign 414 in the first image 504 of the first frame 502, the second image 604 of the second frame 602, or a combination thereof.

[0151] The sign extraction module 708 can process the first image 504, the second image 604, or a combination thereof using the vehicle control circuit 206, the first control unit 312, the second control unit 334, or a combination thereof. The sign extraction module 708 can transmit the feature 724 of the traffic sign 414 using the vehicle communication circuit 204, the first communication unit 316, the second communication unit 336, or a combination thereof. The sign extraction module 708 can store the feature 724 of the traffic sign 414 using the vehicle storage circuit 208, the first storage unit 314, the second storage unit 346, or a combination thereof.

[0152] When the sign extraction module 708 determines that the features 724 of the traffic sign 414 are found in the first image 504 and the second image 604, control can pass to the vehicle location module 710. The vehicle location module 710 can utilize the traffic sign 414 found in the first image 504, the second image 604, or a combination thereof.

[0153] The vehicle position module 710 is configured to determine a vehicle relative position 726 based on the first frame 502, the second frame 602, or a combination thereof. The vehicle relative position 726 is the position of the vehicle 202 relative to the traffic sign 414 in the first frame 502, the second frame 602, or a combination thereof. The vehicle position module 710 measures the movement of the vehicle 202 based on features 724 of the traffic sign 414 in the first frame 502, the second frame 602, or a combination thereof. For example, the vehicle position module 710 can determine the vehicle relative position 726 based on inertial measurement unit readings 728, camera compensation 730, or a combination thereof.

[0154] The inertial measurement unit readings 728 are measurements of the translation and rotation of the vehicle 202 between the first frame 502 and the second frame 602. The inertial measurement unit readings 728 may be obtained from Figure 2 Position-motion sensor 212, Figure 3The first position unit 320 or a combination thereof obtains the translation and rotation of the vehicle 202. For example, the inertial measurement unit readings 728 can provide the movement of the vehicle 202 based on an accelerometer, a gyroscope, or a combination thereof. As another example, the inertial measurement unit readings 728 can be synchronized with the monocular camera 215, the first device 102, the second device 106, or a combination thereof to obtain the translation and rotation of the vehicle 202 between the first frame 502, the second frame 602, or a combination thereof.

[0155] The camera compensation 730 is an adjustment calculated to determine the accurate distance and depth of the object captured in the image 716. The vehicle position module 710 generates the camera compensation 730 based on the parameters of the capture camera capturing the first frame 502, the second frame 602, or a combination thereof. For example, the camera compensation 730 is calculated for the monocular camera 215, the first device 102, the second device 106, or a combination thereof.

[0156] The camera compensation 730 can allow the distance and depth of the traffic sign 414 to be correctly determined based on the first image 504, the second image 604, or a combination thereof. As a specific example, the vehicle position module 710 can generate the camera compensation 730 for the monocular camera 215 to determine the correct distance and depth of the traffic sign 414 in the first image 504, the second image 604, or a combination thereof.

[0157] The vehicle position module 710 can calculate the vehicle relative position 726, the camera compensation 730, or a combination thereof using the vehicle control circuit 206, the first control unit 312, the second control unit 334, or a combination thereof. The vehicle position module 710 can obtain the inertial measurement unit readings 728 using the vehicle communication circuit 204, the first communication unit 316, the second communication unit 336, or a combination thereof. The vehicle position module 710 can store the vehicle relative position 726, the inertial measurement unit readings 728, the camera compensation 730, or a combination thereof using the vehicle storage circuit 208, the first storage circuit 314, the second storage circuit 346, or a combination thereof.

[0158] After determining the vehicle relative position 726 based on the traffic sign 414 in the first frame 502, the second frame 602, or a combination thereof, control can be passed to the global positioning module 712. For example, control can pass the processed result as an output of the vehicle position module 710 to an input of the global positioning module 712.

[0159] The global positioning module 712 is configured to generate global coordinates 732 of the traffic sign 414. The global coordinates 732 are latitude and longitude coordinates on the navigation map 402. The global coordinates 732 of the traffic sign 414 may be determined based on the three-dimensional coordinates 734, global navigation satellite system readings 736, or a combination thereof.

[0160] The three-dimensional coordinates 734 are reconstructions of the traffic sign 414, the vehicle 202, or a combination thereof based on the vehicle relative position 726, the sign position 418, or a combination thereof. The three-dimensional coordinates 734 determine the distance between objects in three dimensions. The three-dimensional coordinates 734 may be generated based on the vehicle relative position 726, the sign position 418, the feature 724 of the traffic sign 414, or a combination thereof between the first frame 502 and the second frame 602. The three-dimensional coordinates 734 may calculate the distance of the traffic sign 414 by calculating the change in the vehicle relative position 726, the sign position 418, the feature 724, or a combination thereof of the traffic sign 414 between the first frame 502 and the second frame 602.

[0161] The global navigation satellite system reading 736 is a reading of the position-motion sensor 212, the first position unit 320, or a combination thereof. The global navigation satellite system reading 736 can provide a global coordinate 732 of the vehicle 202, the first device 102, the second device 106, or a combination thereof. The global navigation satellite system reading 736 can be obtained for the first frame 502, the second frame 602, or a combination thereof.

[0162] The global positioning module 712 calculates the global coordinates 732 of the traffic sign 414 using the distance between the vehicle 202 and the traffic sign 414, which is calculated using the three-dimensional coordinates 734 of the first frame 502, the second frame 602, or a combination thereof and the global navigation satellite system readings 736. The global positioning module 712 can display the traffic sign 414 on the navigation map 402 using the global coordinates 732 of the traffic sign 414.

[0163] The global positioning module 712 can calculate the three-dimensional coordinates 734, the global coordinates 732, or a combination thereof of the traffic sign 414 using the vehicle control circuit 206, the first control unit 312, the second control unit 334, or a combination thereof. The global positioning module 712 can transmit the three-dimensional coordinates 734, the global navigation satellite system readings 736, or a combination thereof using the vehicle communication circuit 204, the first communication unit 316, the second communication unit 336, or a combination thereof. The global positioning module 712 can display the global coordinates 732 of the traffic sign 414 on the navigation map 402 using the first display interface 330, the second display interface 340, or a combination thereof.

[0164] It has been discovered that the navigation system 100 provides improved operating safety of the vehicle 202 by displaying the traffic sign 414 , the sign location 418 , or a combination thereof on the navigation map 402 using the global coordinates 732 of the traffic sign 414 to increase awareness of the user of the vehicle 202 .

[0165] It is further discovered that the navigation system 100 can utilize a cost-effective monocular camera 215 mechanism to track and locate the traffic sign 414 by calculating the global coordinates 732 of the traffic sign 414 using the three-dimensional coordinates 734, the global navigation satellite system readings 736, or a combination thereof.

[0166] It is further found that the navigation system 100 can verify the global coordinates 732 of the traffic sign 414 on the navigation map 402 using the three-dimensional coordinates 734, the global navigation satellite system readings 736, or a combination thereof based on the video 506, the frame 714, the image 716 of the frame 714, or a combination thereof. The navigation system 100 can also update the global coordinates 732 of the traffic sign 414 on the navigation map 402 based on the global coordinates 732 determined based on the three-dimensional coordinates 734, the global navigation satellite system readings 736, or a combination thereof.

[0167] The navigation system 100 has been described using module functions or sequences as examples. The navigation system 100 may divide the modules differently or sequence the modules differently. For example, the landmark recognition module 704 may be coupled to the video capture module 702. As a further example, the landmark extraction module 708 may be coupled to the landmark recognition module 704.

[0168] For the purpose of illustration, various modules have been described as being specific to the first device 102 or the second device 106. However, it should be understood that the modules can be distributed differently. For example, various modules can be implemented in different devices, or the functions of the modules can be distributed on multiple devices. Also as an example, various modules can be stored in non-transient memory media.

[0169] As a more specific example, the one or more modules described above may be stored in a non-transitory memory medium for distribution to different systems, different devices, different users, or a combination thereof, for manufacturing, or a combination thereof. Also as a more specific example, the modules described above may be implemented or stored using a single hardware unit such as a chip or processor or across multiple hardware units.

[0170] The modules described in this application may be hardware implementations or hardware accelerators in the first control unit 312 or the second control unit 334. These modules may also be hardware implementations or hardware accelerators located in the first device 102 or the second device 106 but located outside the first control unit 312 or the second control unit 334, respectively, such as Figure 3 However, it should be understood that the first control unit 312, the second control unit 334, or a combination thereof may collectively refer to all hardware accelerators for these modules.

[0171] The modules described in this application can be implemented as instructions stored on a non-transitory computer-readable medium to be executed by the first control unit 312, the second control unit 334, or a combination thereof. The non-transitory computer medium may include the first storage unit 314, the second storage unit 346, or a combination thereof. The non-transitory computer-readable medium may include a non-volatile memory, such as a hard drive, a non-volatile random access memory (NVRAM), a solid-state storage device (SSD), a compact disk (CD), a digital video disk (DVD), or a universal serial bus (USB) flash memory device. The non-transitory computer-readable medium may be integrated as a part of the navigation system 100 or installed as a removable part of the navigation system 100.

[0172] The physical transformation from determining global coordinates 732 of traffic sign 414 results in displaying traffic sign 414 in the physical world, such as displaying sign location 418 on navigation map 402. Movement in the physical world (e.g., movement of traffic sign 414) results in changes to sign location 418 by updating navigation map 402.

[0173] Now refer to Figure 8 , which shows a flow chart of an operation method 800 of the navigation system 100 in an embodiment of the present invention. The method 800 includes: in box 802, receiving a plurality of frames of images, including a first frame and a second frame; in box 804, detecting a traffic sign from the image between the first frame and the second frame based on a sign recognition model; in box 806, extracting a first image from the first frame and extracting a second image from the second frame; in box 808, matching the traffic signs in the first image and the second image to be the same based on a similarity model; in box 810, generating a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; and in box 812, generating global coordinates of the sign position to be displayed on a navigation map.

[0174] The resulting method, process, apparatus, device, product and / or system is straightforward, cost-effective, uncomplicated, highly versatile, accurate, sensitive and efficient, and may be implemented by adapting known components to readily available, efficient and economical manufacture, application and utilization. Another important aspect of embodiments of the present invention is that it valuable supports and serves the historical trend to reduce costs, simplify systems and improve efficiency.

[0175] Accordingly, these and other valuable aspects of embodiments of the present invention advance the state of the technology to at least the next level.

[0176] Although the present invention has been described in conjunction with a specific best mode, it should be understood that, based on the foregoing description, many substitutions, modifications and variations will be clear to those skilled in the art. Therefore, the present invention is intended to include all such substitutions, modifications and variations that fall within the scope of the included claims. All contents set forth herein or shown in the accompanying drawings should be interpreted as illustrative rather than restrictive.

Claims

1. A method for operating a navigation system, comprising: receiving a plurality of frames of an image, including a first frame and a second frame; detecting a traffic sign from an image between a first frame and a second frame based on a sign recognition model; extracting a first image from the first frame, and extracting a second image from the second frame; Based on the similarity model, matching the traffic signs in the first image and the second image to be the same; generating a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; generating a vehicle relative position associated with a monocular camera for receiving the plurality of frames; generating a camera compensation for the monocular camera to determine a correct distance and depth of the traffic sign in the first image, the second image, or a combination thereof; and generating global coordinates for the location of the landmark for display on a navigation map, Wherein generating the global coordinates for the sign position includes triangulating the sign position based on the vehicle relative position, the camera compensation, and the matching of the traffic sign.

2. The method of claim 1, wherein: Generating the landmark positions includes generating three-dimensional coordinates using the inertial measurement unit readings.

3. The method of claim 1, wherein receiving a plurality of frames of images comprises receiving a video from a monocular camera.

4. The method of claim 1, wherein: Generating the global coordinates for the landmark location includes generating the global coordinates based on the inertial measurement unit readings and global navigation satellite system readings.

5. A navigation system comprising: a communication unit configured to receive a plurality of frames of an image, the plurality of frames including a first frame and a second frame; as well as a control unit coupled to the communication unit, the control unit being configured to: detecting a traffic sign from an image between a first frame and a second frame based on a sign recognition model; extracting a first image from the first frame, and extracting a second image from the second frame; Based on the similarity model, matching the traffic signs in the first image and the second image to be the same; generating a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; generating a vehicle relative position associated with a monocular camera for receiving the plurality of frames; generating a camera compensation for the monocular camera to determine a correct distance and depth of the traffic sign in the first image, the second image, or a combination thereof; and generating global coordinates for the location of the landmark for display on a navigation map, Wherein generating the global coordinates for the sign position includes triangulating the sign position based on the vehicle relative position, the camera compensation, and the matching of the traffic sign.

6. The system of claim 5, wherein: The control unit is configured to generate three-dimensional coordinates using the inertial measurement unit readings.

7. The system of claim 5, wherein: The communication unit is configured to receive video from a monocular camera.

8. The system of claim 5, wherein: The control unit is configured to generate the global coordinates based on the inertial measurement unit readings and global navigation satellite system readings.

9. A non-transitory computer-readable medium comprising instructions executable by control circuitry for a navigation system, the instructions comprising: receiving a plurality of frames of an image, including a first frame and a second frame; detecting a traffic sign from an image between a first frame and a second frame based on a sign recognition model; extracting a first image from the first frame, and extracting a second image from the second frame; Based on the similarity model, matching the traffic signs in the first image and the second image to be the same; generating a sign position of the traffic sign using an inertial measurement unit reading based on the first image and the second image; generating a vehicle relative position associated with a monocular camera for receiving the plurality of frames; generating a camera compensation for the monocular camera to determine a correct distance and depth of the traffic sign in the first image, the second image, or a combination thereof; and generating global coordinates for the location of the landmark for display on a navigation map, Wherein generating the global coordinates for the sign position includes triangulating the sign position based on the vehicle relative position, the camera compensation, and the matching of the traffic sign.

10. The non-transitory computer readable medium of claim 9, wherein: The instructions include generating three-dimensional coordinates using the inertial measurement unit readings.

11. The non-transitory computer readable medium of claim 9, wherein: Receiving a plurality of frames of an image includes receiving a video from a monocular camera.

12. The non-transitory computer readable medium of claim 9, wherein: The instructions include generating the global coordinates for the landmark location, including generating the global coordinates based on the inertial measurement unit readings and global navigation satellite system readings.

Images