Navigation system with independent positioning mechanism and method of operating the same

Abstract

A method of operating a navigation system includes determining a geographic position of a vehicle when a global positioning position is blocked by an obstacle and the geographic position is from an in-vehicle position sensor in the vehicle; extracting image lines from an environment image using an artificial intelligence model and the environment image is from an in-vehicle image sensor in the vehicle; transforming image coordinates of the image lines to world coordinates locally stored from an HD map based on the geographic position; extracting map lines from the HD map local storage based on the world coordinates; determining a pose relationship between the paired image lines and the map lines is horizontal or vertical; generating a map-related position of the vehicle based on the geographic position and the pose relationship; and transmitting the map-related position for display on a user interface.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to navigation systems, and more specifically to systems having an independent positioning mechanism. Background Technology

[0002] With the advent of autonomous vehicles, numerous shortcomings have been discovered in the mapping strategies used by most location-based navigation systems. Relying on satellite-based GPS to determine accurate location has its drawbacks.

[0003] As the number of autonomous vehicles on the road increases, relying solely on satellite-based navigation for accurate location and positioning becomes unreliable. The accuracy required to operate autonomous vehicles must be supported to navigate in tight spaces where any inch could mean an accident.

[0004] Therefore, the need for navigation systems with independent positioning mechanisms remains. Given increasing competitive pressures, rising consumer expectations, and diminishing opportunities for meaningful product differentiation in the market, finding answers to these questions is becoming increasingly critical. Furthermore, the need to reduce costs, improve efficiency and performance, and meet competitive pressures further intensifies the critical necessity of finding answers to these questions.

[0005] Solutions to these problems have been sought for a long time, but existing developments have not taught or suggested any solutions, and therefore those skilled in the art have long been unable to obtain solutions to these problems. Summary of the Invention

[0006] Embodiments of the present invention provide a method for operating a navigation system, comprising: determining the geographical location of a vehicle when its global positioning location is obstructed by an obstacle and its geographical location is derived from an in-vehicle position sensor within the vehicle; extracting image lines from an environmental image using an artificial intelligence model, wherein the environmental image is derived from an in-vehicle image sensor within the vehicle; transforming the image coordinates of the image lines into world coordinates stored locally in a high-resolution (HD) map based on the geographical location; extracting map lines from the local HD map storage based on the world coordinates; determining whether the pose relationship between paired image lines and map lines is horizontal or vertical; generating a map-related location of the vehicle based on the geographical location and the pose relationship; and transmitting the map-related location for display on a user interface.

[0007] Embodiments of the present invention provide a navigation system including a control circuit and an interface circuit. The control circuit is configured to: determine the geographical location of a vehicle based on the fact that its global positioning location is obstructed by obstacles and its geographical location is derived from an in-vehicle position sensor within the vehicle; extract image lines from an environmental image using an artificial intelligence model, wherein the environmental image is derived from an in-vehicle image sensor within the vehicle; transform the image coordinates of the image lines into world coordinates stored locally on a high-resolution (HD) map based on the geographical location; extract map lines from the local HD map storage based on the world coordinates; determine whether the pose relationship between paired image lines and map lines is horizontal or vertical; and generate a map-related position of the vehicle based on the geographical location and the pose relationship. The interface circuit is configured to transmit the map-related position for display on a user interface.

[0008] Embodiments of the present invention provide a non-transitory computer-readable medium including instructions for a navigation system, comprising: determining the geographic location of a vehicle based on an obstacle obstructing its global positioning location and the geographic location being derived from an in-vehicle position sensor within the vehicle; extracting image lines from an environmental image, wherein the environmental image is derived from an in-vehicle image sensor within the vehicle, using an artificial intelligence model; transforming the image coordinates of the image lines into world coordinates stored locally in a high-resolution (HD) map based on the geographic location; extracting map lines from the local HD map storage based on the world coordinates; determining whether a pose relationship between paired image lines and map lines is horizontal or vertical; generating a map-related location of the vehicle based on the geographic location and the pose relationship; and transmitting the map-related location for display on a user interface.

[0009] Some embodiments of the present invention have steps or elements other than those described above, or have other steps or elements in place of those described above. These steps or elements will become clear to those skilled in the art from the following detailed description taken with reference to the accompanying drawings. Attached Figure Description

[0010] Figure 1 This is a block diagram of a navigation system with an independent positioning mechanism in an embodiment of the present invention.

[0011] Figure 2 This is a schematic diagram of an example of a first device configured to provide an independent positioning mechanism in an embodiment.

[0012] Figure 3 This is an example object detection map of the environment scene processed by the navigation system.

[0013] Figure 4 This is an exemplary functional block diagram of the independent positioning mechanism of a navigation system.

[0014] Figure 5 This is an exemplary block diagram of the navigation system in the embodiment.

[0015] Figure 6 yes Figure 1 An exemplary block diagram of the independent positioning module.

[0016] Figure 7 An exemplary control flowchart for the independent positioning mechanism of a navigation system.

[0017] Figure 8 This is a flowchart of the operation method of the navigation system in an embodiment of the present invention. Detailed Implementation

[0018] The following embodiments can accurately identify lanes and objects, enabling vehicle movement control for operating or controlling the physical movement of a vehicle. Vehicle movement control can be based on driver-assisted or autonomous vehicle driving processing, which is safe and reliable due to the accuracy of lane and object detection.

[0019] Vehicle mobility control can also be based on accurate identification of lane position, objects, and restrictions to ensure that driver-assisted or autonomous vehicles can be driven without the risk of damaging the vehicle or any adjacent objects or property.

[0020] The following embodiments are described in sufficient detail to enable those skilled in the art to perform and use the invention. It will be understood that other embodiments will be apparent based on this disclosure, and system, process, or mechanical changes may be made without departing from the scope of the embodiments of the invention.

[0021] Numerous specific details are set forth in the following description to provide a thorough understanding of the invention. However, it will be apparent that the invention may be practiced without these specific details. To avoid obscuring embodiments of the invention, some well-known circuits, system configurations, and processing steps are not disclosed in detail.

[0022] The accompanying drawings illustrating embodiments of the system are semi-illustrative and not to scale, and in particular, some dimensions are exaggerated for clarity and are shown in the drawings. Similarly, although the views in the drawings generally show similar orientations for ease of description, such depictions in the drawings are arbitrary for most purposes. Generally, the invention can operate in any orientation. Embodiments of various components are provided for ease of description and are not intended to have any other meaning or to provide limitation on embodiments of the invention.

[0023] Those skilled in the art will understand that the format used to express navigation information is not critical to some embodiments of the invention. For example, in some embodiments, navigation information is presented in the format (X, Y, Z); where X, Y, and Z are three coordinates such as latitude, longitude, and altitude that define the geographic location (i.e., the location of the vehicle).

[0024] Depending on the context in which the term "module" is used, in this invention, the term "module" as referred to herein can include or be implemented as or comprise software running on dedicated hardware, hardware, or a combination thereof. For example, software can be machine code, firmware, embedded code, and application software. Software can also include functions, calls to functions, blocks of code, or combinations thereof.

[0025] Furthermore, for example, hardware can be a gate, circuit system, processor, computer, integrated circuit, integrated circuit core, memory device, pressure sensor, inertial sensor, microelectromechanical system (MEMS), passive device, physical non-transient storage medium including instructions for performing software functions, a portion thereof, or a combination thereof, to control one or more hardware units or circuits. Additionally, if the word "unit" is written in the following system claims section, "unit" is considered to include the hardware circuit system for the purposes and scope of the system claims.

[0026] The units described below in the embodiments may be coupled or attached to each other as described or shown. The coupling or attachment may be direct or indirect, with or without intermediate items between the coupled or attached modules or units. The coupling or attachment may be achieved through communication between modules or units, such as wireless communication, or through physical contact.

[0027] The term “obstacle” or “barrier” as used in the specification and claims means an object that blocks an area or location from access to the Global Positioning System, including objects such as canyon walls, mountains, skyscrapers, walls of indoor parking garages, tunnels, electric storms, or extreme weather patterns.

[0028] It should also be understood that the nouns or elements in the embodiments can be described as singular instances. It should be understood that the use of the singular is not limited to the singular, but rather that the use of the singular can apply to plural instances of any specific noun or element in this application. Many instances may be the same or similar or may be different.

[0029] Now for reference Figure 1 The diagram illustrates a block diagram of a navigation system 100 with an independent positioning mechanism according to an embodiment of the present invention. The navigation system 100 may include a first device 102, such as a vehicle capable of functioning as a client or server, connected to a second device 106, such as a client or server.

[0030] The navigation system 100 may include a system for identifying independent locations based on the fusion of multiple sources to coordinate and quickly identify the current location to aid in making lane change or route change decisions. The first device 102 may communicate with the second device 106 via a network 104, such as a wireless or wired network.

[0031] For example, the first device 102 can be any computing device among various computing devices, such as a cellular phone, personal digital assistant, laptop computer, wearable device, Internet of Things (IoT) device, automotive telematics navigation system, or other multifunctional device. Furthermore, for example, the first device 102 can include a device or subsystem, an autonomous or self-driving vehicle or object, a driver-assisted vehicle, a remote-controlled vehicle or object, or a combination thereof.

[0032] The first device 102 may be directly or indirectly coupled to the network 104 to communicate with the second device 106, or it may be a standalone device. The first device 102 may also be standalone or integrated with a vehicle such as a car, truck, bus, motorcycle, or drone.

[0033] For illustrative purposes, the navigation system 100 is described as a vehicle including a telematics system, with the first device 102 as the reference, although it should be understood that the first device 102 can be of different types of devices. For example, the first device 102 can also be a non-mobile computing device, such as a server, server cluster, or desktop computer.

[0034] The second device 106 can be any computing device in a variety of centralized or decentralized computing devices. For example, the second device 106 can be a computer, a grid computing resource, a virtualized computing resource, a cloud computing resource, a router, a switch, a peer-to-peer distributed computing device, or a combination thereof.

[0035] The second device 106 can be centralized in a single room, distributed across different rooms, distributed across different geographical locations, or embedded within a telecommunications network. The second device 106 can be coupled to network 104 to communicate with the first device 102. The second device 106 can also be a client-type device as described with respect to the first device 102.

[0036] For illustrative purposes, the navigation system 100 is described as a non-mobile computing device, although it should be understood that the second device 106 may also be a different type of computing device. For example, the second device 106 may also be a mobile computing device, such as a laptop computer, another client device, a wearable device, or a different type of client device.

[0037] Furthermore, for illustrative purposes, the navigation system 100 is described as a computing device, with the second device 106 as the primary device, although it should be understood that the second device 106 can also be a different type of device. Additionally, for illustrative purposes, the navigation system 100 is shown as having the second device 106 and the first device 102 as endpoints of the network 104, although it should be understood that the navigation system 100 can also include different partitions between the first device 102, the second device 106, and the network 104. For example, the first device 102, the second device 106, or a combination thereof can also serve as part of the network 104.

[0038] Network 104 can span and represent various networks. For example, network 104 can include wireless communication, wired communication, optical, ultrasonic, or combinations thereof. Satellite communication, cellular communication, Bluetooth, Infrared Data Association (IrDA) standards, Wi-Fi, and WiMAX are examples of wireless communication that can be included in the communication path. Ethernet, Digital Subscriber Line (DSL), Fiber to the Home (FTTH), and Point-of-Sale (POTS) are examples of wired communication that can be included in network 104. Furthermore, network 104 can traverse many network topologies and distances. For example, network 104 can include direct connections, Personal Area Networks (PANs), Local Area Networks (LANs), Metropolitan Area Networks (MANs), Wide Area Networks (WANs), or combinations thereof.

[0039] The first device 102 may be coupled to an in-vehicle image sensor 110 and a high-resolution (HD) map data local storage 108. The in-vehicle image sensor 110 is an optical sensor located on the first device 102, configured to monitor, observe, record, or a combination thereof of the environment of the first device 102. The in-vehicle image sensor 110 may be a monocular camera, a stereo camera, a video camera, or a combination thereof. The HD map data local storage 108 provides a non-transitory storage medium to store the sensor data stream 111 captured by the in-vehicle image sensor 110. Feature line extraction 113 may be implemented as software executing on a specific hardware set. Feature line extraction 113 may execute within two libraries of memory coupled to or within the HD map data local storage 108, capable of storing feature lines observed by the in-vehicle image sensor 110. Feature line extraction 113 may use an artificial intelligence model 116 to analyze the sensor data stream 111 to identify adjacent traffic lanes and pedestrian crossings. Once processed, sensor data stream 111 can be serially compared with the contents of HD map data local storage 108 to identify the location of the first device 102. Artificial intelligence model 116 may be a software or hardware module capable of matching lane recognition lines on sensor data stream 111 to identify feature lines extracted from in-vehicle image sensor 110.

[0040] For example, the local storage 108 of HD map data can be implemented in many ways, such as non-volatile storage devices, such as disk drives, solid-state storage devices (SSDs), flash memory cards, or combinations thereof, capable of storing high-resolution (HD) maps of the areas traversed by the first device 102.

[0041] The first device 102 may be coupled to an independent positioning module 115 capable of performing pixel evaluation of sensor data stream 111. The independent positioning module 115 may be implemented in software running on dedicated hardware, all-hardware, or a combination thereof. When global positioning location 130 is unavailable, the independent positioning module 115 may maintain a map-related location 109 through pixel analysis of sensor data stream 111. The independent positioning module 115 may be configured to analyze the captured scene to identify traffic in front of and around the first device 102 and the physical location of the first device 102. The independent positioning module 115 may process the sensor data stream 111, including sampled frames of sensor data stream 111, through feature line extraction 113 to identify feature lines of the scene captured by the in-vehicle image sensor 110, such as lane markings and pedestrian crossings. During the training process, the first device 102 may upload sensor data stream 111 to a second device 106 for further analysis or to generate an artificial intelligence model 116 to improve the detection of the first device 102's position.

[0042] The independent positioning module 115 may include an artificial intelligence model 116, a lane recognition module 118, and a location recognizer module 120 capable of generating a map-related location 109. The map-related location 109 may be the location of the first device 102 generated from the global positioning location 130 or the independent positioning module 115 when the global positioning location 130 is unavailable.

[0043] The location identifier module 120 may be a software or hardware module capable of determining a map-related location 109 based on packets of sampled frames passed to the artificial intelligence model 116, and may be stored in the first device 102 or the second device 106. The location identifier module 120 may receive input from the lane recognition module 118, which provides the recognition of lane lines and pedestrian crossings represented in the sensor data stream 111 and identified by the artificial intelligence model 116. The sensor data stream 111 can be analyzed by submitting a scanned data portion of the sensor data stream 111 to the artificial intelligence model 116. It should be understood that other portions of the sensor data stream 111, including time, real-world location, and external parameters of the in-vehicle image sensor 110, may be stored in the first device 102 or the second device 106 for subsequent operations.

[0044] Lane recognition module 118 may be a software or hardware module capable of selecting samples of frames presented by sensor data stream 111. Lane recognition module 118 may include a set of parameters for identifying how many and which sampled frames are presented to artificial intelligence model 116. The results of the analysis by lane recognition module 118 may be stored in the storage circuitry of first device 102 or second device 106. To understand, location recognizer module 120 may converge the output of artificial intelligence model 116 from sensor data stream 111 with a representation of a high-resolution (HD) neighborhood map to identify map-related locations 109.

[0045] The position recognition module 120 can output a map-related position 109 for display on the first device 102. The map-related position 109 can obtain an accurate and stable position. The position fusion module is used to fuse the map-related position and the IMU estimated position to provide sufficient accuracy to provide the vehicle with control commands to increase speed, decrease speed, change lanes, change position within the lane, or enter a parking space.

[0046] HD map data local storage 108 may reside in the first device 102 and be coupled to the in-vehicle image sensor 110 to store sensor data stream 111 and adjustments to the artificial intelligence model 116 returned from the second device 106 during training or when updates are requested. The first device 102 may assemble frames from the in-vehicle image sensor 110 to generate sensor data stream 111 for analysis. Sensor data stream 111 may provide information captured and recorded by the in-vehicle image sensor 110 in the HD map data local storage 108. During the training process, the first device 102 may transmit a request for artificial intelligence (AI) model update 121 to the second device 106 via network 104. AI model update 121 may be performed when a global positioning location is available, allowing map-related locations 109 to be correlated with locations from the global positioning system. The second device 106 may improve the AI ​​model 116 to calculate map-related locations 109 more accurately. AI model update 121 may be sent to the first device 102 for inclusion during navigation.

[0047] The navigation system 100 can be operated by a user 112. The user 112 may include a person or entity that accesses or utilizes the navigation system 100 or devices therein. For example, the user 112 may include a person who owns or operates the first device 102, a service, or a combination thereof. Furthermore, for example, the user 112 may access or utilize the second device 106 through the first device 102, the service, or a combination thereof.

[0048] The navigation system 100 can also process direct user input 114 from user 112. Direct user input 114 may include requests for navigation assistance, location of points of interest, parking assistance, restaurant assistance, accommodation assistance, location of gas stations, event booking, or a combination thereof. Direct user input 114 may be provided by or from user 112 to the first device 102 or directly on the first device 102. Direct user input 114 may include input or stimuli directly applicable to or related to the corresponding software, application, feature, or combination thereof.

[0049] The navigation system 100 can implement one or more embodiments without direct user input 114. The navigation system 100 can also implement one or more embodiments using direct user input 114, which is independent of the direct user input 114. Direct user input 114 may include prompts from user 112 such as speed increase, speed decrease, change of position within the lane, or lane change.

[0050] The second device 106 may periodically or when the first device 102 requests services for route planning or identification of points of interest along the current road. The second device 106 may distribute the artificial intelligence model updates 121 during training and as requested by the first device 102.

[0051] The second device 106 can improve the AI ​​model update 121 and generate improvements to the AI ​​model 116 for use by the first device 102 during training. As an example, the second device 106 can apply the AI ​​model update 121 to the map verification manager 122. The map verification manager 122 can manipulate the AI ​​model update 121 to verify the background map database 124, the real-time location model 126, and the high-resolution map convergence model 128.

[0052] Background map database 124 may include a graphical display of roads, highways, and intersections in a given area. Real-time location model 126 may apply AI model updates 121 requested by first device 102 to background map database 124 and samples from sensor data stream 111 to verify the accuracy of navigation system 100. High-resolution map convergence model 128 may verify that AI model updates 121 can quickly and accurately merge samples from sensor data stream 111 with data from background map database 124 to identify map-related locations 109. It should be understood that global positioning location 130 is used to verify the accuracy of map-related locations 109; it is not used to calculate map-related locations 109. Global positioning location 130 is provided by Global Positioning System 132. Global positioning location 130 is susceptible to obstruction by obstacles such as canyons, mountains, skyscrapers, tunnels, or indoor parking structures.

[0053] Navigation system 100 has been found to reliably identify map-related locations 109 to provide real-time updates of the actual location of the first device 102 when global positioning location 130 is unavailable. Map-related locations 109 can be normalized over fixed time periods to generate artificial intelligence model updates 121. By sending the artificial intelligence model updates 121 from the second device 106, communication can be distributed to other users of navigation system 100 for route planning, traffic or accident warnings, lane selection warnings, construction warnings, etc. Navigation system 100 can improve the safety of the first device 102 by providing real-time traffic updates, lane suggestions, alternative routes, or combinations thereof. When global positioning location 130 is blocked, navigation system 100 can seamlessly maintain accurate positioning information through map-related locations 109.

[0054] Now for reference Figure 2 The document shows the target Figure 1 The illustration shows an example top plan view of the vehicle 201 of the navigation system 100. The navigation system 100 may include or interact with a first device 102. By way of example, the vehicle 201 may be the first device 102. For this discussion, the first device 102 is considered to be the vehicle 201.

[0055] The first device 102 may be an object or machine for transporting people or goods, capable of automatically manipulating or operating the object or machine. The first device 102 may include components that can be... Figure 1 User 112 accesses a vehicle for control, manipulation, operation, or a combination thereof. For example, the first device 102 may include a car, truck, trolley, drone, or a combination thereof.

[0056] It can also control or manipulate the first device 102 without Figure 1 The direct user input 114 corresponds to manipulation or movement. For example, the first device 102 may include an autonomous vehicle or a vehicle with automatic manipulation features, such as intelligent cruise control or preventive braking. The first device 102 may include intelligent cruise control features that enable setting and adjusting the speed of the first device 102 without the direct user input 114. Furthermore, for example, the first device 102 may be controlled or manipulated by the navigation system 100, including controlling or setting cruise speed, lane position, or other physical manipulation or movement of the first device 102.

[0057] The navigation system 100 can also utilize map-related locations 109 from one or more vehicles or devices. The artificial intelligence model update 121 can include information about the recognition of lane markings observed by the first device 102. Figure 1Enhancement of the artificial intelligence model 116 for processing the image sensor 110 within the vehicle by the first device 102 Figure 1 Sensor data stream 111.

[0058] The first device 102 or other vehicle coupled to the navigation system 100 may include devices, circuitry, one or more specific sensors such as environmental sensor 210, or combinations thereof, for providing auxiliary or additional information to a user 112 who controls, manipulates, or operates the first device 102. The first device 102 or any other vehicle may include vehicle communication circuitry 204, vehicle control circuitry 206, vehicle storage circuitry 208, other interfaces, or combinations thereof.

[0059] The carrier storage circuit 208 may include functional units or circuits that constitute the corresponding first device 102 and are configured to store and recall information. The carrier storage circuit 208 may be volatile memory, non-volatile memory, internal memory, external memory, or a combination thereof. For example, the carrier storage circuit 208 may be non-volatile memory such as non-volatile random access memory (NVRAM), flash memory, or disk storage, or volatile memory such as static random access memory (SRAM).

[0060] The vehicle storage circuit 208 can store vehicle software, such as input information, information from sensors, processing results, information predetermined or preloaded by the navigation system 100 or the vehicle manufacturer, or other related data such as combinations thereof.

[0061] The vehicle control circuit 206 may include functional units or circuits that constitute the first device 102 and are configured to execute or carry out instructions. The vehicle control circuit 206 may execute or implement vehicle software to provide intelligence for the vehicle 201, the navigation system 100, or a combination thereof.

[0062] The vehicle control circuit 206 can be implemented in many different ways. For example, the vehicle control circuit 206 can be a processor, an application-specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. As a more specific example, the vehicle control circuit 206 may include an engine control unit, one or more central processing units, or a combination thereof.

[0063] The vehicle communication circuit 204 may include functional units or circuits that comprise the vehicle 201, such as the first device 102, another vehicle, or a combination thereof. The vehicle communication circuit 204 may be configured to enable external communication to and from the vehicle 201. For example, the vehicle communication circuit 204 may allow the first device 102 to communicate with… Figure 1 The second device 106 communicates.

[0064] The vehicle communication circuit 204 can also be used as a communication hub, which allows the vehicle 201 to be used as Figure 1 The vehicle communication circuit 204 is part of network 104 and is not limited to endpoint or terminating circuitry to network 104. The vehicle communication circuit 204 may include active and passive components, such as microelectronic devices or antennas, for interacting with network 104. For example, the vehicle communication circuit 204 may include a modem, transmitter, receiver, port, connector, or a combination thereof for wired communication, wireless communication, or a combination thereof.

[0065] Vehicle communication circuit 204 can be coupled to network 104 to directly send or receive information between vehicle communication circuit 204 and a second device 106, which serves as a communication endpoint, such as for direct line-of-sight communication or peer-to-peer communication. Vehicle communication circuit 204 can also be coupled to network 104 to send or receive information through another intermediate device or server between the communication endpoints.

[0066] The first device 102 or other vehicle 201 may also include various interfaces. The first device 102 may include one or more interfaces for interaction or internal communication between functional units or circuits of the first device 102. For example, the first device 102 may include one or more interfaces for vehicle storage circuit 208, vehicle control circuit 206 or combinations thereof, such as drivers, firmware, wired connections or buses, protocols or combinations thereof.

[0067] The first device 102 or other vehicle may also include one or more interfaces for interacting with occupants, operators or drivers, passengers or combinations thereof relative to the vehicle 201. For example, the first device 102 or other vehicle may include a user interface 212, which includes input or output devices or circuitry, such as a screen or touchscreen, speaker, microphone, keyboard or other input devices, dashboard or combinations thereof.

[0068] The first device 102 may also include one or more interfaces and switches or actuators for physically controlling movable parts of the first device 102. For example, the first device 102 may include one or more interfaces and control mechanisms to physically perform and control the manipulation of the first device 102, such as for autonomous driving, smart cruise control, or maneuvering features.

[0069] The functional units or circuits in the first device 102 can operate individually and independently of other functional units or circuits. The first device 102 can operate individually and independently of the network 104, the second device 106, other devices or vehicles, or combinations thereof.

[0070] The aforementioned functional units or circuits can be implemented in hardware. For example, one or more of the functional units or circuits can be implemented using gates, circuit systems, processors, computers, integrated circuits, integrated circuit cores, pressure sensors, inertial sensors, microelectromechanical systems (MEMS), passive devices, physical non-transient storage media containing instructions for performing software functions, or a combination thereof.

[0071] Each of the environmental sensors 210 is a device or circuit for detecting or identifying the environment of the first device 102. The environmental sensors 210 can detect, identify, determine, or combine thereof the state, environment, or movement of the vehicle 201. The environmental sensors 210 can detect, identify, determine, or combine thereof the environment inside the vehicle 201, the environment outside the vehicle 201 and its surroundings, or combinations thereof. The environmental sensors 210 can be implemented in the first device 102.

[0072] For example, environmental sensor 210 may include user interface 212, in-vehicle image sensor 110, in-vehicle position sensor 214, radar sensor 216, Global Positioning System (GPS) position sensor 218, or combinations thereof. User interface 212 may include a projector, video screen, touchscreen, speaker, or any combination thereof. User interface 212 may display... Figure 1 Map-related locations 109, planned routes, lane suggestions, speed warnings, vehicle system alerts, and combinations thereof.

[0073] The in-vehicle image sensor 110 may include sensors for detecting or determining visual information representing the exterior and surroundings of the vehicle 201. The in-vehicle image sensor 110 may include a camera attached to or integrated with the vehicle 201 or device. For example, the in-vehicle image sensor 110 may include a camera, such as a forward-facing camera, a video camera, a rear-view or reversing camera, a side-view or blind-spot camera, or a combination thereof. Furthermore, for example, the in-vehicle image sensor 110 may include an infrared sensor, a night vision camera, or a night vision sensor.

[0074] The in-vehicle image sensor 110 may also include a camera on the first device 102 or another user device of the user 112 that is connected to and interacts with the first device 102, such as the vehicle. The in-vehicle image sensor 110 may also include a cabin camera for detecting or determining visual information within the vehicle or the vehicle's compartment.

[0075] The in-vehicle position sensor 214 can be a combination of software executed on specific hardware to implement an inertial measurement unit (IMU). The in-vehicle position sensor 214 can monitor... Figure 1The vehicle 201 maintains a copy of its geographic location 226 by using the GPS location sensor 218 as the output of the GPS location sensor 218. The geographic location 226 indicates the vehicle 201's position on Earth. If the GPS location sensor 218 is blocked or obstructed by an obstacle 202 and the GPS location 130 is unavailable, the vehicle's in-vehicle location sensor 214 can update the map-related location 109 based on the geographic location 226 to maintain the vehicle 201's position awareness.

[0076] The in-vehicle position sensor 214 may include a gyroscope, accelerometer, and magnetometer for monitoring the movement of the vehicle 201 when an obstacle 202 prevents the operation of the GPS position sensor 218. The in-vehicle position sensor 214 can only maintain the precise position of the vehicle 201 for a short period and must be supported by other devices. The obstacle 202 is shown as surrounding the vehicle 201, but it should be understood that the obstacle 202 may cover an area of ​​several square miles and may simultaneously affect multiple vehicles within the vehicle 201. As a concrete example, the obstacle 202 could be created by a skyscraper or indoor parking garage in downtown New York City.

[0077] Radar sensor 216 may include an object detection system, device, or circuitry. Radar sensor 216 can determine or identify the presence of an object or target, such as an obstacle or another vehicle, outside the corresponding device or vehicle, the relative position or distance between the object or target and the corresponding device or vehicle, or a combination thereof.

[0078] Radar sensor 216 can use radio waves to determine or identify the presence of an object or target, its relative position or distance relative to the first device 102 or other corresponding devices or vehicles, or a combination thereof. For example, radar sensor 216 may include a proximity sensor or warning system, such as for areas geographically or physically in front of, behind, adjacent to, or to the side of the first device 102, or a combination thereof.

[0079] GPS position sensor 218 may be a sensor used to identify or calculate the geographic location 226 of vehicle 201 or equipment, determine the movement or speed of vehicle 201 or equipment, or a combination thereof. GPS position sensor 218 may include an accelerometer, speedometer, Global Positioning System (GPS) receiver or device, gyroscope or compass, or a combination thereof. First device 102 may include an environmental sensor 210, different from or attached to GPS position sensor 218. GPS position sensor 218 provides a global positioning location 130 and may be susceptible to obstacles 202, and requires a clear line of sight to multiple satellites (not shown) or cell towers for proper operation.

[0080] The navigation system 100 may use one or more in-vehicle image sensors 110 corresponding to one or more devices, one or more vehicles, or combinations thereof to generate a map-related location 109 describing or representing information about the environment surrounding the corresponding device or vehicle. The map-related location 109 may also be processed by vehicle control circuitry 206, stored in vehicle storage circuitry 208, and transmitted to another device or vehicle, or combinations thereof, via vehicle communication circuitry 204. The in-vehicle image sensors 110 may support in-vehicle position sensors 214 to maintain an accurate version of the map-related location 109.

[0081] As a more specific example, vehicle communication circuit 204, vehicle control circuit 206, vehicle storage circuit 208, in-vehicle image sensor 110, one or more interfaces, or combinations thereof, may be included in or constitute the first device 102.

[0082] The navigation system 100 can use map-related locations 109 from devices, vehicles, or combinations thereof to dynamically determine and map traffic and road conditions in a geographic area, as well as vehicles, pedestrians, objects, or combinations thereof within the geographic area. As a more specific example, the navigation system 100 can use map-related locations 109 to dynamically control vehicle 201. The navigation system 100 can also use map-related locations 109 to control the movement of the first device 102 at the lane-level granularity.

[0083] The navigation system 100 can provide vehicle movement control 228 as a suggestion for the user 112 to manipulate or operate the first device 102. Details regarding the utilization and processing of map-related locations 109 are discussed below.

[0084] The navigation system 100 can process and generate vehicle movement control 228 for controlling or manipulating the first device 102. Vehicle movement control 228 is an instruction, signal, process, method, mechanism, or combination thereof that guides or controls the physical movement or travel of the first device 102.

[0085] The navigation system 100 can transmit AI model updates 121 from the second device 106 to the first device 102 at fixed intervals, such as a one-minute interval during training. For illustrative purposes, the navigation system 100 supports the second device 106 in communicating AI model updates 121 to the first device 102 and other vehicles nearby or planning to enter an area reported by the first device 102 as including obstacle 202.

[0086] Continuing this example, navigation system 100 can use map-related locations 109 generated or provided from first device 102 without user input 114. Navigation system 100 can utilize map-related locations 109 to provide information, assist in manipulation, control manipulation, or a combination thereof to first device 102.

[0087] Continuing this example, navigation system 100 can transmit artificial intelligence model updates 121 from second device 106 to other devices or vehicles, or directly to other devices or vehicles, as is the case for peer-to-peer communication systems. Navigation system 100 can transmit artificial intelligence model updates 121 to inform other devices or vehicles of the location or status of first device 102 itself, and of other vehicles or combinations thereof monitored and identified around first device 102.

[0088] As a more specific example, navigation system 100 can use map-related locations 109 to generate vehicle movement controls 228, such as for steering, braking, setting or adjusting speed, auxiliary controls, or combinations thereof.

[0089] Now for reference Figure 3 The illustration shows an exemplary object detection map of an environmental image 301 processed by the navigation system 100. The example of the environmental image 301 describes bounding boxes 302 around each object in the scan area 303, including parked vehicles 304, motorcycles 306, signs 308, image lines 310 marked L1, L2, and L3, and the right curb 312 marked L4.

[0090] The right-side curb bounding box 314 can be defined by a box center 316, a box length 318, and a box width 320. Each object detected in the scan area 303 can be surrounded by a bounding box 302. It should be understood that the scan area 303 is assumed to be horizontal and has a constant value in the vertical (Z) plane. The box center 316 is defined by X coordinate 322 and Y coordinate 324. The box length 318 is defined as the measurement of the bounding box 302 when measured through the Y coordinate 324, while keeping the X coordinate 322 constant. The box width 320 is defined as the measurement of the bounding box 302 when measured through the X coordinate 322, while keeping the Y coordinate 324 constant.

[0091] By removing detected objects, including parked vehicles 304 and motorcycles 306, the number of objects detected in scan area 303 can be reduced, leaving specific lines. Signage 308 can indicate the availability of right-turn lanes or bike lanes.

[0092] Image lines 310 labeled L1 and L2 can be defined by their image coordinates 311 labeled P1, P2, P3, and P4. Image coordinates P1 326 and P2 328 can define image line 310 labeled L1. Image coordinates P3 330 and P4 332 can define image line 310 labeled L2. It should be understood that image coordinates P1 326, P2 328, P3 330, and P4 332 are each defined by the X-coordinate 322 and Y-coordinate 324 of each image coordinate 311.

[0093] Image line 310, labeled L1, can be transformed based on the position of the in-vehicle image sensor 110, as provided by the first device 102, and the in-vehicle image sensor calibration 336 of the in-vehicle image sensor 110, through geometric transformations of image coordinates P1 326 and image coordinates P2 328 in real-world coordinates. The in-vehicle image sensor calibration 336 can provide extrinsic parameters 338 of the in-vehicle image sensor 110 and may include lens distortion, lens rotation, distance translation due to lens curvature, and a predefined distance 340 within the capabilities of the in-vehicle image sensor 110.

[0094] Image line 310, labeled L2, can be transformed based on the position of in-vehicle image sensor 110 provided by first device 102 and external parameters 338 of in-vehicle image sensor 110 through geometric transformation of image coordinates P4 332 and image coordinates P3 330 in world coordinates 334.

[0095] It has been found that defining the bounding box 302 and image coordinates 311 provides sufficient information to calculate the image coordinates 311 in world coordinates 334 using geometric transformations. Accurate depiction of the relative position of image line 310 with respect to the first device 102 simplifies the movement of the first device 102 along the planned route. Figure 2 The generation of vehicle movement control 228.

[0096] Now for reference Figure 4 , which shows Figure 1 Navigation system 100 Figure 1 An exemplary functional block diagram 401 of the independent positioning mechanism 115 is shown. The exemplary functional block diagram 401 of the independent positioning mechanism 115 depicts an in-vehicle image sensor 110 coupled to an environment recognition image module 402. In the environment recognition image module 402, the in-vehicle image sensor 110 can... Figure 3 The scan area 303 is captured as a photographic image. The scan area 303 may contain... Figure 3 Image lines 310 include lane lines, pedestrian crossings, vehicle lanes, and dedicated lanes.

[0097] The environmental image recognition module 402 can Figure 1 The scan data stream 111 is transmitted from the image sensor 110 inside the vehicle to the environmental image 404 of the AI ​​model module. This is achieved by submitting the environmental image to... Figure 1 The artificial intelligence model 116 identifies the image lines 310 captured from the scan area 303, and the environmental image 404 of the AI ​​model module can process the scan data stream 111. The artificial intelligence model 116 can ignore vehicles and any obstacles shown in the scan area 303 and extract only lane lines and crosswalks in the form of image lines 310.

[0098] Meanwhile, the position sensor 214 inside the vehicle can monitor the position sensor 214 inside the vehicle. Figure 1 Satellite 132 provides Figure 1 The global positioning location is 130. When Figure 2 When an obstacle 202 prevents satellite 132 from updating its global positioning position 130, the position sensor 214 inside the vehicle can monitor... Figure 2 The movement of vehicle 201 is used to estimate changes in position.

[0099] The vehicle-mounted position sensor 214 can be coupled to the geographic location 226. The geographic location 226 can be... Figure 2 The predicted location is provided in the form of geolocation 226. As the vehicle 201 moves along its route, the in-vehicle position sensor 214 monitors this process. The accuracy of the geolocation may decrease over time, therefore the value must be updated periodically. The geolocation can be passed to the environmental image 404 of the AI ​​model module to assist the artificial intelligence model 116 in processing the image line 310.

[0100] Geographic location 226 is coupled to neighboring map line extraction module 406 for processing. Neighboring map line extraction module 406 can select smaller portions of the neighboring map, stored in a high-resolution map providing the area surrounding geographic location 226. Figure 1 The HD map is stored locally in 108. Map lines corresponding to the area surrounding geographic location 226 can be extracted from the neighboring map. Since the in-vehicle position sensor 214 provides at most an accurate estimate of geographic location 226, the size of a smaller portion of the neighboring map can become larger over time. The neighboring map line extraction module 406 can select map lines present in the neighboring map to verify the image line 310 from the optical path 408 against the map lines from the mechanical path 410.

[0101] Optical path 408 can utilize scan data stream 111 as an information source to generate the position of vehicle 201 based on image lines 310 extracted by artificial intelligence model 116. Conversely, mechanical path 410 relies on in-vehicle position sensor 214 and geographic location 226 to monitor changes in the position of vehicle 201. The convergence of optical path 408 and mechanical path 410 is discussed below.

[0102] Based on the environmental image 404 from the AI ​​model module and the neighboring map line extraction module 406, two sets of image lines 310 and map lines are submitted to the horizontal image line recognition 412 and the vertical image line recognition 414. The horizontal image line recognition 412 can identify matching lines from the map lines and image lines 310 in the horizontal direction. The vertical image line recognition 414 can identify matching lines from the map lines and image lines 310 in the vertical direction.

[0103] Matching groups from identified horizontal image lines 412 and identified vertical image lines 414 are submitted to alignment feature line module 416, which can align with map lines and image lines 310 from HD map local storage 108. Alignment feature line module 416 can then pass the matching groups to verification location matching module 418. Verification location matching module 418 can assign physical coordinates based on the map lines from HD map local storage 108, providing known locations with image lines, to provide updated locations 419 to geographic location 226, thereby improving the accuracy of geographic location 226.

[0104] The verification location matching module 418 can also provide the updated location 419 to the location fusion module 420. The map-related location 109 can obtain an accurate and stable position. The location fusion module 420 is used to fuse the map-related location 109 and the updated location 419, such as the IMU estimated location, to provide sufficient accuracy to provide maneuvering commands to the vehicle 201 to increase speed, decrease speed, change lanes, change position within a lane, or enter a parking space. The location fusion module 420 can provide... Figure 1 The map location 109 is used to display to Figure 1 User 112.

[0105] It has been found that the independent positioning mechanism 115 of the navigation system 100 can provide accurate and timely updates to the geographic location 226 of the vehicle 201. By comparing the results of the optical path 408 with the results of the mechanical path 410, a feedback signal in the form of an updated position 419 can reduce the deviations generated in the mechanical path 410 and maintain the accuracy of the geographic location 226. The updated position signal 419 can be verified during a training session. Figure 1The accuracy of the AI ​​model update 121. The second device 106 can forward the AI ​​model update 121 to other vehicles communicating with the navigation system 100. This communication can provide... Figure 2 Obstacle 202 is a safer and more efficient path for vehicle 201 to enter.

[0106] Now for reference Figure 5 An exemplary block diagram of a navigation system 100 in an embodiment is shown. The navigation system 100 may include a first device 102, a network 104, and a second device 106. The first device 102 may transmit information to the second device 106 via the network 104 in a first device transmission 508. The second device 106 may transmit information to the second device 106 via the network 104 in a second device transmission 510. Figure 2 The vehicle 201 or the first device 102.

[0107] For illustrative purposes, navigation system 100 is shown as having a first device 102 as a client device, although it is understood that navigation system 100 may also include a first device 102 as a different type of device. For example, the first device 102 may be a vehicle 201 that includes a first display interface 530 coupled to user interface 212.

[0108] Furthermore, for illustrative purposes, navigation system 100 is shown as having a second device 106 as a server, although it should be understood that navigation system 100 may also include a second device 106 as a different type of device. For example, the second device 106 may be a client device. By way of example, navigation system 100 may be fully implemented on the first device 102. The second device 106 may provide Figure 1 Artificial intelligence model 116 Figure 1 The artificial intelligence model is updated through training and enhancement in the form of 121.

[0109] Furthermore, for illustrative purposes, the navigation system 100 is shown to include interaction between a first device 102 and a second device 106. However, it should be understood that the first device 102 may be part or all of an autonomous vehicle, an intelligent vehicle, or a combination thereof. Similarly, the second device 106 may similarly interact with the first device 102, which represents an autonomous vehicle, an intelligent vehicle, or a combination thereof.

[0110] For the sake of brevity in this embodiment of the invention, the first device 102 will be described as a client device, carrier 201, and the second device 106 will be described as a server device. Embodiments of the invention are not limited to this choice of device type. This choice is merely an example of an embodiment of the invention.

[0111] The first device 102 may include a first control circuit 512, a first storage circuit 514, a first communication circuit 516, a first interface circuit 518, and a first positioning circuit 520. The first control circuit 512 may include a first control interface 522. The first control circuit 512 may execute first software 526 to provide intelligence for the navigation system 100.

[0112] The first control circuit 512 can be implemented in many different ways. For example, the first control circuit 512 can be a processor, an application-specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof. The first control interface 522 can be used for communication between the first control circuit 512 and other functional units or circuits in the first device 102. The first control interface 522 can also be used for communication outside the first device 102.

[0113] The first control interface 522 can receive information from other functional units / circuits or from external sources, or can transmit information to other functional units / circuits or external destinations. External sources and external destinations refer to sources and destinations outside the first device 102.

[0114] The first control interface 522 can be implemented in different ways and may include different implementation schemes, depending on which functional units / circuits or external units / circuits are engaged with the first control interface 522. For example, the first control interface 522 can be implemented using a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), an optical circuit system, a waveguide, a wireless circuit system, a wired circuit system, or a combination thereof.

[0115] The first storage circuit 514 can store the first software 526 and the HD map local storage 108. The first storage circuit 514 can also store related information, such as data representing an input image, data representing a previously presented image, sound files, or combinations thereof. The HD map local storage 108 can be a high-resolution map containing detailed map fragments, including… Figure 3 The world coordinates of the area being traveled are 334. The HD map local storage 108 may include a neighboring map 529 representing a smaller segment of the HD map local storage 108 centered on geographic location 226. The neighboring map 529 may be part of the HD map local storage 108 containing all known lane details (including world coordinates 334 around geographic location 226).

[0116] The first storage circuit 514 may be a volatile memory, a non-volatile memory, an internal memory, an external memory, or a combination thereof. For example, the first storage circuit 514 may be a non-volatile memory such as non-volatile random access memory (NVRAM), flash memory, disk memory, or a volatile memory such as static random access memory (SRAM).

[0117] The first storage circuit 514 may include a first storage interface 524. The first storage interface 524 may be used for communication between the first storage circuit 514 and other functional units or circuits in the first device 102, such as… Figure 1 HD map data is stored locally 108. The first storage interface 524 can also be used for external communication of the first device 102.

[0118] The first storage interface 524 can receive information from other functional units / circuits or from external sources, or it can send information to other functional units / circuits or external destinations. External sources and external destinations refer to sources and destinations outside the first device 102. The first storage interface 524 can receive input from the independent positioning module 115 and source data to the independent positioning module 115. The independent positioning module 115 can send the geographic location 226 to the first control circuit 512 through the first storage interface 524.

[0119] The first storage interface 524 may include different implementations depending on which functional units / circuits or external units / circuits are coupled to the first storage circuit 514. The first storage interface 524 may be implemented using techniques and skills similar to those used in the implementation of the first control interface 522.

[0120] The first communication circuit 516 enables external communication to and from the first device 102. For example, the first communication circuit 516 can allow the first device 102 to communicate with the second device 106 and the network 104.

[0121] The first communication circuit 516 can also be used as a communication hub that allows the first device 102 to be used as part of the network 104 and is not limited to being an endpoint or terminal circuit of the network 104. The first communication circuit 516 may include active and passive components, such as microelectronic devices or antennas, for interacting with the network 104.

[0122] The first communication circuit 516 may include a first communication interface 528. The first communication interface 528 can be used for communication between the first communication circuit 516 and other functional units or circuits in the first device 102. The first communication interface 528 can receive information from the second device 106 for allocation to other functional units / circuits, or can transmit information to other functional units or circuits.

[0123] The first communication interface 528 may include different implementations depending on which functional units or circuits are interacting with the first communication circuit 516. The first communication interface 528 may be implemented using techniques and methods similar to those used in the implementation of the first control interface 522.

[0124] The first interface circuit 518 allows... Figure 1 User 112 engages and interacts with first device 102. First interface circuitry 518 may include input and output devices. Examples of input devices for first interface circuitry 518 may include a keypad, touchpad, soft keys, keyboard, microphone, infrared sensor for receiving remote signals, in-vehicle image sensor 110, or any combination thereof to provide data and transmit input. As an example, in-vehicle image sensor 110 may be connected to first interface circuitry 518 via a wired or wireless connection to deliver sensor data stream 111 to first control circuitry 512. First interface circuitry 518 may pass input from in-vehicle image sensor 110 to first control circuitry 512 for processing and storage. During training of independent positioning module 115, first communication interface 528 may transmit input from in-vehicle image sensor 110, the position of in-vehicle image sensor 110, and external parameters of in-vehicle image sensor 110 to second device 106 to enhance... Figure 1 Artificial intelligence model 116 Figure 1 Lane recognition module 118 and Figure 1 The accuracy and reliability of the location identifier module 120.

[0125] The first interface circuit 518 may include a first display interface 530. The first display interface 530 may include an output device. The first display interface 530 may couple to a user interface 212 including a projector, video screen, touchscreen, speaker, microphone, keyboard, and combinations thereof. The user interface 212 may output to... Figure 1 User 112 displayed map-related location 109.

[0126] The first control circuit 512 can operate the first interface circuit 518 to display information generated by the navigation system 100 and receive input from the user 112. The first control circuit 512 can also execute first software 526 for other functions of the navigation system 100, including receiving position information from the first positioning circuit 520. Figure 2 When an obstacle 202 obstructs the function of the positioning circuit 520, the first control circuit can retrieve the geographical location 226 from the independent positioning module 115. The first control circuit 512 can also execute first software 526 for interacting with the network 104 via the first communication circuit 516. The first control unit 512 can operate... Figure 1 The independent positioning mechanism 115.

[0127] The first control circuit 512 can operate the first interface circuit 518 to collect data from the in-vehicle image sensor 110. The first control circuit 512 can also receive location information from the first positioning circuit 520. The first control circuit 512 can operate the independent positioning module 115 to deliver map-related locations 109 for display on the user interface 212 and generate... Figure 2 The vehicle movement control 228 provides control guidance commands and vehicle movement control 228 to enable autonomous or assisted driving for the first device 102. Vehicle movement control 228 may include speed increase, speed decrease, lane change suggestion, lane boundary warning, and traffic avoidance alert. This can be based on... Figure 2 The vehicle movement control 228 is generated based on feedback from the environmental sensor 210 and the map-related location 109.

[0128] The first positioning circuit 520 can be based on and Figure 1 The first positioning circuit 520 generates location information through interaction with satellite 132, and can be implemented in many ways. For example, the first positioning circuit 520 can be used as at least a part of a global positioning system, an inertial navigation system, a cellular tower positioning system, a gyroscope, or any combination thereof. Furthermore, for example, the first positioning circuit 520 can utilize components such as an accelerometer, a gyroscope, or a global positioning system (GPS) receiver.

[0129] The first positioning circuit 520 may include a first positioning interface 532. The first positioning interface 532 may be used for communication between the first positioning circuit 520 and other functional units or circuits (including environmental sensor 210) in the first device 102.

[0130] The first positioning interface 532 can receive information from other functional units / circuits or from external sources, or can transmit information to other functional units / circuits or external destinations. External sources and external destinations refer to sources and destinations outside the first device 102. The first positioning interface 532 can receive information from other functional units / circuits or external destinations. Figure 1 GPS 132 receiver Figure 1 The global positioning location is 130.

[0131] The first positioning interface 532 may include different implementations depending on which functional units / circuits or external units / circuits are engaged with the first positioning circuit 520. The first positioning interface 532 may be implemented using techniques and skills similar to those used in the implementation of the first control circuit 512.

[0132] The second device 106 can be optimized for implementing embodiments of the invention in a multi-device embodiment having the first device 102. The second device 106 can provide additional or higher performance processing capabilities compared to the first device 102. The second device 106 may include a second control circuit 534, a second communication circuit 536, a second user interface 538, and a second storage circuit 546.

[0133] The second user interface 538 allows an operator (not shown) to engage and interact with the second device 106. The second user interface 538 may include input devices and output devices. Examples of input devices for the second user interface 538 may include a keypad, touchpad, soft keys, keyboard, microphone, or any combination thereof to provide data and transmission input. Examples of output devices for the second user interface 538 may include a second display interface 540. The second display interface 540 may include a monitor, projector, video screen, speaker, or any combination thereof.

[0134] During the training process, the second control circuit 534 can transmit the artificial intelligence model update 121 via the second communication circuit 536. The second control circuit 536 can verify that the content of the artificial intelligence model update 121 indeed marks the geographic location 226 calculated for the sensor data stream 111 sent from the first device. Once the artificial intelligence model update 121 has been verified with the background map database 124, the second control circuit 534 can transmit the artificial intelligence model update 121 to the map verification manager 122 via the second storage circuit 546 for processing and further allocation.

[0135] The second control circuit 534 can execute the second software 542 to provide intelligence to the second device 106 of the navigation system 100. The second software 542 can operate in conjunction with the first software 526. Compared with the first control circuit 512, the second control circuit 534 can provide additional performance.

[0136] The second control circuit 534 can operate the second user interface 538 to display information. The second control circuit 534 can also execute the second software 542 for other functions of the navigation system 100, including operating the second communication circuit 536 to communicate with the first device 102 via the network 104.

[0137] The second control circuit 534 can be implemented in many different ways. For example, the second control circuit 534 can be a processor, an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), a digital signal processor (DSP), or a combination thereof.

[0138] The second control circuit 534 may include a second control interface 544. The second control interface 544 can be used for communication between the second control circuit 534 and other functional units or circuits in the second device 106. The second control interface 544 can also be used for communication with the outside of the second device 106.

[0139] The second control interface 544 can receive information from other functional units / circuits or from external sources, or it can send information to other functional units / circuits or external destinations. External sources and external destinations refer to sources and destinations outside of this second device 106.

[0140] The second control interface 544 can be implemented in different ways and can include different implementations depending on which functional units / circuits or external units / circuits are engaged with the second control interface 544. For example, the second control interface 544 can be implemented using a pressure sensor, an inertial sensor, a microelectromechanical system (MEMS), an optical circuit system, a waveguide, a wireless circuit system, a wired circuit system, or a combination thereof.

[0141] The second storage circuit 546 can store the second software 542. The second storage circuit 546 can also store information such as data representing an input image, data representing a previously presented image, sound files, or combinations thereof. The second storage circuit 546 can be sized to provide additional storage capacity to supplement the first storage circuit 514. During the training process, the second storage circuit 546 can receive sensor data stream 111 and geographic location 226 for generating an AI model update 121 for the first device 102. The second storage circuit 546 can transmit the AI ​​model update 121 to the first device 102 in real time. The AI ​​model update 121 can then be transmitted to the vehicle 201 via the second communication circuit 536.

[0142] For illustrative purposes, the second storage circuit 546 is shown as a single element, although it should be understood that the second storage circuit 546 may also be a distribution of storage elements. Furthermore, for illustrative purposes, the navigation system 100 is shown as having the second storage circuit 546 as a single-level storage system, although it should be understood that the navigation system 100 may also include the second storage circuit 546 in different configurations. For example, the second storage circuit 546 may be formed using different storage technologies that form a memory hierarchy system including different levels of cache, main memory, rotating media, or offline storage.

[0143] The second storage circuit 546 may be a controller for volatile memory, non-volatile memory, internal memory, external memory, or a combination thereof. For example, the second storage circuit 546 may be a controller for non-volatile memory such as non-volatile random access memory (NVRAM), flash memory, disk storage, or volatile memory such as static random access memory (SRAM).

[0144] The second storage interface 548 can receive information from other functional units / circuits or from external sources, or it can send information to other functional units / circuits or external destinations. External sources and external destinations refer to sources and destinations outside the second device 106.

[0145] The second storage interface 548 may include different implementations depending on which functional units / circuits or external units / circuits are coupled to the second storage circuit 546. The second storage interface 548 may be implemented using techniques and methods similar to those used in the implementation of the second control interface 544.

[0146] The second communication circuit 536 enables external communication to and from the second device 106. For example, the second communication circuit 536 can allow the second device 106 to communicate with the first device 102 via network 104. As an example, the second device 106 can provide artificial intelligence model updates 121 to the first device 102.

[0147] The second communication circuit 536 can also be used as a communication hub that allows the second device 106 to be used as part of the network 104 and is not limited to being an endpoint or terminal unit or circuit of the network 104. The second communication circuit 536 may include active and passive components such as microelectronic devices or antennas for interacting with the network 104.

[0148] The second communication circuit 536 may include a second communication interface 550. The second communication interface 550 can be used for communication between the second communication circuit 536 and other functional units or circuits in the second device 106. The second communication interface 550 can receive information from other functional units / circuits, or can send information to other functional units or circuits.

[0149] The second communication interface 550 may include different implementations depending on which functional units or circuits are coupled to the second communication circuit 536. The second communication interface 550 may be implemented using techniques and methods similar to those used in the implementation of the second control interface 544.

[0150] During the training process, the first communication circuit 516 can be coupled to the network 104 to send a request for an AI model update 121 to the second device 106 in a first device transmission 508. The second device 106 can receive information from the first device transmission 508 of the network 104 in a second communication circuit 536.

[0151] The second communication circuit 536 can be coupled to the network 104 to send information to the first device 102, including updates to the independent positioning module 115 and the artificial intelligence model update 121 in the second device transmission 510. The first device 102 can receive information from the second device transmission 510 of the network 104 in the first communication circuit 516. The navigation system 100 can be executed by the first control circuit 512, the second control circuit 534, or a combination thereof. For illustrative purposes, the second device 106 is shown as having partitions including a second user interface 538, a second storage circuit 546, a second control circuit 534, and a second communication circuit 536, although it should be understood that the second device 106 may also include different partitions. For example, the second software 542 may be partitioned differently such that some or all of its functions are in the second control circuit 534 and the second communication circuit 536. In addition, the second device 106 may include other functional units or circuits, which are not shown for clarity. Figure 5 middle.

[0152] The functional units or circuits in the first device 102 can operate individually and independently of other functional units or circuits. The first device 102 can operate individually and independently of the second device 106 and the network 104.

[0153] The functional units or circuits in the second device 106 can operate individually and independently of other functional units or circuits. The second device 106 can operate individually and independently of the first device 102 and the network 104.

[0154] The aforementioned functional units or circuits can be implemented in hardware. For example, one or more of the functional units or circuits can be implemented using gate arrays, application-specific integrated circuits (ASICs), circuit systems, processors, computers, integrated circuits, integrated circuit cores, pressure sensors, inertial sensors, microelectromechanical systems (MEMS), passive devices, physical non-transient storage media containing instructions for performing software functions, some of these, or combinations thereof.

[0155] For illustrative purposes, the navigation system 100 is described by the operation of the first device 102 and the second device 106. It should be understood that the first device 102 and the second device 106 can operate any module and function of the navigation system 100.

[0156] It has been discovered that the second device 106 can provide an AI model update 121 to the first device 102, which provides updates to the independent positioning module 115. As an example, the second control circuit 534 can verify the AI ​​model update 121 by generating a map-related location 109 using scan data stream 111 and geographic location 226, and comparing it with the geographic location 226 provided by the in-vehicle position sensor 214. Once the AI ​​model update 121 has been verified, the second device 106 transmits the AI ​​model update 121 to the first device 102 via network 104. By incorporating the AI ​​model update 121, the first device 102 can complete the training process of the AI ​​model 116 and prepare to handle the loss of global positioning location 130 due to obstacle 202.

[0157] Now for reference Figure 6 An exemplary block diagram 601 of the independent positioning module 115 is shown. The exemplary block diagram 601 of the independent positioning module 115 may include an artificial intelligence model 116, a lane recognizer module 118, and a position recognizer module 120. It should be understood that the independent positioning module 115 and its sub-components may be software executed by a second control circuit 534, a first control circuit 512, or a combination thereof. It should also be understood that the independent positioning module 115 and its sub-components may have specific hardware-assisted logic where it is necessary to accelerate the object recognition process.

[0158] The artificial intelligence model 116 may include a deep learning object detector 602, such as a mask-based convolutional neural network (R-CNN), a fast segmented convolutional neural network (Fast-SCNN), U-Net, a fast region-based convolutional neural network (Fast R-CNN), You Only See Once (YOLO), a single-shot detector (SSD), etc. The deep learning object detector 602 can receive data from... Figure 1 The sensor data stream contains 111 photos, and is processed based on deep learning semantic segmentation and image morphology processing. Figure 2 The data in the scanned region 204. The deep learning object detector 602 can be coupled to the image feature module 604.

[0159] Image feature module 604 can process the output of deep learning object detector 602 to identify objects detected from the initial processing. By focusing on high-probability objects, more robust determinations can be made. Image feature module 604 can also process objects detected with high probability. This process can provide recognition... Figure 3 The highest chance of all objects in the scan area 303.

[0160] The output of image feature module 604 can be further processed by fully connected module (FC) 606, which can define the labeling or classification of objects in scan area 303. By way of example, fully connected module 606 can identify... Figure 3 Parked vehicle 304 Figure 3 Motorcycle 306 Figure 3 Sign 308 Figure 3 Image lines 310 and L4, labeled L1, L2, L3 and L4 Figure 3 The right-hand curb 312. The marking of the identified objects can simplify subsequent processing.

[0161] The output of the image feature module 604 can also be processed by the multilayer perceptron (MLP) module 608.

[0162] The multilayer sensing module (MLP) 608 can be composed of Figure 1 The AI ​​update 121 training, this AI update is by Figure 5 A second control circuit 534 is provided to more effectively identify image lines 310 labeled L1, L2, L3, and L4, as well as the right curb 312. The training process performed by the AI ​​update 121 may involve adjusting the weight functions in the multilayer perceptron (MLP) 608. The MLP 608 may be defined as a neural network whose task is to identify endpoints and lines in the scanned region 303. The output of the MLP 608 may be a bounding box 302 surrounding each of the image lines 310 and the right curb 312. The MLP 608 may also output the endpoints 610 of each image line 310 identified during analysis.

[0163] Lane recognition module 118 can receive endpoints 610, bounding boxes 612, and classifications 614 for each line detected in scan area 303. Endpoints 610 paired with their bounding boxes 612 can be manipulated by endpoint module 616 to be transformed by world coordinate module 620. World coordinate module 620 can utilize data transmitted from first device 102... Figure 2 The world endpoint 624 is determined by the geographic location 226 and the external parameters 338 provided by the in-vehicle sensor calibration 336 of the in-vehicle image sensor 110. Figure 2 The in-vehicle position sensor 214, world endpoint 624 can be defined as the estimated position of each endpoint 610.

[0164] Classification 614 can label objects identified in scan area 303. Objects may include parked vehicles 304, motorcycles 306, and signs 308, and will not be processed by world coordinate module 620. Objects can be removed to simplify the analysis performed by endpoint module 616 and bounding box module 618.

[0165] The bounding box module 618 can receive bounding boxes 612 sequentially, provided that the endpoint module 616 receives the endpoint 610. The bounding box module 618 can determine the image slope 630 of the image line 310 being processed. The endpoint 610 can indicate the tilt and image slope 630 of the image line 310 being analyzed. The bounding box module 618 can output separately classified world boxes 627 to classify the image line 310 into... Figure 4 Horizontal line 412 or Figure 4 The vertical line is 414.

[0166] The bounding box module 618 can separate image line 310 into horizontal line 412 or vertical line 414. A threshold 632 can be applied to assist in separating horizontal line 412 and vertical line 414. If the image slope 630 of image line 310 is higher than the threshold 632, then image line 310 is determined to be horizontal line 412. If the image slope 630 of image line 310 is less than or equal to the threshold 632, then image line 310 is determined to be vertical line 414.

[0167] The location identifier module 120 can receive world endpoints 624 and world boxes 627 that define the image lines 310 as horizontal lines 412 and vertical lines 414. The neighborhood map 529 can provide... Figure 5 The HD map locally stored 108 corresponds to the scan area 303 and extends a predetermined distance 340 from the geographic location 226 of the in-vehicle image sensor. The predetermined distance 340 is a distance defined by the external parameters 338 of the in-vehicle image sensor 110, within the resolution specifications of the in-vehicle image sensor 110.

[0168] The location recognizer module 120 may provide a rule memory 625 for storing a set of reloadable line rules 626 for recognizing pose relationships 628 for aligning map lines 622 selected from neighboring maps 529 within a predefined distance 340 from geographic location 226 with image lines 310 defined by world endpoints 624 and world boxes 627. It should be understood that geographic location 226 may include error components due to its mechanical properties.

[0169] As an example, the line rules used to determine the pose relationship 628 between map line 622 and image line 310 may include:

[0170] 1) The world endpoint 624 of image line 310 should be offset by approximately the same distance from map line 622.

[0171] 2) The length of image line 310 should perfectly match the length of map line 622.

[0172] 3) The slope of image line 310, 630, should be equal to the slope of image line 622, 630.

[0173] 4) The converging image line 310 and map line 622 should be precisely matched to form a pose relationship 628.

[0174] To understand, world endpoint 624 can provide a depiction of the overlay of image line 310 and map line 622 through the geometric projection of image line 310. Independent positioning module 115 can update geographic location 226 based on the actual location read from map line 622 in neighboring map 529. The update of geographic location 226 can allow... Figure 2 The vehicle 201 can continue to operate without relying on GPS location 130.

[0175] It has been found that the detection of world endpoint 624 allows image line 310 to be aligned with map line 622, which has substantially the same geographic location 226. Map line 622 can provide a verified version of geographic location 226, while image line 310 has an estimate of geographic location 226 provided by in-vehicle position sensor 214. By establishing a pose relationship between image line 310 and map line 622, geographic location can be updated to a verified version of geographic location 226 provided by map line 622. The update of geographic location 226 can be extended. Figure 5 The capability of the first control circuit 512 to generate over a long period of time. Figure 2 Vehicle movement control 228.

[0176] Now for reference Figure 7 The exemplary operation flowchart 701 of the independent positioning mechanism 115 of the navigation system 100 is shown. The exemplary operation flowchart 701 of the navigation system 100 depicts a start box 702, which indicates... Figure 1 The first device 102 due to Figure 2 The obstacle 202 was lost Figure 1 The global positioning location is 130. The process continues until the geographic location is provided by the position sensor 704 inside the vehicle. Figure 2 The vehicle-mounted position sensor 214 can provide Figure 2 The geographic location 226 is used as an estimate based on the last known value of the global positioning location 130. The in-vehicle position sensor 214 may use mechanical devices, such as gyroscopes, accelerometers, force gauges, and inclinometers, to monitor the movement of the first device 102 relative to the last known value of the global positioning location 130.

[0177] The process continues to the image frame 706 recording the scanned area, where the first device 102... Figure 1The in-vehicle image sensor 110 records the scan area 303. The in-vehicle image sensor calibration 336 can generate extrinsic parameters 338 for the in-vehicle image sensor 110. The extrinsic parameters 338 can be used by geometric transformations to locate the world coordinates 334 of the image line 310 in the scan area 303.

[0178] The process continues to the identification and classification, bounding box, and endpoint box 708, where... Figure 1 Artificial intelligence model 116 Figure 6 The deep learning object detector 602 analyzes the sensor data stream 111 of the image sensor 110 within the vehicle. The output of the artificial intelligence model 116 can be for each image line 310 detected in the scan region 303. Figure 6 endpoint 610, Figure 6 box 612 and Figure 6 Category 614.

[0179] The process continues to box 710, which transforms the image coordinates to world coordinates. Here, Figure 1 The line identification module 118 can receive endpoints 610, frames 612, and categories 614. Figure 6 The World Coordinates Module 620 can be based on Figure 3 The external parameter 338 is transformed through geometric transformation Figure 6 Endpoint 610 is transformed into world endpoint 624. Classification 614 can identify items in the scanned area 303 that can be removed from the analysis. This leaves image lines 310 for further analysis. The bounding box module 618 can... Figure 4 Horizontal line 412 and Figure 4 The vertical line 414 is separated. Each set of horizontal lines 412 and vertical lines 414 can be paired with world endpoint 624.

[0180] The process then continues until the map-related location boxes 712 are generated. Figure 1 The location identifier module 120 can Figure 6 Map line 622 is paired with known values ​​of geographic location 226, where image line 310 has an estimated value to geographic location 226. By establishing a pose relationship 628 between map line 622 and image line 310, geographic location 226 can be updated to known and verified values ​​based on map line 622.

[0181] The process continues to the display of the map-related location and manipulation command box 714. The location recognizer module 120 can establish a verified version of the geographic location 226. This allows the first control circuit 512 to generate the next set of vehicle movement controls 228 for controlling the first device 102 as it moves forward through the scan area 303. The first control circuit 512 can... Figure 2The user interface 212 sends the map-related location 109 for display to... Figure 1 User 112.

[0182] Now for reference Figure 8 An embodiment of the invention is shown therein. Figure 1 A flowchart of a method 800 for operating a navigation system 100 is provided. Method 800 includes: determining the geographic location of a vehicle when its global positioning location is obstructed by an obstacle and its geographic location is derived from an in-vehicle position sensor within the vehicle (box 802); in box 804, extracting image lines from an environmental image sensor within the vehicle using an artificial intelligence model; in box 806, transforming the image coordinates of the image lines to world coordinates from a nearby map based on the geographic location; in box 808, extracting map lines from the nearby map based on the world coordinates; in box 810, determining whether the pose relationship between paired image lines and map lines is horizontal or vertical; in box 812, generating a map-related location of the vehicle based on the geographic location; and in box 814, transmitting the map-related location for display on a user interface.

[0183] The resulting methods, processes, apparatuses, devices, products, and / or systems are direct, cost-effective, uncomplicated, highly versatile, accurate, sensitive, and efficient, and can be implemented by adapting known components to readily available, efficient, and economical manufacturing, application, and utilization. Another important aspect of embodiments of the invention is its valuable support for and service to historical trends of cost reduction, system simplification, and performance improvement.

[0184] Therefore, these and other valuable aspects of the embodiments of the present invention advance the state of the technology to at least the next level.

[0185] While the invention has been described in conjunction with specific best practices, it is to be understood that many substitutions, modifications, and variations will be apparent to those skilled in the art based on the foregoing description. Therefore, the invention is intended to cover all such substitutions, modifications, and variations falling within the scope of the included claims. All content set forth herein or shown in the accompanying drawings should be interpreted as illustrative rather than restrictive.

Claims

1. A method of operating a navigation system, comprising: determining a geographic position of a vehicle when a global positioning position is blocked by an obstacle and the geographic position is from an in-vehicle position sensor in the vehicle; extracting image lines from an environment image using an artificial intelligence model and the environment image is from an in-vehicle image sensor in the vehicle; transforming image coordinates of the image lines to world coordinates of a high definition (HD) map local storage based on the geographic position; extracting map lines from a neighboring map in the HD map local storage based on the world coordinates, the map lines corresponding to an area around the geographic position, the area around the geographic position corresponding to a smaller portion in the neighboring map, the smaller portion getting larger over time; classifying the image lines as horizontal image lines or vertical image lines, comprising: determining the image line as a horizontal image line when an image slope of the image line is above a threshold; and determining the image line as a vertical image line when the image slope of the image line is less than or equal to the threshold; pairing the image lines with the map lines, comprising: identifying matching lines from the horizontal image lines and map lines in a horizontal direction; and identifying matching lines from the vertical image lines and map lines in a vertical direction; generating a map-related position of the vehicle based on the geographic position and a pose relationship between the image lines and the map lines; and transmitting the map-related position for display on a user interface.

2. The method of claim 1, wherein, generating the map-related position of the vehicle includes fusing the geographic position based on the in-vehicle position sensor and based on a map matching pose relationship on the HD map local storage.

3. The method of claim 1, wherein transforming the image coordinates of the image line to the world coordinates locally stored by the HD map comprises: transforming the image coordinates of the image lines to the world coordinates of the HD map local storage based on an in-vehicle image sensor calibration of the in-vehicle image sensor.

4. The method of claim 1, wherein, extracting the map lines from the HD map local storage based on the world coordinates includes matching the map lines with the image lines.

5. The method of claim 1, wherein generating the map-relative position of the vehicle comprises: improving the geographic position of the vehicle.

6. The method of claim 1, further comprising receiving an updated artificial intelligence model that has been validated by a background map database, a live position model, and a high definition map convergence model when the global positioning position is available.

7. A navigation system, comprising: a control circuit configured to: determine a geographic position of a vehicle based on a global positioning position being blocked by an obstacle and the geographic position being from an in-vehicle position sensor in the vehicle, extract image lines from an environment image using an artificial intelligence model and the environment image being from an in-vehicle image sensor in the vehicle, transform image coordinates of the image lines to world coordinates of a HD map local storage based on the geographic position, extract map lines from a neighboring map in the HD map local storage based on the world coordinates, the map lines corresponding to an area around the geographic position, the area around the geographic position corresponding to a smaller portion in the neighboring map, the smaller portion getting larger over time, classifying the image line as a horizontal image line or a vertical image line, comprising: determining the image line as a horizontal image line when an image slope of the image line is higher than a threshold; and determining the image line as a vertical image line when the image slope of the image line is less than or equal to the threshold; pairing the image line with the map line, comprising: identifying a matching line from the horizontal image line and a map line in a horizontal direction; and identifying a matching line from the vertical image line and a map line in a vertical direction; generating a map-related position of the vehicle based on the geo-location and a pose relationship between the image line and the map line; and an interface circuit configured to transmit the map-related position for display on a user interface.

8. The system of claim 7, wherein, the control circuit configured to generate the map-related position of the vehicle comprises fusing the geo-location based on an in-vehicle position sensor and based on a map matching of the pose relationship on the HD map local storage.

9. The system of claim 7, wherein the control circuitry configured to transform the image coordinates of the image line to the world coordinates locally stored by the HD map comprises: transforming the image coordinates of the image line into the world coordinates of the HD map local storage based on an in-vehicle image sensor calibration of the in-vehicle image sensor.

10. The system of claim 7, wherein, the control circuit configured to extract the map line from the HD map local storage based on the world coordinates comprises matching the map line with the image line.

11. The system of claim 7, wherein, the control circuit configured to generate the map-related position of the vehicle comprises refining the geo-location of the vehicle.

12. The system of claim 7, wherein, the control circuit configured to receive an updated artificial intelligence model that has been validated by a background map database, a live location model, and a high definition map convergence model when the global positioning location is available.

Images