TRAFFIC SIGN-BASED MAP GRAPH UPDATE DEVICE

Abstract

A vehicle contains at least one imaging sensor. A controller communicates with the at least one imaging sensor and includes a processor and memory. The memory stores a traffic sign inference module configured to receive a lane-level map graph and an image generated by the at least one imaging sensor. The traffic sign inference module is configured to identify a traffic sign's traffic flow message in the image and update the lane-level map graph based on the traffic flow message. The traffic sign inference device is a generative artificial intelligence (AI) and includes a connection to at least one external information source.The traffic sign inference device is configured to identify the traffic flow message at least partially based on a retrieval-extended generation analysis of the external information source.

Description

[0001] The present disclosure relates to vehicles and in particular to a system for updating map graphs for vehicle systems based on detected traffic signs and external knowledge.

[0002] Vehicles equipped with autonomous driving and driver assistance systems use map graphs to identify lanes on roads and to graphically represent transitions between lanes, intersections, and the like. In certain examples, identification can be based on stored high-resolution maps containing lane-level details (HD maps) or medium-resolution maps containing road-level details (MD maps), combined with global navigation satellite systems (GNSS systems) and images provided by vehicle-mounted imaging systems. However, in some cases, the map graph is incomplete and / or cannot be fully completed based solely on stored information. Accordingly, in some instances, the map graph uses HD map data as its basis.

[0003] For example, construction zones with lane closures and lane merges are often not reflected in map graph databases or in high-definition maps used to generate map graphs on the fly due to their temporary nature. Similarly, intersections may have varying lane connections (e.g., right-turn lanes, turning into more than one possible lane, etc.) that are not delineated by lane markings and are not stored in a map database. Completing map graphs across intersections is further complicated by the fact that lanes allowing connections to other lanes may depend on a vehicle's specific location due to regional traffic regulations.

[0004] Some vehicles do not use HD maps or MD maps and instead generate maps spontaneously using only images received from one or more cameras. The methods used by these vehicles are referred to as HD-map-free methods. HD-map-free methods generate lane-level maps and lane-level graphs spontaneously and may contain errors due to the quality of the images received from the camera, occlusions (a large truck obstructing the view), or because a particular topology, such as a lane merge, is far from the camera and is not captured.

[0005] Accordingly, it is desirable to provide a system for updating map graphs that takes into account external knowledge of regional traffic regulations and text and images displayed by signage at or near a location where the map graph is incomplete. Furthermore, it is desirable to spontaneously generate complete and accurate map graphs based on image data generated by the vehicle using the map graph, thereby reducing reliance on pre-calculated data. SUMMARY

[0006] In an exemplary embodiment, a vehicle includes at least one imaging sensor. A controller communicates with the at least one imaging sensor and includes a processor and memory. The memory stores a traffic sign inference module configured to receive a lane-level map graph and an image generated by the at least one imaging sensor. The traffic sign inference module is configured to identify a traffic sign's traffic flow message in the image and update the lane-level map graph based on the traffic flow message. The traffic sign inference device is a generative artificial intelligence (AI) and includes a connection to at least one external information source.The traffic sign inference device is configured to identify the traffic flow message at least partially based on a retrieval-extended generation analysis of the external information source.

[0007] In addition to one or more of the features described here, the traffic sign inference module includes an extraction and marking feature, a function call feature, an external knowledge retrieval feature, a memory, a traffic sign position feature, and a map graph generator.

[0008] In addition to one or more of the features described here, the extraction and marking feature is configured to interpret a scenario and / or maneuver and / or attribute and / or vehicle type exclusion and / or time limit and / or lane vehicle type from the traffic flow message and an initial map graph.

[0009] In addition to one or more of the features described here, the traffic sign inference module includes a large language model (LLM).

[0010] In addition to one or more of the features described here, the LLM includes a relevance detection feature, where the relevance detection feature is configured to determine the relevance of the traffic flow message to the vehicle based on an output of the marker and extraction feature, the image, and an initial map graph.

[0011] In addition to one or more of the features described here, the relevance detection feature is further configured to call at least one rule-based function, and the at least one rule-based function is configured to respond with supplementary data.

[0012] In addition to one or more of the features described here, the at least one rule-based function includes a time and date function, an obstacle detection function, a scene information function, and a vehicle type function.

[0013] In addition to one or more of the features described herein, the LLM further includes a map graph update facility, wherein the map graph update facility is configured to update the initial map graph based on the image using a rule-based map graph update process and / or a map graph update process based on a deep learning network graph generator (GNN) and / or an LLM-based map graph update process.

[0014] In addition to one or more of the features described here, the map graph update facility is configured to update the initial map graph using each of the rule-based map graph update process, the GNN update process, and the rule-based map graph update process.

[0015] In addition to one or more of the features described here, an output from the map graph update device is provided to memory as an updated map graph, with memory configured to retain the updated map graph as long as the relevance detection feature indicates that the traffic flow message is relevant to the vehicle.

[0016] In addition to one or more of the features described herein, an output of the map graph update device is provided for at least one vehicle system, wherein the at least one vehicle system is configured to operate an autonomous vehicle function and a semi-autonomous vehicle function at least partially based on the updated map graph.

[0017] In addition to one or more of the features described here, the updated map graph is specific to a vehicle context, where the vehicle context includes a vehicle type and / or a time of day and / or an ambient lighting condition.

[0018] In addition to one or more of the features described here, the external knowledge retrieval feature is a retrieval extended generation (RAG) configured to identify a traffic sign traffic flow message and to respond to the identified traffic flow message being incomplete by querying the external information and completing the traffic flow message using the external information.

[0019] In addition to one or more of the features described here, completing the traffic flow message using external information includes deriving missing information using context-dependent clues in the external information.

[0020] In addition to one or more of the features described here, completing the traffic flow message using the external information includes identifying an explicit explanation of the missing information in the external information and updating the traffic flow message with the explicitly stated missing information.

[0021] In addition to one or more of the features described here, the lane level map graph is a Maples map graph and is generated at least partially based on edge detection of the image.

[0022] In another exemplary embodiment, a process for updating a map graph for a vehicle includes receiving an initial map graph and an image of a traffic sign from a traffic sign inference device and determining a traffic flow message of the traffic sign, at least partially, based on accessing an external information source using a retrieval-enhanced generation process. The initial map graph is updated to an updated map graph based on the traffic flow message and the initial map graph. The updated map graph is provided to at least one vehicle operating system. The at least one vehicle operating system is one for autonomous vehicle operation and one for semi-autonomous vehicle operation.

[0023] In addition to one or more of the features described here, updating the initial map graph to the updated map graph involves using an extraction and marker feature of a large language model to interpret a scenario and / or maneuver and / or attribute and / or vehicle type exclusion and / or time limit and / or lane vehicle type from the traffic flow message and an initial map graph.

[0024] In addition to one or more of the features described here, the relevance of the traffic flow message for the vehicle is determined at least partially based on the initial map graph, the output of the retrieval-supplemented generation process, and the output of a rule-based function call, and the updated map graph is stored in a memory as long as the relevance of the traffic flow message is determined to be relevant, the relevance being based at least partially on the determined time limit and the lane vehicle type.

[0025] In addition to one or more of the features described here, the retrieval-enhanced generation process is configured to identify a traffic flow message of the traffic sign and to respond to the fact that the identified traffic flow message is incomplete by querying the at least one external information source and completing the traffic flow message using the external information.

[0026] The features and advantages described above, and further features and advantages of the disclosure, will become apparent from the following detailed description when taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Further features, advantages and details appear only as examples in the following detailed description, which refers to the drawings; they show: Fig. 1 a vehicle that includes a map graph verification and update architecture; Fig. 2 a high-level map graph update process; Fig. 3. An exemplary implementation of the map graph update process of Fig. 2; Fig. 4. An example architecture for the map graph update system of Fig. 1- Fig. 3; Fig. 5. A precise implementation of the example architecture of Fig. 4 and Fig. 6. An implementation of the exact architecture of Fig. 5 according to an example. DETAILED DESCRIPTION

[0028] The following description is merely exemplary and is not intended to limit the present disclosure, its application, or uses. It should be understood that throughout the drawings, corresponding reference numerals denote similar or corresponding sections and features. As used herein, the term "module" refers to a processing circuit arrangement that may include an application-specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or grouped) with memory executing one or more software or firmware programs, a combinational logic circuit, and / or other suitable components providing the described functionality.

[0029] As used here, a map graph refers to a directed graph formed as an intersection of finitely many simply connected and internally disconnected regions of a Euclidean plane. A map graph is a graph used to represent the physical environment, the road network, and the surroundings that a map represents. The map graph typically contains nodes and edges, with each element representing different aspects of the driving environment. Alternative implementations may vary somewhat from the one described here and still fall under the umbrella term map graphs. Nodes represent lane segments (a portion of a lane), and edges represent how lanes are connected and how a vehicle can move from one lane segment to another. Nodes and edges may have corresponding attributes such as...a speed limit, a stop sign, a yield sign, and similar features.

[0030] As used here, a controller refers to a system containing a processor and memory configured to implement a described function. The system can be a dedicated controller for the described function, a general-purpose controller containing a module for performing the described function, a distributed network of processors and memory configured to work together to implement the described function, or any similar structure configured to implement the function.

[0031] In a general example, the map graph update system and architecture described here generate map graphs based on existing map data using a neural network and / or retrieve existing map graphs from a map graph source. A camera or other image sensor captures an image of a traffic flow sign and provides the image to the map graph update system. The map graph update system accesses an external knowledge base containing rules and regulations about a current area in which the vehicle running the map graph update system is traveling. The map graph update system can also gather information about the environment in which the vehicle is traveling. This includes information about the time, date, objects on the road, and information about the type of vehicle (bus, passenger car, truck, etc.).

[0032] The map graph update system analyzes the image and / or text on the traffic flow sign to identify a traffic flow message. The system then either checks the existing map graph or identifies a gap in the existing map graph and an inconsistency between the traffic flow sign and the map graph. As used here, a traffic flow sign is a subset of traffic signs that contain a message or communication specifying a map graph feature. For example, a right-turn-only sign is a traffic flow sign, but a seatbelt sign is not.

[0033] In the event of an inconsistency, the map graph update system assumes that any traffic flow information communicated by the traffic sign will override any stored map graph information. The map graph update system then updates the map graph using a combination of the analyzed image and external knowledge, and the updated map graph is stored for use by one or more vehicle systems.

[0034] In certain examples, the updated map graph is provided to an external data store and can be provided to or retrieved by other vehicles traveling in the same area. In such examples, the data includes context-dependent information such as the vehicle type, the time and date the data was generated, object detections, and the like.

[0035] An exemplary embodiment illustrates Fig. 1. A vehicle 10. The vehicle 10 contains a controller 20 and an imaging sensor 30 communicating with the controller 20. The controller 20 contains a map graph update architecture 22, a processor 23, and memory 24 for storing map graphs and external information. Additionally, the controller 20 communicates with one or more external information sources 26. This communication can be direct via a cellular connection or indirect via a cloud computing service or a large-scale network (e.g., the internet). In certain examples, the external information sources 26 may, in addition to storing some or all of the external information, contain processing resources that can work in conjunction with the processor 23 to assist in processing one or more functions in the map graph update architecture 22.

[0036] The imaging sensor 30 defines a field of view 32 capable of detecting and capturing an image of a traffic sign 34 in front of the vehicle 10. In alternative examples, additional imaging sensors 30 can be used anywhere in the vehicle 10 and / or the imaging sensor 30 can be located at an alternative location on the vehicle 10 and operate in a similar manner. Furthermore, the example of Fig. 1 For reasons of expediency, it is described with reference to a traffic sign 34 in front of the vehicle. In a practical implementation, the traffic sign 34 can be at any position relative to the vehicle 10 when it is detected by the imaging sensor 30, and the operations of the map graph update architecture 22 remain the same.

[0037] In certain examples, the Controller 20 may also include a map graph generation module capable of generating a map graph of a current road using lane markings and edge detection, without using HD map data. This process is called Maples map graph generation.

[0038] With continued reference to Fig. 1 illustrates Fig. 2 a high-level process of operations of the map graph update architecture 22. The general structure includes a traffic sign inference device 240 that receives a lane-level map graph 210 of the road on which the vehicle 10 is currently traveling, any related external knowledge 220 for that road and / or the area in which the vehicle is currently traveling, and an image of the traffic flow sign 34, a message extracted from the traffic flow sign 34, or both the image and the message extracted from the traffic flow sign 34. The information from the traffic flow sign 34 is referred to as a traffic sign message 230.

[0039] In certain examples, the traffic sign message 230 is extracted using a visual and linguistic model analysis of traffic sign 34. Such examples are used when the traffic sign message is textual in nature and / or traffic sign 34 is not a standardized traffic sign. In other examples, where traffic sign 34 is a standardized traffic sign or contains standardized symbols, information from external knowledge 220 can be used to extract the meaning of the symbols and traffic flow messages. For universal or well-known symbols (e.g., a red octagon as a stop sign), the meaning of the traffic sign can be extracted using classification. However, less well-known traffic signs and / or unclear and incomplete messages require proper interpretation of external knowledge 220.An example of a visual traffic sign that requires external knowledge might be a complex, rare visual traffic sign, such as a warning of falling rocks. A primary use of external knowledge is when the traffic sign message does not contain all the necessary information. For example, a traffic sign that simply states "Merging ahead" does not specify where the merging occurs or which lane is merging. Similarly, a traffic sign that states "Right lane must turn right" does not need to specify which lane the right lane should turn into.

[0040] The traffic sign inference device 240 uses the combined information to identify discrepancies and gaps in the map graph 210 and corrects these discrepancies or gaps. The corrected map graph is then output by the traffic sign inference device as an updated map graph 250. The updated map graph 250 is made available to any operating autonomous or semi-autonomous vehicle systems that use map graphs (e.g., a motion planner). Additionally, the map graph 250 can be made available to external information sources 26 or any other connected map graph repository and stored for access by other vehicles or systems that use map graphs.

[0041] With continued reference to Fig. 1 and Fig. 2 illustrates Fig. 3. An example operation of architecture 22 is described as follows: while vehicle 10 approaches an intersection 212, where the road is divided into lanes (L1, L3, L4, L6, L9, and L10) by means of lane markings, the lane markings not extending across the intersection 212. In the example operation, as vehicle 10 approaches the intersection 212, a map graph of the intersection is generated using a deep learning network graph generator (GNN) based on the lane markings and the structure of the intersection 212 and edge detection in the image collected by the camera. In alternative examples, the initial map graph 210 can be retrieved from a data source or generated using any alternative method of generation.

[0042] Due to the lack of lane markings crossing intersection 212, the initially generated map graph 210 does not connect the lanes across intersection 212. The illustration of Fig. Figure 3 contains a schematic top view of the intersection 212 and a data representation 214 of the map graph 210, wherein the data representation 214 specifies the features (lanes 213 and connections 215, where the lanes 213 can be traversed by means of a lateral movement between lanes 213) that are identified in the map graph 210. The generated map graph 210 is provided for the traffic sign inference device 240.

[0043] Additionally, the imaging sensor 30 on the vehicle 10 captures an image of a traffic sign 230, and the image is provided to the traffic sign inference device 240. In the illustrated example, the traffic sign 230 contains lane indicators showing that a left lane (L1) is only permitted to proceed straight through intersection 212, and the right lane (L4) is only permitted to turn right at intersection 212. Because the traffic sign 240 uses standardized symbology, the traffic sign inference device 240 uses a symbol interpretation database to identify the meaning of the symbols contained in the traffic sign 230. In alternative examples where the traffic sign 34 uses text to indicate the permitted traffic flow (e.g.,To communicate a construction lane closure or a detour traffic sign, the traffic sign inference device 240 interprets the text on traffic sign 230 using a large language model (LLM) or a similar machine learning structure. For example, the similar machine learning structure could contain a generative artificial intelligence (Gen AI) structure.

[0044] Based on the gap at intersection 212 and the traffic flow message from traffic sign 230, the traffic sign inference device 240 can determine that lane L1 connects to L3 via a lane segment L2, which must be created and added to the graph. However, lane L4 could turn right into either lane L9 (using lane segment L7) or lane L10 (using lane segment L8). Which lane segment L7, L8, or both can be used is area-dependent. For example, in some states, a vehicle in lane L4 is permitted to turn into lane L10, while in others, the vehicle in lane L4 is only permitted to turn into lane L9. To make this determination, the traffic sign inference device 240 accesses the external information sources 220, which contain the remote system 26.The external information sources 220 contain codes of law and regulations for various areas and municipalities, and the traffic sign inference device 240 analyzes the codes of law and regulations using an LLM to determine the correct connections for lanes L1 and L4 across intersection 212.

[0045] When the correct connections have been determined, the traffic sign inference device 240 outputs the updated map graph 250, which contains the complete set of lanes 213 and junctions 215, and the output map graph 250 is made available to a planner module of one or more additional systems in the vehicle 10, as described above.

[0046] With continued reference to Fig. 1- Fig. 3 illustrates Fig. 4 An example architecture 400 of the traffic sign inference device 240. The traffic sign inference device 240 contains a large language model machine learning network (LLM 410) that is trained, an extraction and marker feature module 412 for interpreting the traffic flow sign 34, and a function call module 414 for calling and executing predefined functions 420 used in the data collection to support the LLM. For example, the predefined functions 420 might include a node generator function 422 that is configured to create new nodes for the new map graph (e.g., L2, L7, and L8 of the map). Fig. 3) to generate, a time / date function 424 configured to provide the current time / date to the LLM 410, a blockage detection function 426 configured to identify cones, blocks, and other obstructions above the road based on images captured by the imaging sensor 30, a query scene information function 428 to identify information regarding the environment, and a query vehicle type function 430 to identify a type of vehicle 10. Each of the functions 420 operates using predefined computer code and module interconnections and does not use machine learning as a basis. The functions 420 can be performed according to any conventional process for performing a defined function 420.Furthermore, depending on the details of that particular implementation, alternative implementations may include a subset of these functions 420 and / or additional functions 420.

[0047] In certain examples where the graph node is stored for future use and / or made available to other vehicles, the LLM 410 may include an optional transience detection function 411. The optional transience detection function 411 determines whether the traffic sign 34 is of a permanent nature, such as a right turn, or merely of a temporary nature, such as a construction lane merging sign. Based on this determination, a graph generation feature 440 may determine whether the updated map graph should be stored for future use or whether the application of the updated map graph should be limited to the current operations of vehicle 10. Additionally, the graph generation feature 440 applies context-dependent data as metadata to the map.The context-dependent data defines the scope of the map in such a way that, if it is stored for use by other vehicles, the map will only be applied by similar vehicles in a similar situation (e.g., vehicles that match a certain threshold of the stored context).

[0048] A traffic sign position feature 442 identifies where traffic sign 34 is located in relation to map graph 210 and determines where the information provided by traffic sign 34 should be applied on map graph 210. For example, a traffic sign indicating an upcoming lane closure indicates a closure of one of the lanes, and the traffic sign position feature identifies where the lane is closed on map graph 210. This determination can be based on factors such as the vehicle's longitudinal direction, the lane the vehicle is in, other traffic features, information contained on the traffic sign (e.g., 200 feet ahead), and the like.

[0049] The LLM 410 also includes a memory 418 for temporarily storing information during the operation of the traffic sign inference device 240 and a retrieval extended generator (RAG) 416 for coupling to and retrieving information from the external knowledge source 26.

[0050] A graph generation feature 440 generates the final map graph 250 based on the complete operations of the LLM 410 and outputs the final map graph to the one or more systems that will use the map graph 250.

[0051] With continued reference to Fig. 1- Fig. 4 illustrates Fig. 5. A detailed process flow using the example architecture of Fig. 4 according to an implementation 500 with arrows 505 that connect different modules and indicate a general direction of the data flow. It should be acknowledged that the general direction of the data flow indicated by the arrows 505 is not restrictive and that certain modules or features can communicate with and exchange data with other modules, even if this is not explicitly indicated by the data flow arrows 505.

[0052] In the exact implementation, an initial traffic sign message 501 and an initial map graph 503 are provided for the extraction and marking feature 412. Using the traffic sign message 501, the extraction and marking feature identifies a scenario 502, a maneuver and direction of the vehicle 10 (maneuver 504), applicable attributes 506, vehicle type exclusions 508, time limits 510, and lane vehicle types 512, and provides the extracted information to the LLM 410. In alternative examples, additional features can be extracted from the extraction and marking feature 412 and provided to the LLM 410 (as in Fig. 4 is illustrated).

[0053] Scenario 502 identifies the type of operation encountered. Examples of scenarios include merges, intersections, four-way intersections, lane closures, lane obstructions (such as a disabled vehicle), T-junctions, and similar situations.

[0054] Maneuver 504 identifies the direction of movement that vehicle 10 must follow to execute the traffic flow message, as well as any specific factors that restrict or otherwise affect the execution of the traffic flow message (e.g., a restriction to turning right only, which applies only to the rightmost lane).

[0055] Attribute detection 506 identifies any relevant attributes for the traffic flow information contained in the traffic sign message 510.

[0056] For example, the relevant information may include a speed limit, a yield sign, a stop sign, and the like. In certain examples, attribute detection can further detect whether the traffic flow message contained in the 501 traffic sign message is of a transitory nature (e.g., temporary roadworks) or a permanent nature.

[0057] The vehicle type exclusions 508 determine whether any features included in the traffic sign message apply only to certain vehicles (e.g. bus lanes, heavy goods vehicle exclusions, and the like).

[0058] The Time Limits 510 define any time limits that apply to Traffic Sign Message 501. For example, the Time Limits may include weekend exceptions, daytime exceptions, school hour exceptions, or any other time that restricts when the traffic flow information contained in Traffic Sign Message 501 is applicable.

[0059] The lane vehicle type detection 512 identifies restrictions for specific lanes (e.g., bus lanes, high-occupancy vehicle lanes, bicycle lanes, etc.) that limit which types of vehicles can operate in that lane.

[0060] The data from the extraction and marking feature 412 are obtained by the LLM 410 and the LLM 410 determines the relevance of any extracted information in a relevance step 550.

[0061] Based on the extracted information, relevance step 550 determines whether the extracted traffic flow information (e.g., a lane merging) is currently relevant for vehicle 10.

[0062] After determining the relevance of the traffic flow message to vehicle 10, the LLM 410 uses the function caller 414 and the RAG 416 to operate functions 420 and the information retrieval using RAG 416 in step 560. Functions 420 gather additional information, including the current time, date, vehicle 10 type, road obstructions, scene information, the longitudinal location 522 where the map graph change should apply, the lane 524 to turn into, a lateral location 526 where the map graph change should apply, applicable traffic rules 528, and the like.

[0063] The graph update device 440 and the RAG 416 receive the traffic sign message, the initial map graph, and any additional information retrieved by functions 420 and the RAG 416, and update the map graph based on the traffic flow information. The map graph update process uses a combination of rule-based map graph updates 542, map graph updates 544 based on a graphical neural network (GNN), and LLM-based map graph update device 546. First, the graph update device 440 uses the rule-based updates 542 to apply a set of rules, which use known information as a basis, to update the map graph.

[0064] A GNN is then applied to the graph, and any image information from the lane markings, captured by the imaging sensor 30 on the vehicle 10, is used to create new messages and / or update node attributes. The GNN extrapolates as much map graph data as possible from the drawn markings, construction buoys, guardrails, and other visible features of the road itself.

[0065] When the final updated map graph has been verified by the LLM 546, the updated map graph is stored in memory 418 in step 570 until the LLM 410 determines, using function caller 414, that the map graph is currently relevant.

[0066] In alternative examples, one or two of the map generation features (the rule-based generator 542, the GNN-based generator 544 and the LLM-based generator 546) may be omitted and the general function of the system may be maintained.

[0067] In certain examples, the relevance of the map graph in memory 418 is maintained until the vehicle 10 has traveled a predetermined distance beyond a point where the traffic sign is relevant. In one particular example, a lane merge traffic sign indicating that the lane will merge in 200 feet might result in a generated map graph that is maintained until 300 feet beyond the traffic sign (100 feet beyond the merge point). In an alternative example, the map graph is maintained until the merge is complete, as determined by the imaging sensor 30.

[0068] Before vehicle 10 reaches the point where the map graph becomes relevant, the map graph is provided to a planner module in a map graph output step 580. The planner module uses the map graph according to any standard usage to operate autonomous or semi-autonomous vehicle operation.

[0069] With continued reference to Fig. 1- Fig. 5 illustrates Fig. 6. An example operation of the implementation, which is in Fig. Figure 5 illustrates this when the traffic sign message 501 is a traffic sign for "right lane closed at 1500 feet". In the example of Fig. 6. It must be acknowledged that the structure is the same as in Fig. 5 is.

[0070] In the extraction and marker feature module 412, the traffic flow message identified on the traffic sign is, “A lane merge between the rightmost lane and the immediately adjacent lane occurs 1500 feet from this point.” Based on this traffic flow message, the scenario is extracted as a merge scenario 502 with a maneuver 504 on the right side of vehicle 10. No other information is included in the traffic flow message. The extracted information is provided to function caller 414, which, by calling functions 420, determines that the traffic sign is relevant to vehicle 10.

[0071] In the example of Fig.Relevance feature 414 specifies that the longitudinal location of the merge with respect to vehicle 10 and the transverse location of the merge with respect to vehicle 10 are required to determine if the updated map graph is relevant. Relevance step 550 then calls functions 420 to determine the appropriate information. The functions 420 that are called include a longitudinal location function 602, which determines that the traffic flow message applies 1500 feet from the location of the traffic sign message 501, and a transverse location function 604, which determines that the traffic flow message applies to the right of vehicle 10.

[0072] Additionally, Function 420 includes a check 606 to determine any information not present in the traffic sign message 501 and a use of RAG 416 to search a relevant traffic manual for the relevant information. For example, if the traffic sign for the upcoming right lane closure omits a distance, RAG 416 can manually query the traffic sign to determine what regulations (if any) apply to the current traffic sign. If a regulation is relevant (e.g., traffic signs to close a lane must be placed 500 feet before the closure), the information is identified by the RAG 416 search. The identified information is provided back to the relevance module 414, which then determines whether the traffic sign 501 is relevant to the current map graph.

[0073] In certain examples, the required information is directly mentioned in external knowledge 220. For example, external knowledge 220 may explicitly state that "merging traffic signs are installed 500 feet before a merge."

[0074] In alternative examples, external knowledge 220 does not directly provide the missing information, and this information is inferred using LLM 410. For example, external knowledge 220 may contain statements indicating that a merge sign should be installed at a position that provides the driver with 14 to 14.5 seconds of vehicle maneuvering time before the merge. Based on this time and additional information (such as speed limits, direction and severity of the maneuver, etc.), LLM 410 infers the distance from the traffic sign to the merge point.

[0075] In further examples, LLM 410 can use several different statements from different points in external knowledge to construct the conclusion, and LLM 410 is not limited to identifying or resorting to single statements to provide the context.

[0076] The map graph update unit 440 updates the map graph to reflect lane merging, using any of the map graph generation techniques, and makes the updated map graph 606 available to memory 418, where it is retained for the specified period. While stored in memory, the updated map graph 606 is published (made available) to any planner module for an autonomous or semi-autonomous vehicle system.

[0077] Furthermore, the processing for operating the traffic sign inference device 240 can be carried out entirely in the vehicle controller 20. In other examples, the processing can be split between the controller 20 and an external controller, or carried out entirely by the external controller, and still fall within the systems described here.

[0078] The terms "a" and "an" do not denote a limit on the number of elements, but rather indicate the presence of at least one of the referenced element. The term "or" means "and / or" unless clearly indicated otherwise by context. A reference to "an aspect" in the application text means that a specific element (e.g., a feature, a structure, a step, or a property) described in connection with that aspect is contained in at least one aspect described therein and may or may not be present in other aspects. It should also be understood that the described elements in the various aspects may be combined in any suitable manner.

[0079] When an element, such as a layer, a thin layer, an area, or a substrate, is described as "attached" to another element, it may be located directly adjacent to that other element, or there may be intervening elements. Conversely, when an element is described as "directly adjacent" to another element, there are no intervening elements.

[0080] Unless otherwise specified herein, all testing standards shall be the most recent valid standard as of the filing date of this application or, if priority is claimed, as of the filing date of the earliest priority application in which the testing standard appears.

[0081] Unless otherwise defined, technical and scientific terms used herein have the same meaning as would normally be understood by a person skilled in the field to which this disclosure belongs.

[0082] While the disclosure described above has been described with reference to exemplary embodiments, those skilled in the art will understand that various modifications can be made and elements can be replaced by their equivalents without altering its scope. Furthermore, many adaptations can be made to fit a particular situation or material to the instructions given in the disclosure without deviating from its essential scope. Therefore, it is intended that the present disclosure is not limited to the specific embodiments disclosed, but includes all embodiments that fall within its scope.

Claims

[1] Vehicle comprising: at least one imaging sensor and a controller in communication with the at least one imaging sensor, wherein the controller includes a processor and a memory, the memory storing a traffic sign inference module configured to receive a lane-level map graph and an image generated by the at least one imaging sensor, and configured to identify a traffic flow message from a traffic sign in the image and to update the lane-level map graph based on the traffic flow message; wherein the traffic sign inference module is a generative artificial intelligence (AI) and includes a connection to at least one external information source and the traffic sign inference module is configured to identify the traffic flow message at least partially based on a retrieval-extended generation analysis of the external information source. [2] Vehicle according to claim 1, wherein the traffic sign inference module includes an extraction and marking feature, a function call feature, an external knowledge retrieval feature, a memory, a traffic sign position feature and a map graph generator, the extraction and marking feature being configured to interpret a scenario and / or a maneuver and / or an attribute and / or a vehicle type exclusion and / or a time limit and / or a lane vehicle type from the traffic flow message and an initial map graph, the traffic sign inference module comprising a large language model (LLM) having a relevance detection feature, and the relevance detection feature being configured to determine a relevance of the traffic flow message for the vehicle based on an output of the marking and extraction feature, the image and an initial map graph. [3] Vehicle according to claim 2, wherein the relevance detection feature is further configured to call at least one rule-based function, and the at least one rule-based function is configured to respond with supplementary data. [4] Vehicle according to claim 3, wherein the at least one rule-based function includes a time and date function, an obstacle detection function, a scene information function and a vehicle type function. [5] Vehicle according to claim 4, wherein the LLM further includes a map graph update device and the map graph update device is configured to update the initial map graph based on the image using a rule-based map graph update process and / or a map graph update process based on a deep learning network graph generator (GNN) and / or an LLM-based map graph update process. [6] Vehicle according to claim 5, wherein the map graph update device is configured to update the initial map graph using each of the rule-based map graph update process, the GNN update process and the rule-based map graph update process. [7] Vehicle according to claim 5, wherein an output of the map graph update device is provided as an updated map graph for the memory, wherein the memory is configured to retain the updated map graph as long as the relevance detection feature indicates that the traffic flow message is relevant to the vehicle, and an output of the map graph update device is provided for at least one vehicle system, wherein the at least one vehicle system is configured to operate an autonomous vehicle function and a semi-autonomous vehicle function at least partially on the basis of the updated map graph. [8] Vehicle according to claim 7, wherein the updated map graph is specific for a vehicle context, the vehicle context includes a vehicle type and / or a time of day and / or an ambient lighting condition, and the external knowledge retrieval feature is a retrieval extended generation (RAG) configured to identify a traffic sign traffic flow message and to respond to the identified traffic flow message being incomplete by querying the at least one external information source and completing the traffic flow message using the external information. [9] Vehicle according to claim 7, wherein completing the traffic flow message using the at least one external information source includes deriving missing information using context-dependent clues in external information. [10] Vehicle according to claim 7, wherein completing the traffic flow message using the external information includes identifying an express declaration of missing information in the at least one external information source and updating the traffic flow message with the expressly stated missing information.

Images