System and method for translating natural language traffic rules into formal logic for autonomous moving vehicles

Abstract

Methods and systems for translating natural language traffic rules into formal logic for autonomous moving vehicles. The present invention relates to a system and method for translating natural language traffic rules into formal logic, including a training method for machine learning systems, and may be used in the context of advanced driving assistance systems or automatic driving systems (ADAS or AD) for vehicles or other autonomous moving vehicles such as robots and drones. A method for preparing a machine learning system for translating natural language traffic rules into formal logic representations for autonomous moving (or autonomous driving) vehicles. The machine learning system includes a (large) pre-trained language model that has been trained on a large amount of text data to assign probabilities to a sequence of words. The machine learning system is prepared by one of the following selectable schemes: i. Prompting using a pre-trained language model in the following manner:-the inputs of the pre-trained language model are natural language instructions and language traffic rules,-presenting multiple inputs and corresponding formal logic target outputs to the machine learning system as (less sample) instances, the machine learning system is enabled to learn and generate formal logic traffic rules when the natural language instructions and the natural language traffic rules are prompted. Ii. Fine tuning the machine learning system by using training data comprising language traffic rules as training inputs and corresponding formal logic traffic rules as target outputs for the pre-trained language model (or accordingly, the machine learning system). These language traffic rules include natural language rules and / or synthetic language traffic rules.

Description

[0001] The present invention relates to a system and method for translating natural language traffic rules into formal logic (including a training method for a machine learning system), and can be used in the context of advanced driver assistance systems or automated driving systems (ADAS or AD) for vehicles or other autonomous mobile vehicles (AVs) such as robots and drones.

[0002] DE 10 2004 019337 A1 shows an assistance system for a motor vehicle, in particular a right-of-way assistant. The assistance system comprises at least one control unit which selects data and simultaneously processes data streams from at least three groups: global data (e.g. navigation system information), local data provided by ADAS sensors, for example, and internal data provided by vehicle sensors (e.g. providing information input interactively by the driver, for example via the steering wheel). In order for the right-of-way assistant to assist the driver, it is essential to know the traffic rules in advance. Some examples are given here about who has the right of way in specific situations according to legal provisions and, for example, when there are traffic signs or traffic lights at intersections.

[0003] US 8,452,524 B2 proposes a method for identifying traffic-related information in a driven motor vehicle, wherein sensor data from sensors and map data from a navigation system are interpreted for identification purposes. In a first scenario interpretation, the sensor data are checked for the presence of traffic-related information and the data are compressed into relevant sensor data. In a second scenario interpretation, the map data are checked for the presence of traffic-related information and the data are compressed into relevant map data. The relevant sensor data and the relevant map data are fed to a regulator for interpreting the sensor and map data, wherein the regulator evaluates the sensor data and the map data and then emits an output signal corresponding to the evaluation. In the regulator, the relevant sensor and map data are interpreted based on a specific set of rules, and the actions specified by the regulator parameterize the observed behavior based on the recognized rules. The regulator forwards the output signal to a finite state machine. The finite state machine performs transitions in a state space with predefined states and emits information, which is assigned to a certain state or the transition.

[0004] Both approaches rely on rules that may be codified through statutes and other legal provisions.

[0005] US10,133,275B1 describes a system that uses temporal logic (e.g., linear temporal logic or signal temporal logic) to generate trajectories for autonomous vehicles. Temporal logic formulas (e.g., linear temporal logic (LTL)) can be used to verify whether possible trajectories satisfy road rules. The state of the environment and the autonomous vehicle can be represented using this formal logic.

[0006] US10,372,130B1 describes the communication of the reasons for vehicle actions. This article describes a technology for communicating feedback to passengers of an autonomous vehicle about the reasons for the actions taken by the autonomous vehicle to build trust with the passengers. For example, the autonomous vehicle can associate various objects with symbols and / or predicates when traveling through the path to evaluate linear temporal logic (LTL) formulas. Events occurring on the path may require the autonomous vehicle to perform a certain action. Based on the evaluation of the LTL formula, the vehicle can determine to communicate events and / or actions to the passengers to provide the reasons why the autonomous vehicle takes the action. In some instances, the autonomous vehicle can communicate with the passengers via one or more of visual cues, auditory cues, and / or tactile cues. In this way, the autonomous vehicle can build trust with the passengers by comforting and informing the passengers of the reasons for taking the action before, during, or after taking the action. When determining the vehicle action, the "cost" associated with different actions can be determined. Complying with traffic rules is an example of such a cost factor.

[0007] In Henry Prakken's conceptual work (On the problem of making autonomous vehicles conform to traffic law, Artificial Intelligence and Law, 25(3):341–363), it is pointed out that (traffic) rules have clear conditions and moral-logical structure.

[0008] Bhuiyan, H., Governatori, G., Rakotonirainy, A., Bond, A., Demmel, S., and Islam, M. B. (Traffic rules encoding using defeasible deontic logic, Proceedings of JURIX 2020: 33rd International Conference on Legal Knowledge and Information Systems) proposed the concept of a deontic logic structure for traffic rules and suggested using Turnip ( https: / / turnipbox.netlify.com, retrieved on August 11, 2022) as an inference engine to implement traffic rules in the form of defeasible deontic logic (DDL). In DDL, the effects of traffic rules are expressed as obligations (a certain behavior is mandatory), prohibitions (a certain maneuver is not allowed), and permissions (an autonomous mobile vehicle AV can perform certain actions). The authors built a knowledge base including AV behavior and environment ontologies, thereby collecting knowledge base facts about the environment from a driving simulator (for example, the presence or absence of certain lane markings or traffic signs). The traffic rules themselves were encoded into DDL through a process of roughly manually crafted translations from the natural language original text in the traffic rulebook.

[0009] Unlike DDL, Alves, G. V., Dennis, L., and Fisher, M. (Formalisation and implementation of road junction rules on an autonomous vehicle modelled as anagent), Sekerinski, E., Moreira, N., Oliveira, J. N., Ratiu, D., Guidotti, R., Farrell, M., Luckcuck, M., Marmsoler, D., Campos, J., Astarte, T., Gonnord, L., Cerone, A., Couto, L., Dongol, B., Kutrib, M., Monteiro, P., and Delmas, D. (Eds.), Formal Methods, FM2019 International Workshops 2019 International Symposium on Traffic Control, pp. 217-232, Kamm, Springer International Publishing) proposed using temporal logic to formalize traffic rules and embed them into AVs in an agent-based architecture. The main focus is on traffic rules regarding traffic conditions at intersections. The formalization of traffic rules is done using a (presumably manually crafted) grammatical rule in the style of Extended Backus-Naur Form (EBNF).

[0010] Rizaldi, A. and Althoff, M. (Formalising traffic rules for accountability of autonomous vehicles, 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pp. 1658-1665) formalize traffic rules in linear temporal logic (LTL) and verify them using the Isabelle theorem prover. The authors focus on the part of the Vienna Convention on Road Traffic that applies to highway scenarios.

[0011] Esterle, K., Gressenbuch, L. and Knoll, AC (Formalizing Traffic Rules for Machine Interpretability, CoRR, abs / 2007.00330) also formalized traffic rules in LTL, but focused on the German StVO regulations. Their study dealt with traffic rules applicable to dual carriageways, focusing on speed, road and lane use, overtaking, safe distance, being overtaken and priority. The authors suggested dividing the rules into premises and conclusions. For example, the rule "On motorways and motorways, parking is prohibited, including on the side of the road" is transformed into On motorways (premise) and No parking (conclusion), and then encoded in LTL.

[0012] Karimi, A. and Duggirala, P.S. (Formalizing traffic rules for uncontrolled intersections, 2020 ACM / IEEE 11th International Conference on Cyber-Physical Systems, pp. 41-50) refer to the California Driver's Manual ( https: / / www.dmv.ca.gov / portal / handbook / california-driver-handbook / , retrieved on August 11, 2022) contains some traffic rules (related to intersections) that are manually translated into a formal language. The authors formalize the traffic rules using the logic programming paradigm of Answer Set Programming (ASP).

[0013] Maierhofer, S., Rettinger, A.-K., Mayer, EC, and Althoff, M. (Formalization of interstate traffic rules in temporal logic, 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 752–759) formalized selected traffic rules from the German Road Traffic Ordinance (StVO) and the Vienna Convention on Road Traffic in metric temporal logic (MTL). MTL allows "specifying time intervals within which a property must be satisfied," thus capturing the temporal nature of traffic rules. The selected traffic rules in this study concern traffic situations such as keeping a safe distance, avoiding unnecessary braking, obeying speed limits, observing traffic flow, etc.

[0014] While the research we presented above focuses on the temporal aspects of traffic rules through formalization in temporal logic, Schwammberger, M. and Alves, GV (Extending urban multi-lane spatial logic to formalise road junction rules, arXiv preprint arXiv:2110.12583) also address the spatial aspects of the rules by extending urban multi-lane spatial logic (Schwammberger, M., An abstract model for proving safety of autonomous urban traffic, Theoretical Computer Science, 744:143–169). Spatial aspects include, for example, safe gaps. As their motivating use case, the authors investigate the road junction rules of the UK Highway Code.

[0015] So far, we have summarized relevant research on traffic rule formalization that relies on manually crafted traffic rules defined in various forms of formal logic. However, another interesting aspect of rule formalization is how to (partially) automatically generate these formal representations from natural language variants of the formal representations present in the original rulebook. Although not in the field of autonomous driving, the methods proposed by He et al. and Singh et al. may also be applicable to traffic rules.

[0016] He, J., Bartocci, E., Nickovic, D., Isakovic, H., and Grosu, R. (From english to signal temporal logic, CoRR, abs / 2109.10294) present a method for converting natural language expressions into signal temporal logic (STL). STL "allows the expression of real-time requirements for continuous-time real-valued behavior." The authors formulate the conversion of natural language requirements as a machine translation task and implement it using the Transformer architecture (Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN, Kaiser, Lu, and Polosukhin, I., Attention is all you need, Neural Information Processing Systems 30, pp. 5998-6008). The application area is the analysis of requirements for cyber-physical systems.

[0017] Similarly, Singh, H., Aggarwal, M., and Krishnamurthy, B. (Exploring neural models for parsing natural language into first-order logic, CoRR, abs / 2002.06544) proposed a system to investigate the translation from English sentences to first-order logic (FOL). The authors used an LSTM-based encoder-decoder architecture to treat the generation of FOL as a sequence-to-sequence task.

[0018] For autonomous vehicles (AVs) rather than human drivers, Shalev-Shwartz, S., Shammah, S., and Shashua, A. (On a formal model of safe and scalable self-driving cars, CoRR, abs / 1708.06374) introduced the Responsibility-Sensitive Safety (RSS) model. The model formalizes a minimal set of five “common sense” traffic rules that are claimed to allow collision-free driving when followed by all AVs. For example, one of the minimalist rules states: “right-of-way is given, not taken.” While this is true, the RSS model does not formalize the AV’s responsibility to take the right of way. when The concept of actually having the right of way, for example, varies in different countries.

[0019] Known approaches address this challenge by defining traffic rules in formal logic, which the AV can then use to check, for example, whether it has the right of way, whether a lane change is allowed, etc.

[0020] However, known methods propose a process of manually translating traffic rules from natural language representations into formal logic. However, such manual translation of traffic rules is not scalable and requires human experts to manually create a formal logic representation of the traffic rules.

[0021] It is desirable to provide a system that allows for the automatic translation of natural language traffic rules into formal logic.It is desirable to have as basic input a standard rulebook designed for human drivers (eg national laws and legal regulations) as the basic input to the provided system.

[0022] The development of a solution to provide such a system was based on the following considerations.

[0023] The system leverages the language understanding capabilities of large pre-trained language models to bootstrap formal representations of traffic rules, such as T5 (Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J., Exploring the limits of transfer learning with a unified text-to-text transformer, Journal of Machine Learning Research, 21(140): 1–67); GPT-3 (Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Hen ighan,T.,Child,R.,Ramesh,A.,Ziegler,D.,Wu,J.,Winter,C.,Hesse,C.,Chen,M.,Sigler,E.,Litwin,M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I. and Amodei, D., Language models are few-shot learners [Language models are few-shot learners], Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (Eds.), Advances in Neural Information Processing Systems [Advances in Neural Information Processing Systems], 33, pp. 1877–1901, Curran Associates, Inc); OPT (Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, XV, Mihaylov, T., Ott, M., Shleifer, S., Shuster, K., Simig, D., Koura, P. S., Sridhar, A., Wang, T., and Zettlemoyer, L., OPT: Open pre-trained transformer language models [OPT: Open pre-trained transformer language model]); or MT-NLG (Smith, S., Patwary, M., Norick, B., LeGresley, P., Rajbhandari, S., Casper, J., Liu, Z., Prabhumoye, S., Zerveas, G., Korthikanti, V., Zheng, E., Child, R., Aminabadi, R. Y., Bernauer, J., Song, X., Shoeybi, M., He, Y., Houston, M., Tiwary, S., and Catanzaro, B., Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model[Megatron-Turing NLG 530b — A Large-Scale Generative Language Model Trained with Deepspeed and Megatron], CoRR, abs / 2201.11990).

[0024] We envision a human-in-the-loop scenario, where our system takes natural language traffic rules as input and generates formal representations of these rules in the best possible way. Since some traffic rules may be difficult even for humans to encode into formal representations, we believe that human experts are needed in the downstream process to review and fine-tune the generated formal representations of traffic rules so that they can be used by AVs.

[0025] The proposed system aims to solve the "open problem" of creating a representation in formal logic from natural language traffic rules (Wellhausen, L. and Jacob, MG, Map-optimized probabilistic traffic rule evaluation, 2016 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3012-3017), a challenge that was "beyond the scope of [their] paper." While previous research has mainly involved methods for manually encoding traffic rules, until now, there has been no system that can automatically encode natural language traffic rules into a formal system in the field of autonomous driving.

[0026] A first aspect of the invention relates to a method for preparing a machine learning system for translating natural language traffic rules into a formal logic representation for an autonomous mobile (or self-driving) vehicle. The machine learning system comprises a (large) pre-trained language model that has been trained on a large amount of text data to assign probabilities to word sequences. The machine learning system is prepared by one of the following options:

[0027] i. Use the pre-trained language model to prompt as follows:

[0028] -The input of the pre-trained language model is natural language instructions and language traffic rules,

[0029] -Presenting multiple (e.g., five or ten) inputs and corresponding formal logic target outputs as (few sample) instances to the machine learning system so that the machine learning system learns to generate formal logic traffic rules when prompted with natural language instructions and natural language traffic rules.

[0030] ii. Fine-tuning the machine learning system by applying training data including linguistic traffic rules as training input and corresponding formal logic traffic rules as target output to the pre-trained language model (or, respectively, the machine learning system).

[0031] The language traffic rules include natural language rules and / or synthetic language traffic rules.

[0032] Fine-tuning can be performed by means of supervised learning methods or similar methods. The parameters of the machine learning system can be adjusted by minimizing a loss function that characterizes the deviation between the output generated by the machine learning system when fed with linguistic traffic rules as input and the corresponding (target) output known from the training data. A deviation threshold can be defined that, when reached during training, ends the training process.

[0033] The machine learning system may include a neural network, a recurrent neural network, a gated recurrent neural network, or a long short-term memory network (LSTM). The neural network may have an encoder-decoder architecture. A specific example is the Transformer architecture proposed by Vaswani et al. cited above.

[0034] Large amounts of text data may include more than 1 million or more than 1 billion tokens. Examples of pre-trained language models trained on large amounts of text data (“large” pre-trained language models) have been discussed above (Raffel et al.'s T5, Brown et al.'s GPT-3, Zhang et al.'s OPT, and Smith et al.'s MT-NLG).

[0035] According to an embodiment, a pre-trained language model is used for prompting. During preparation, a plurality of training inputs and corresponding (expected) formal logic target outputs are presented to the machine learning system as (few sample) examples, so that the machine learning system adapts during reasoning time to generate formal logic traffic rules (expressions) when prompted with natural language instructions and natural language traffic rules.

[0036] In one example, the natural language instruction includes a request for generating a corresponding formal traffic rule for a certain language traffic rule, which is provided as an input to the machine learning system.

[0037] In one example, a pre-trained language model is used for prompts, and the input includes synthetic language traffic rules or pseudo natural language rules.

[0038] According to another embodiment, the pre-trained language model is fine-tuned by providing the machine learning system with pairs of language traffic rules and corresponding (expected) formal logic traffic rules, so that after the fine-tuning training is completed, the machine learning system learns to generate formal logic traffic rules when fed with natural language traffic rules as input.

[0039] In an example, the training data used to fine-tune the model includes generated formal logic traffic rule representations and corresponding synthetic language traffic rules.

[0040] In an example, the corresponding synthetic language traffic rules are used to generate corresponding pseudo natural language rules.

[0041] In an example, these synthetic language traffic rules or pseudo natural language traffic rules are enhanced by existing datasets including natural language traffic rules and (predetermined, e.g., human-defined, corresponding / ground truth) formal logic traffic rule representations.

[0042] In an example, the formal logic traffic rules (representations) include atomic propositions and implications, and optionally Boolean operations.

[0043] A second aspect of the present invention relates to a machine learning system configured to translate natural language traffic rules into formal logic traffic rules (representation) for autonomous driving, wherein the machine learning system is prepared using a method according to the first aspect of the present invention.

[0044] A third aspect of the invention relates to a method for translating natural language traffic rules from written laws applicable to a geographical area into formal logic traffic rules (representations) using a machine learning system prepared according to the second aspect of the invention. The machine learning system is configured to generate a formal logic representation of these traffic rules applicable to the geographical area. The method can be implemented in a computer or a suitable processing unit.

[0045] A fourth aspect of the invention relates to a system for autonomously moving (particularly driving) a vehicle, the system comprising a processing unit. The processing unit is configured to process data from at least one sensor (e.g., a camera, a radar or a lidar sensor) sensing the surroundings of the vehicle in order to control the autonomous movement (or driving) of the vehicle. The autonomously moving vehicle is controlled based on the processed sensor data while taking into account the generated formal logic representation of traffic rules applicable to the geographical area in which the vehicle is currently driving, generated by adopting the method according to the third aspect of the invention.

[0046] A fifth aspect of the invention relates to an autonomous mobile vehicle comprising a system according to the fifth aspect of the invention.

[0047] For a person skilled in the art, examples of how to consider the resulting formal logical representation of traffic regulations can be derived from the patent references presented in the introduction.

[0048] Using the presented possibility of automatically translating traffic rules into formal logic, the generated formal rules can be used, for example, to guide the navigation of an autonomous vehicle along the lines of US10,133,275B1, or to communicate the reasons for AV behavior to the user along the lines of US10,372,130B1.

[0049] The processing unit may correspond to a microcontroller or microprocessor, a CPU (central processing unit), a GPU (graphics processing unit), a TPU (tensor processing unit), an NPU (neural processing unit), a digital signal processor (DSP), an ASIC (application-specific integrated circuit), an FPGA (field programmable gate array), etc., and may include software for executing the steps of the method.

[0050] The geographic region may be a state or country where specific traffic regulations apply. The region in which the vehicle is currently driving may be obtained through the vehicle's navigation system or other type of positioning device.

[0051] A fifth aspect relates to an autonomous driving vehicle comprising a system according to the fourth aspect of the invention.

[0052] Further variations and embodiments are presented in the following paragraphs.

[0053] The proposed system can have the following characteristics:

[0054] 1. The system operates on traffic rules formulated in natural language.

[0055] 2. The system generates synthetic traffic rule instances. To create such instances, the formal logic is first generated in formal logic—following the template-based approach (He et al.), and then the formal logic template is converted into natural language using a large pre-trained language model (e.g., T5 (Raffel et al.), GPT-3 (Brown et al.), OPT (Zhang et al.), or MT-NLG (Smith et al.)), so that pairs of (formal logic, natural language) are generated.

[0056] Synthetic traffic rules example applied to

[0057] a. in hint methods, where they are used to indicate the model, or

[0058] b. Fine-tuning methods, where they are used to fine-tune large pre-trained language models.

[0059] 3. The system uses a neural network, such as the Transformer architecture (Vaswani et al.) as its underlying architecture, and is built on top of a large pre-trained language model to convert natural language traffic rules into formal logic.

[0060] 4. The system outputs traffic rules in the form of formal logic.

[0061] These aspects will be described in detail below.

[0062] Figure 1 A flowchart showing the overall method, and

[0063] Figure 2A flow chart for generating and improving training data including synthetic language and formal logic traffic rule pairs is shown.

[0064] 1. Traffic rules formulated in natural language as input

[0065] Today, traffic rules are written in natural language, which is easy for humans to read. An example of a traffic rulebook is the German Road Traffic Code (Straβenverkehrsordnung; StVO): https: / / www.gesetze-im- internet.de / stvo_2013 / StVO.pdf , retrieved on 11 August 2022; Highway Code for England, Scotland and Wales: https: / / www.gov.uk / guidance / the-highway-code , retrieved on August 11, 2022; and the California Driver Handbook, California, United States: https: / / www.dmv.ca.gov / portal / handbook / california-driver-handbook / , retrieved on August 11, 2022. These regulations govern the rights and obligations of all road users.

[0066] Traffic rules range from very general rules to highly specific rules that govern very specific traffic situations. Regarding the nature of traffic rules, Henry Prakken states that "[traffic] rules have a clear conditional and deontic structure", while Esterle et al. challenge this positive assumption, stating that in fact "traffic rules are vague and not clearly defined". We argue that in fact the nature of traffic rules varies significantly, and that the proposed system for translating traffic rules into a representation in formal logical form may indeed work better for rules that exhibit a clear "conditional and deontic structure".

[0067] In general, we propose a system that operates directly on such standard traffic rules found in the standard rulebook, whereby we restrict the system to processing rules with a clear “conditional and deontic logical structure”.

[0068] 2. Generate synthetic examples through templating and using large pre-trained language models

[0069] We propose to generate synthetic instances of traffic rules in the following way. He et al. introduced a templated approach for compiling rules in formal logic based on hierarchical compositional layers. These layers range from atomic propositions of Boolean combinations to tense phrases and nested tense phrases. In our case, we adopt the core idea and propose a similar approach for traffic rules.

[0070] Figure 2 A diagram of the following steps is shown in FIG.

[0071] a. Generate synthetic examples using formal logic

[0072] An example of the basic syntax might look like this:

[0073] Atomic proposition α: at (vehicle, T-junction)

[0074] In (vehicle, red traffic light)

[0075] In (vehicle, rightmost lane)

[0076] Turn(vehicle,left)

[0077] Boolean operations:

[0078] At (vehicle, T-junction)∧turn (vehicle, left)

[0079] In (vehicle, rightmost lane) ∧ In (vehicle, red traffic light)

[0080] In (vehicle, red traffic light)

[0081] Contains:

[0082]

[0083] Once a catalog of atomic propositions α and constraints on Boolean operations (e.g., constraints on the nesting depth of phrases) are established, synthetic traffic rules can be automatically generated in formal logic.

[0084] We emphasize that the formal rules are generated by a random generator, which does not check whether the generated rules conform to reasonable traffic rules. This is not necessary because the synthesized examples are only used to align the language model with their formal logic counterparts, and it is not important whether the synthesized examples are reasonable at this stage.

[0085] b. Generate pseudo natural language rules

[0086] Once completed, we also need to define the transformation steps that convert formal logic expressions into natural language expressions. For example:

[0087] Atomic proposition α: in (x, y) "x is at y"

[0088] turn(x, direction) "x turns in a certain direction"

[0089] Parking required "Parking required"

[0090] Boolean operation: ∧ “and”

[0091] "No"

[0092] Contains: “If P, then P”

[0093] Therefore, the following rules can be translated into natural language expressions (admittedly, some of them are grammatically incorrect):

[0094] enter:

[0095] Output: If the vehicle is at a red traffic light, it does not need to stop.

[0096] b. Generate “more natural” natural language rules

[0097] We propose to use large pre-trained language models (e.g., T5 (Raffel et al.), GPT-3 (Brown et al.), OPT (Zhang et al.), or MT-NLG (Smith et al.)) to generate paraphrases of the generated rules. By using a large pre-trained language model, we can obtain a set of outputs, such as the following. Since all these outputs are semantically equivalent, we can map them to the same formal expression.

[0098] Input: If the vehicle is at a red traffic light, it does not need to stop.

[0099] Output: (1) If the vehicle is not at a red traffic light, there is no need to stop.

[0100] (2) If your vehicle is at a red traffic light, you do not need to stop.

[0101] Thus, a pair (formal logic, natural language) can be formed by taking the original instance presented in formal logic (see above) and its resulting natural language representation. For example:

[0102] Example 1:

[0103] If your vehicle is not at a red traffic light, you do not need to stop.

[0104] Example 2:

[0105] If your vehicle is at a red traffic light, you do not need to stop.

[0106] c. Enhance synthetic examples using existing datasets

[0107] In addition to these generated synthetic examples, we can further combine synthetic datasets with traffic rules as paired (formal expressions, natural language) found in the literature.

[0108] For example, we can enhance our dataset by utilizing traffic rules proposed by Karimi and Duggirala.

[0109] 3. A large pre-trained language model using a neural network for translating traffic rules into a formal representation

[0110] Large pre-trained language models (e.g., T5 (Raffel et al.), GPT-3 (Brown et al.), OPT (Zhang et al.), or MT-NLG (Smith et al.)) are the basis for many language processing applications today.

[0111] This type of model is also called a foundation model (Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N. S., Chen, A. S., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., G. Illespie, L., Goel, K., Goodman, N.D., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D.E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P.W., Krass, M.S., Krishna, R., Kuditipudi, R., et al., On the opportunities and risks of foundation models [On the opportunities and risks of foundation models], CoRR, abs / 2108.07258) builds on the Transformer architecture (Vaswani et al.) and is “trained on extensive data at scale” (Bommasani et al.) to learn general language understanding before adapting to the task and data at hand. The data used for training usually covers encyclopedic knowledge (e.g., Wikipedia), book corpora, and public data scraped from the Web.

[0112] Most previous research in autonomous driving has focused on manually translating traffic rules into formal representations. However, the methods proposed by He et al. and Singh et al. may also be applicable to traffic rules, with the former's system also built on the Transformer architecture.

[0113] Different from He et al., we propose to leverage the language understanding capabilities of large pre-trained language models to translate natural language traffic rules into formal representations.

[0114] Given these models, the pairs of (formal logic, natural language) generated in step 2 can then be used to simultaneously prompt the model or fine-tune it.

[0115] i. Prompt method

[0116] In the hinting approach, we use these synthetic examples to elicit formal representations from a language model.

[0117] At inference time, the model is provided with instructions such as "Translate the following traffic rule into formal logic: the speed limit on any side street is 15 mph", along with a set of 1...n traffic rule synthesis instances (n>=1) and their corresponding parts presented in formal logic.

[0118] These instances are intended to guide and instruct the model as it generates its output. Through prompts, the model is expected to provide the desired output at the end of the prompt.

[0119] ii. Fine-tuning method

[0120] In our fine-tuning approach, we use these synthetic examples to fine-tune a large pre-trained language model instead of hinting it.

[0121] At inference time, the model is only fed with natural language traffic rules to be converted into formal logic.

[0122] The fine-tuned model then generates formal logic in a typical sequence-to-sequence manner without providing prompt instructions.

[0123] 4. Representation of traffic rules in formal logic

[0124] In his conceptual work, Henry Prakken discusses three design approaches to making autonomous vehicles comply with traffic laws: regulation, training, and reasoning. Regulation refers to model checking capabilities that allow the AV to check, in any situation, whether its current actions comply with given traffic rules. Training, on the other hand, refers to the practice of exposing the AV to as many different traffic situations as possible during the training phase so that the AV can implicitly learn how to comply with traffic laws. Reasoning is similar to regulation, but goes a step further and equips the AV with "normative reasoning capabilities" about its current actions and a given set of traffic rules. In terms of the reliability of reasoning, Henry Prakken concludes that a formal representation of traffic rules is a necessary prerequisite. As for the nature of this formal representation, the author states that "[traffic] rules have clear conditional and deontic-logical structures."

[0125] Previous work on formally representing traffic rules has encoded the rules in various formal systems such as first-order logic (Wellhausen and Jacob), temporal logic (Alves et al.; Rizaldi and Althoff; Esterle et al.; Maierhofer et al.; He et al.), spatial logic (Schwammberger and Alves), answer set programming (Karimi and Duggirala), or defeasible deontic logic (Bhuiyan et al.). We argue that the specific formal logic system into which natural language traffic rules are translated depends on the chosen downstream usage scenario and application requirements.

[0126] In general, we propose to keep the formal logic system as simple as the downstream usage scenarios and application requirements allow. For example, if we follow the idea of ​​behavioral-semantic scene description (BSSD; Glatzki, F., Lippert, M., and Winner, H., Behavioral attributes for a behavior-semanticscenery description (BSSD) for the development of automated driving functions for traffic situations, 2021 IEEE International Intelligent Transportation Systems Conference [2021 IEEE International Intelligent Transportation Systems Conference (ITSC)], pp. 667-672), we can decompose the traffic situation into a sequence of atomic traffic segments. Within each segment, we do not need any temporal components in the formal logic system, because each segment is an atomic temporal unit. Therefore, when using BSSD, the expressiveness of temporal logic may be unnecessary. Similarly, when choosing BSSD as the modeling method, the expressiveness of spatial logic may also be unnecessary, because each segment is atomic not only from a temporal point of view, but also from a spatial point of view.

[0127] The proposed solution has the following advantages:

[0128] 1. Bootstrapping the automatic translation of traffic rules from a standard rulebook formulated in natural language to a formal representation.

[0129] 2. Leverage a large pre-trained language model (i.e., base model) to translate traffic rules without having to train a Transformer model from scratch.

[0130] 3. The formal representation of traffic rules equips autonomous vehicles with “normative reasoning capabilities.” (Henry Prakken)

[0131] 4. Autonomous vehicles can account for different jurisdictions (and therefore, different sets of traffic rules) or new sets of rules (e.g., revisions to traffic regulations) by swapping in / out relevant formal representations.

[0132] 5. Since there is no ambiguity in the logical clauses, the interpretability of traffic rules for autonomous vehicles is improved.

[0133] Embodiments of the present invention may be used in any situation where autonomous navigation decisions are made in a space where traffic regulations apply.Within the scope of the present invention, the autonomous vehicle may also be, for example, a last-mile delivery robot or drone.

[0134] Below we give a specific end-to-end example to illustrate how the present invention can be used to translate traffic rules from an official rulebook formulated in natural language into a formal representation presented in formal logic.

[0135] 1. Traffic rules formulated in natural language as input

[0136] In this example, we illustrate the translation of traffic rules taken from the Laws and Rules of the Road section of the California Driver's Manual.

[0137] Traffic rules: The speed limit on any side street is 15 mph.

[0138] The rule has a clear "conditional and deontic logical structure" (Henry Prakken) and can therefore be considered a valid input according to our definition of rules supported by the system.

[0139] 2. Generate synthetic examples through templating and using large pre-trained language models

[0140] Following the process described above, we define a set of few-shot examples that can be used to hint or fine-tune a large pre-trained language model.

[0141] Few-shot instances that are semantically relevant to the traffic rules we are studying should be preferred over less relevant instances.

[0142] Instance pairs including few-shot examples (natural language traffic rules, formal representation) may include:

[0143] "If the vehicle is in the rightmost lane, the speed limit is 40."

[0144] →In (vehicle, rightmost lane)->Speed ​​limit (40)

[0145] "For vehicles within school zones, the speed limit is 10."

[0146] →Located in (vehicle, school zone)->Speed ​​limit (10)

[0147] 3. A large pre-trained language model using a neural network for translating traffic rules into a formal representation

[0148] Figure 1 A flow chart showing a general method of translating traffic rules into a formal representation. In this end-to-end example, we illustrate how the present invention can be used to translate traffic rules from an official rulebook written in natural language into a formal representation presented in formal logic.

[0149] As the basis of our system, we propose to try different large pre-trained language models to identify the one that shows the best performance. Examples of large pre-trained language models include T5 (Raffel et al.), GPT-3 (Brown et al.), OPT (Zhang et al.), or MT Promising NLG (Smith et al.). In an initial set of exploratory studies, GPT-3 has shown preliminary results.

[0150] i. Prompt method

[0151] An example prompt for a large pre-trained language model could be:

[0152] Hint: Please translate traffic rules into formal logic.

[0153] Traffic rules: "If the vehicle is in the rightmost lane, the speed limit is 40."

[0154] Formal logic: in (vehicle, rightmost lane) -> speed limit (40)

[0155] Traffic rules: "For vehicles located in school zones, the speed limit is 10."

[0156] Formal logic: Located in (vehicle, school zone) -> speed limit (10)

[0157] Traffic rules: The speed limit on any side street is 15 mph.

[0158] Formal logic: ______

[0159] We expect the language model to fill in the gap after the final “formal logic” label.

[0160] ii. Fine-tuning method

[0161] In the fine-tuning approach, we use synthetic examples from (2) to optimize the parameters of a large pre-trained language model.

[0162] We then apply the optimized (i.e., fine-tuned) model to the classic sequence-to-sequence translation task, where the input is natural language traffic rules and the model is expected to translate the rules into formal logic. For example:

[0163] "If the vehicle is in the rightmost lane, the speed limit is 40."

[0164] ↓

[0165] [Optimized model]

[0166] ↓

[0167] In (vehicle, rightmost lane) -> speed limit (40)

[0168] Figure 2 Shows how to define a set of synthetic examples that can be used to hint or fine-tune a large pre-trained language model.

[0169] 4. Representation of traffic rules in formal logic

[0170] Henry Prakken concluded that a formal representation of traffic rules is a necessary prerequisite. Previous studies on the formal representation of traffic rules have encoded the rules in various formal systems, such as first-order logic (Wellhausen and Jacob), temporal logic (Alves et al.; Rizaldi and Althoff; Esterle et al.; Maierhofer et al.; He et al.), spatial logic (Schwammberger and Alves), answer set programming (Karimi and Duggirala), or defeasible deontic logic (Bhuiyan et al.).

[0171] As mentioned above, we believe that the specific formal logic system for translating natural language traffic rules depends on the chosen downstream usage scenario and application requirements.

Claims

1. A method for preparing a machine learning system for translating natural language traffic rules into a formal logic representation for an autonomous moving vehicle, in, The machine learning system includes a pre-trained language model that has been trained on a large amount of text data to assign probabilities to word sequences, wherein preparing the machine learning system includes: Use this pre-trained language model to prompt as follows: The input of the pre-trained language model is natural language instructions and language traffic rules. presenting a plurality of inputs and corresponding formal logic target outputs as examples to the machine learning system so that the machine learning system learns to generate formal logic traffic rules when prompted with natural language instructions and natural language traffic rules; or fine-tuning the machine learning system by applying training data including linguistic traffic rules as training input and corresponding formal logic traffic rules as target output to the pre-trained language model; Among them, these language traffic rules include natural language rules and / or synthetic language traffic rules.

2. The method according to claim 1, wherein: The pre-trained language model is used for prompting, and wherein the natural language instructions include a request for generating a corresponding formal traffic rule for a certain language traffic rule.

3. The method according to claim 1 or 2, wherein: The pre-trained language model is used for prompting, and wherein the inputs include synthetic language traffic rules or pseudo natural language rules.

4. The method according to claim 1, wherein: The pre-trained language model is fine-tuned by providing the machine learning system with pairs of language traffic rules and corresponding formal logic traffic rules, so that after the fine-tuning training is completed, the machine learning system learns to generate formal logic traffic rules when fed with natural language traffic rules as input.

5. The method according to claim 4, wherein: The training data used to fine-tune the model includes generated formal logic traffic rule representations and corresponding synthetic language traffic rules.

6. The method according to claim 5, wherein: These corresponding synthetic language traffic rules are used to generate corresponding pseudo natural language rules.

7. The method according to claim 5 or 6, wherein: These synthetic language rules or pseudo natural language rules are augmented by existing datasets including natural language traffic rules and formal logic traffic rule representations.

8. A method according to any one of the preceding claims, wherein: Formal logic consists of atomic propositions and implications, and optionally Boolean operations.

9. A machine learning system configured to translate natural language traffic rules into formal logic traffic rules for autonomous driving, wherein: The machine learning system is prepared using the method according to one of claims 1 to 8.

10. The machine learning system according to claim 9, wherein: The machine learning system includes a neural network, a recurrent neural network, a gated recurrent neural network or a long short-term memory network.

11. A machine learning system according to claim 9 or claim 10, wherein: The machine learning system includes an encoder-decoder architecture.

12. The machine learning system of claim 11, wherein: The machine learning system includes the Transformer architecture.

13. A method for translating natural language traffic rules from written laws applicable to a geographical area into formal logic traffic rules using a machine learning system according to one of claims 9 to 12, the machine learning system being configured to generate a formal logic representation of these traffic rules applicable to the geographical area.

14. A system for an autonomous mobile vehicle, the system comprising a processing unit, wherein: The processing unit is configured to process data from at least one sensor sensing the vehicle's surroundings so as to control the autonomous driving of the vehicle based on the processed sensor data while taking into account a formal logical representation of traffic rules applicable to the geographical area in which the vehicle is currently driving, generated by adopting the method according to claim 13.

15. An autonomous mobile vehicle comprising the system according to claim 14.

Images