Method of generating code

Abstract

The present invention relates to a method of generating code, in particular to a method of generating code for at least partially automated driving functions of a motor vehicle, comprising: receiving a data set comprising a plurality of driving scenario data, each describing a driving scenario; processing each driving scenario data using a first large language model to obtain a textual description of the driving scenario for each driving scenario; processing each textual description of each driving scenario using a second large language model to create a rule set from the textual descriptions, the rule set consisting of rules summarizing each textual description; generating a finite state machine describing the at least partially automated driving functions based on the created rule set; verifying the generated state machine; generating code for the at least partially automated driving functions of the motor vehicle based on the finite state machine and depending on the result of the verification of the finite state machine. The present invention relates to an apparatus, a computer program and a machine-readable storage medium.

Description

Technical Field

[0001] The present invention relates to a method, an apparatus, a computer program, and a machine-readable storage medium for generating code for at least partially automated driving functions of a motor vehicle. Background Technology

[0002] The publication “DriveLM: Driving With Graph Visual QuestionAnswering” by Sima et al. discloses the use of a large language model (available at “https: / / arxiv.org / abs / 2312.14150”).

[0003] Publication DE 10 2022 207616 A1 discloses a computer implementation method for verifying software components for automated driving functions.

[0004] Publication EP 4 332 824 A1 discloses a system and method for converting traffic rules into formal logic for autonomous vehicles.

[0005] Patent document US 8,452,524 B2 discloses a method and apparatus for identifying traffic-related information.

[0006] Patent document US 11,132,211 B1 discloses a finite state machine.

[0007] Patent document US 8,948,936 B2 discloses a vehicle management system using a finite state machine. Summary of the Invention

[0008] The objective of this invention is to provide a concept for generating code for at least some automated driving functions of motor vehicles.

[0009] This task is addressed by the respective subject matter of the independent claims. The advantageous designs of the invention are the subject matter of the respective dependent claims.

[0010] According to a first aspect, a method for generating code for at least some automated driving functions of a motor vehicle is provided, comprising the following steps:

[0011] Receive a dataset that includes multiple driving scenario data, each describing a driving scenario.

[0012] The first large language model is used to process the data of each driving scenario separately in order to obtain a text description of the driving scenario for each driving scenario;

[0013] A second large language model is used to process each text description of each driving scenario to create a rule set from the text descriptions, which consists of rules that summarize each text description.

[0014] Generate a finite state machine describing at least some of the automated driving functions based on the created rule set;

[0015] Verify the generated state machine;

[0016] Based on this finite state machine and depending on the verification results of this finite state machine, code for at least some automated driving functions of motor vehicles is generated.

[0017] According to the second aspect, an apparatus is provided, which is configured to perform all the steps of the method according to the first aspect.

[0018] According to a third aspect, a computer program is provided that includes instructions that, when executed by a computer (e.g., the means according to the second aspect), cause the computer to perform the method according to the first aspect.

[0019] According to a fourth aspect, a machine-readable storage medium is provided having a computer program according to a third aspect stored thereon.

[0020] This invention is based on and includes the understanding that the aforementioned task is solved as follows: The driving scenario is processed by a first large language model to obtain a textual description of the driving scenario. Based on these textual descriptions, a rule set is created using a second large language model, consisting of rules summarizing the various textual descriptions. A finite state machine describing the at least partially automated driving function is then generated from this rule set. The generated state machine can be efficiently verified, particularly its correctness can be checked. Depending on the verification check, code for the at least partially automated driving function is generated based on the finite state machine.

[0021] Therefore, code for at least partially automated driving functions is ultimately provided, which is indirectly checked by verifying its underlying finite state machine. This contributes to the safety and integrity of at least partially automated driving functions of motor vehicles.

[0022] The code referred to in this manual is software code. For example, the code referred to in this manual may be native program code in a programming language.

[0023] The term "Large Language Model" as used in this specification can be abbreviated as LLM. The expression "Large Language Model" can also be translated as "Large Language Model". An LLM is a language model characterized by its text generation capabilities. It is a probabilistic model in computational linguistics that learns statistical word and sentence sequence relationships from a large number of text documents through a computationally intensive training process. Such language models, in a broader sense, are particularly artificial neural networks and can therefore be trained, for example, a priori through self-supervised learning or semi-supervised learning methods.

[0024] For example, the first large language model is the same as the second large language model. Alternatively, the first large language model may differ from the second large language model. For example, it may be specified that different first large language models are used for processing data from different driving scenarios. Alternatively, it may be specified that the same first large language model is used for processing data from different driving scenarios.

[0025] For example, it is stipulated that if the verification results of the finite state machine indicate that one or more errors have occurred, i.e., the finite state machine has one or more errors, and / or if the check of the correctness of the finite state machine during the verification process indicates that the finite state machine is incorrect or cannot operate correctly, then no code is generated, i.e., the generation of code for at least part of the automated driving functions of a motor vehicle is abandoned. Here, "correct" specifically means that the finite state machine does not operate correctly within the required specifications.

[0026] In one implementation of this method, a summary rule is predefined for a second large language model, which summarizes each text description into conditional statements, thereby creating a rule set containing conditional statements as rules.

[0027] This results in, for example, the following technical advantages: providing particularly suitable rules based on which finite state machines can be generated with exceptional efficiency.

[0028] The conditional statement referred to in this specification is, for example, an "if-then" statement or an "if-then-else" statement. Therefore, a conditional statement can be an If-Then statement or an If-Then-Else statement.

[0029] In one implementation of the method, it is stipulated that a first large language model should be predefined to describe what happens in each driving scenario and, in particular, why it happens, so that each text description indicates what happens in each driving scenario and, in particular, why it happens.

[0030] This results in, for example, the following technical advantages: each text description contains particularly meaningful content, which enables the creation of rules based on this.

[0031] In one implementation of the method, a formalization is performed, based on which states and state transitions for a finite state machine are derived from a set of rules created, wherein the finite state machine is generated based on the derived states and state transitions.

[0032] According to this implementation, it is therefore stipulated that a finite state machine is generated by performing formalization. According to this formalization, the created rule set is converted into a finite state machine by deriving the states and state transitions for the finite state machine from the created rule set.

[0033] This results in the following technical advantages, for example: the ability to efficiently generate finite state machines.

[0034] In one implementation of the method, the verification includes: visualizing the finite state machine in a human-understandable form for human verification, and collecting inputs representing the results of human verification, wherein the code is generated based on the collected inputs.

[0035] This results in technological advantages such as enabling humans to efficiently verify finite state machines.

[0036] In one implementation of the method, the verification is specified to include: using model checking to verify the finite state machine.

[0037] This results in the following technical advantages, for example: verification can be performed efficiently.

[0038] Model checking is a process of fully automated verification of the system description (currently the state of the state machine) against the specification. For example, a "symbolic model validator" (also abbreviated as "SMV") can be used to verify finite state machines.

[0039] The finite state machine referred to in this specification may also be called a finite automaton or finite state automaton. In English, the term "Finite State Machine (FSM)" is used. The state machine referred to in this specification contains or consists of states, state transitions, and actions. Therefore, the state machine referred to in this specification is specifically a model of behavior (currently, at least the behavior of partially automated driving functions).

[0040] The method described in the first aspect is, for example, a computer-implemented method.

[0041] The device is, for example, a device configured to execute the computer program using programming techniques.

[0042] The device is, for example, a computer.

[0043] Method features are similarly derived from corresponding device features, and vice versa.

[0044] The technical functions and features of the method are similarly derived from the corresponding technical functions and features of the device, and vice versa.

[0045] The method is performed, for example, by the device.

[0046] The implementation methods and examples described herein can be combined with each other in any way, even if not explicitly described.

[0047] The driving scenario data referred to in this manual includes, for example, one or more of the following: video data, radar data, lidar data, ultrasonic data, and infrared data.

[0048] The driving scenarios referred to in this manual are specifically from road traffic scenarios, and therefore, driving situations in road traffic are described, for example.

[0049] The expression "at least partially automated" includes the following expressions: partially automated, highly automated, fully automated, and autonomous.

[0050] Partial automation corresponds to Level 2 of automation as defined by the German Federal Highway Research Institute (BASt). High automation corresponds to Level 3 of automation as defined by BASt. Full automation corresponds to Level 4 of automation as defined by BASt. Autonomous automation corresponds to Level 5 of automation as defined by SAE (J3016), where SAE stands for "Society of Automotive Engineers". Attached Figure Description

[0051] The present invention will now be described in more detail based on preferred embodiments. In the accompanying drawings:

[0052] Figure 1 A flowchart of the method according to the first aspect is shown.

[0053] Figure 2 The apparatus according to the second aspect is shown.

[0054] Figure 3 A machine-readable storage medium according to the fourth aspect is shown, and

[0055] Figure 4 A block diagram is shown. Detailed Implementation

[0056] Figure 1 A flowchart illustrating a method for generating code for at least partially automated driving functions of a motor vehicle is shown, comprising the following steps:

[0057] Receive a dataset of 101, which includes multiple driving scenario data, each describing a single driving scenario.

[0058] The first large-scale language model was used to process data from 103 different driving scenarios to obtain a textual description of each scenario.

[0059] A second large language model was used to process the text descriptions of 105 different driving scenarios to create a rule set consisting of rules summarizing the individual text descriptions.

[0060] Based on the created rule set, generate 107 finite state machines describing at least some of the automated driving functions.

[0061] Verify the state machine generated by 109.

[0062] Based on this finite state machine and depending on the verification results of this finite state machine, code 111 is generated for at least some automated driving functions of motor vehicles.

[0063] Figure 2 An apparatus 201 is shown, which is configured to perform all the steps of the method according to the first aspect.

[0064] Figure 3 A machine-readable storage medium 301 is shown, on which a computer program 303 is stored. The computer program 303 includes instructions that, when executed by a computer, cause the computer to perform the method according to the first aspect.

[0065] Figure 4 Block diagram 401 is shown, which exemplarily illustrates the concepts described herein.

[0066] The system receives a dataset 401, which includes multiple driving scenario data sets 403 and 405, each describing a driving scenario. The driving scenario data 405 may be, for example, video data.

[0067] Each driving scenario data 405 is processed using a first large language model 407 to obtain a textual description of each driving scenario. As a result of these processes, another dataset 409 is thus provided, which includes textual descriptions of driving scenarios.

[0068] A second large language model 411 is used to process text descriptions based on another dataset 409 to create a rule set 413 from the text descriptions, which consists of rules that summarize the individual text descriptions.

[0069] From the created rule set 413, the states and state transitions for the finite state machine are derived according to function block 415, so that the following state machine is generated based on the derived content: The generated finite state machine in Figure 4The figure is denoted by reference numeral 417. This finite state machine 417 describes or represents at least part of the automated driving function of a motor vehicle.

[0070] The finite state machine 417 is verified according to function block 419. For example, the verification 419 includes formal white-box verification.

[0071] The term "white-box verification" refers to the process where the object under test (e.g., code, currently a state machine) is known, and errors are directly checked (automatically here; otherwise, semi-automatic and / or manual checks could be performed alternatively and / or additionally). This term is contrasted with "black-box verification," which involves not directly examining the code but only observing its execution, such as in classic testing.

[0072] The verification 419 includes, for example, using model checking to verify the finite state machine 417.

[0073] Depending on the verification 419, code 423 for at least some automated driving functions of the motor vehicle is generated from the finite state machine 417 by a code generator 421 (e.g., which may be implemented in software and / or hardware). For example, native program code is generated.

[0074] The input to each of the first large-scale language models 407 could include, for example, the following: "Explain what happened and why in the driving scenario in as much detail as possible." For example, the input could include: "Explain what happened and why in the video in as much detail as possible."

[0075] The input to the second large language model 411 could include, for example, the following: "summarize the description as a rule of 'if X, then Y'". Here, "description" refers to a text description.

[0076] The concepts described in this paper allow for scalable, automatic generation of rule-based, and thus traceable, at least partially automated driving functions that are verifiable to humans and through formal verification.

[0077] As input, a dataset of traffic conditions that is as diverse as possible can be used, for example, from fleet measurements and / or simulations and / or internet videos. For example, this dataset is processed in the form of textual descriptions using generative artificial intelligence (e.g., GBT and / or LLM and / or base models), such as: "If the vehicle in front performs acceleration x, then the EGO vehicle performs acceleration y."

[0078] The result of this processing can be further processed into rules for a finite state machine within a post-processing scope. Therefore, as output, for example, rule-based code for generating a finite state machine can be produced.

[0079] The processing chains that can be implemented as described in this article may include, for example, the following or the following:

[0080] Dataset driving conditions – Generative Artificial Intelligence → Textual description of driving style – FSM extraction → Rule-based driving function.

[0081] FSM can be advantageously visualized in a human-understandable form through tools, and its correctness can be checked through formal methods.

[0082] Therefore, code for at least partially automated driving functions of motor vehicles can be generated efficiently, and this code has been checked or verified in an efficient and indirect manner. "Indirectly" here means that the finite state machine on which the generated code is based has been verified according to the scheme described herein, or with the aid of the tools and methods described herein.

[0083] Typically, rule-based driving functions have historically been manually written and struggle to cover all possible scenarios. On the other hand, in machine learning approaches, driving functions are implemented using deep neural networks, which are largely considered black boxes, potentially leading to issues with interpretability and verifiability. This means that because deep neural networks are often black boxes, the code generated by such networks is difficult to verify, or rather, the deep neural networks themselves that underpin the code are difficult to verify.

[0084] In contrast, the approach described in this paper enables the generation of interpretable and verifiable code for at least some of the automated driving functions of a motor vehicle, which means that the safety of such at least some automated driving functions of the motor vehicle is improved.

[0085] Cross-reference to related applications

[0086] This application claims priority to German patent application filed on December 11, 2024 (previous application number: 10 2024 211806.0). All disclosures of the above application are incorporated herein by reference.

Claims

1. A method for generating code (423) for at least some automated driving functions of a motor vehicle, the method comprising the steps of: Receive (101) a dataset (401), the dataset including multiple (403) driving scenario data (405), each driving scenario data describing a driving scenario. The first large language model (407) is used to process (103) each driving scenario data to obtain a text description of each driving scenario. A second large language model (411) is used to process (105) the individual text descriptions (409) of each driving scenario to create a rule set (413) from the text descriptions (409), the rule set consisting of rules summarizing the individual text descriptions (409). Based on the created rule set (413), generate (107) a finite state machine describing at least some of the automated driving functions. Verify the state machine generated by (109), Based on the finite state machine and depending on the verification results of the finite state machine (417), code (423) for the at least partially automated driving function of the motor vehicle is generated (111).

2. The method according to claim 1, wherein, The second large language model (411) is predefined with summarizing rules so that it summarizes the various text descriptions (409) into conditional statements, thereby creating a rule set (413) containing conditional statements as rules.

3. The method according to claim 1 or 2, wherein, The first large language model (407) is predefined to describe what happens and, in particular, why happens in the various driving scenarios, so that the various text descriptions (409) indicate what happens and, in particular, why happens in the various driving scenarios.

4. The method according to any one of the preceding claims, wherein, Perform formalization, and derive the states and state transitions for the finite state machine from the created rule set (413) according to the formalization, wherein the finite state machine (417) is generated based on the derived states and state transitions.

5. The method according to any one of the preceding claims, wherein, The verification includes: visualizing the finite state machine (417) in a human-understandable form for human verification, and collecting inputs representing the results of the human verification, wherein the code (423) is generated based on the collected inputs.

6. The method according to any one of the preceding claims, wherein, The verification includes: using model checking to verify the finite state machine (417).

7. An apparatus (201) configured to perform all the steps of the method according to any one of the preceding claims.

8. A computer program (303) comprising instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 6.

9. A machine-readable storage medium (301) on which a computer program (303) according to claim 8 is stored.

Images